Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings
Authors: Erel Naor
Ofir Lindenbaum
Venue: NeurIPS 2025 poster ()
Keywords: Tabular data
Self-supervised learning
Autoencoders
Hybrid models
Feature selection
Oblivious soft decision trees
Model-based augmentation
Primary area: general_machine_learning
TL;DR: We train a hybrid autoencoder with separate gating for a neural and tree-based encoder, achieving strong low-label classification and regression using only the gated neural encoder at inference.

Abstract: Deep neural networks often under-perform on tabular data due to their sensitivity to irrelevant features and a spectral bias toward smooth, low-frequency functions. These limitations hinder their ability to capture the sharp, high-frequency signals that often define tabular structure, especially under limited labeled samples. While self-supervised learning (SSL) offers promise in such settings, it remains challenging in tabular domains due to the lack of effective data augmentations. We propose a hybrid autoencoder that combines a neural encoder with an oblivious soft decision tree (OSDT) encoder, each guided by its own stochastic gating network that performs sample-specific feature selection. Together, these structurally different encoders and model-specific gating networks implement model-based augmentation, producing complementary input views tailored to each architecture. The two encoders, trained with a shared decoder and cross-reconstruction loss, learn distinct yet aligned representations that reflect their respective inductive biases. During training, the OSDT encoder (robust to noise and effective at modeling localized, high-frequency structure) guides the neural encoder toward representations more aligned with tabular data. At inference, only the neural encoder is used, preserving flexibility and SSL compatibility. Spectral analysis highlights the distinct inductive biases of each encoder. Our method achieves consistent gains in low-label classification and regression across diverse tabular datasets, outperforming deep and tree-based supervised baselines.

Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings
Erel Naor
Ofir Lindenbaum
Bar-Ilan University erel. naorlivee biu. ac.
Bar-Ilan University ofir. Saar.iinden 1indenbaum@biu.cc ac.
Abstract
Deep neural networks often underperform on tabular data due sensitivity ir- relevant features and a spectral bias toward smooth, low-frequency functions, lim- iting their ability capture sharp, high-frequency signals in low-label regimes. While self-supervised learning (SSL) holds promise in such settings, it remains challenging in tabular domains due the limited availability of effective data aug- mentations. We introduce TANDEM (Tree-And-Neurd) Dual Encoder Model), a hybrid autoencoder that trains neural encoder alongside an oblivious soft deci- sion tree (OSDT) encoder, both guided by dedicated stochastic gating networks for sample-specific feature selection. The encoders share decoder and are coupled via alignment losses, encouraging complementary yet consistent representations. The training-only use of the tree operates as model-based augmentation, nudging representations toward tabular-relevant features while preserving a lean inference path (only the neural encoder is deployed). Spectral analysis highlights distinct yet complementary inductive biases across encoders, and experiments classifi- cation and regression benchmarks in low-label settings show consistent gains over strong deep, tree-based, and SSL baselines.
1 Introduction
In many real-world applications, tabular data is the dominant data format, especially in domains such as healthcare and finance. Tree-based methods, such as gradient-boosted decision trees (GBDT) (6), XGBoost [5), CatBoost [14), are often the go-to models for classification tasks on tabular data, consistently delivering strong performance with minimal tuning. While deep neural networks have achieved impressive results in domains like vision and language, they often struggle match the per- formance of tree- based models in tabular settings. One contributing factor is the spectral inductive bias of neural networks: they tend favor smooth, fow-frequency functions, which may not align well with the complex, irregular, and often high-frequency patterns found in tabular datasets [15]. This limitation becomes especially pronounced when modeling interactions between heterogeneous features, such as mixed categorical and numerical variables. Moreover, tabular datasets often include nuisance features-varialee
that are irrelevant or mislcading in specific contexts. Neural networks often struggle isolate and suppress these features, resulting in overfitting or poor generalization.
In many real-world tabular domains, such as healthcare, biology, and finance, labeled data scarce and expensive, while unlabeled data is often abundant. This has led to growing interest in self- supervised learning (SSL) methods that can leverage unlabeled data to improve performance in low-label settings. However, applying SSL to tabular data is particularly challenging. Common augmentations such as noise injection or feature value swapping often distort critical feature re- lationships or create unrealistic samples, especially in the presence of categorical variables [18]. a result, general-purper augmentation strategies are challenging design and often require dataset-specific tuning.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To address this, Masked Autoencoders (MAEs) have been proposed as more structure preserving alternative. By learning to reconstruct selectively masked imputs, they provide a prineipled method for training models that does not rely on potentially unreliable augmentations. Yet even MAFs face limitations when applied heterogeneous tabular data, where the semanties of categorical features may be lost or misrepresented during corruption and reconstruction [1].
To mitigate these challenges, we propose TANDEM (Tree-And-Neural Dual Encoder Model), an alternative strategy that shifts the focus from data augmentation to model enrichment. Our approach employs a hybrid self--uperrer supe-siueesed autoencoder architecture composed of two fundamentally different encoder types: a deep neural network (NN) and an oblivious soft decision tree (OSDT), both trained jointly through a shared decoder. Alignment losses are used to promote consistency across the representations learned by each encoder. The OSDT encoder introduces strong inductive biases that are particularly well suited to tabular data, capturing sharp, high-frequency patterns and conditional feature interactions. Through joint training, it shapes the NN encoder's 's representations, encouraging more structured and robust encoding. At inference time, only the NN encoder is used, preserving the flexibility and compatibility of neural networks with downstream tasks. By combining these complementary behaviors during training, the model learns a more effective latent space for few- shot classification without relying predefined augmentation schemes.
In addition, we incorporate separate sample-specific stochastic gating networks for each encoder, which are trained jointly with the autoencoder using the same reconstruction objective. These gat- ing networks function as dynamic, sample-dependent feature selectors, serving as model-specific data transformations. The gating mechanism selectively filters out less relevant variation while pre- serving the information necessary for accurate reconstruction, resulting in input transformations that preserve semantic structure while tailoring the feature space to the strengths of each encoder. Rather than distorting the data through fixed augmentations, the gating networks act as learnable, sample- specific filters that suppress irrelevant or noisy variation. This process helps the neural encoder overcome its bias toward smooth, low-frequency patterns, enabling both encoders to capture the signal more effectively in low-label regimes.
Our main contributions are as follows:
(1) We introduce TANDEM, a hybrid self-supervised autoencoder that combines a neural encoder, an oblivious soft decision tree encoder, shared decoder, and sample-specific stochastic gating networks, enabling the learning of complementary representations suited to tabular data.
(2) We demonstrate that the representations learned by the neural encoder enable strong performance for both classification and regression tasks under low-label conditions, surpassing established deep learning and tree-based baselines.
(3) We conduct extensive experiments across a diverse suite of tabular datasets and systematically vary the number of labeled samples (from 50 1000 per dataset), establishing the robustness of TANDEM in range of low-label regimes.
(4) We provide both qualitative spectral analysis and quantitative comparison of gating activations, revealing how the two encoders capture distinct and complementary inductive biases.
2 Related Work
Regression and Classification on Tabular Data: Tree-Based and Deep Learning Models Tab- ular data remains central in real-world applications such as healthcare, finance, and recommendation systems. Classical approaches, including linear regression, logistic regression, and decision trees, have long been used due their simplicity and interpretability. Among modern methods, ensemble- based models like GBDT, XGBoost, and CatBoost consistently dominate benchmarks, due to their ability to model nonlinear feature interactions and handle heterogeneous inputs with minimal tun- ing [6] 5 14] Motivated by the successes of deep learning in vision and language, several works have adapted neural architectures to tabular data. DeepTLF aligns numerical and categorical fea- tures 44. TabM simulates ensembles within compact MLPs Z7 and TabPFN frames classification as probabilistic inference via synthetic pretraining [10]. SAINT and FT-Transformer apply attention to tabular structure [19 g. while LSPIN introduces locally sparse neural networks with sample- specific masks (24. Other lines of work incorporate tree-based structures into differentiable mod-
els. Soft Decision Trees (SDTs) 27 use smooth gating functions for gradient-based optimization.

---

Output
Gating Net ghr
8(7)
OSDT Encoder 2T
Decoder
(2)
X
Lus
Inchuctive Blases
Jepar
Gating Net
X(N
Neural Encoder x
Decoder
XNN
Inference
MLP Classifier
Figure 1: Overview of the TANDEM architecture. Input X is augmented via two distinct stochas- tic gating networks, producing separate masked views for a neural encoder and an OSDT encoder, each of which is illustrated to the right to reflect their respective inductive biases. TANDEM is trained using reconstruction loss, alignment loss, and latent representation similarity (LRS) loss. During inference, only the neural encoder and the MIP classifier are used to predict the output label ú. based on the view gated by the neural encoder's gating net.
Oblivious Soft Decision Trees (OSDTs) [33 impose depth-wise consistency in splits, improving interpretability and stability. These models, however, rely solely on labeled data.
SeperSupe Learning in Tabular Data Several self eef-suuervised methods apply reconstruc- tion objectives to tabular data. SubTab reconstructs full inputs from masked subsets 22T) VIME poruption-prediition adds masking and noise with an auxiliary corruption prediction task [25]. and SCARF aligns clean and corrupted views via contrastive learning or handcrafted views, which may not generalize across datasets. In contrast, our approach applies learnable, sample-specific transformations tailored the input distribution. Tree-based autoen- coders have also been proposed, using differentiable trees as both encoder and decoder (26). While these models reduce dimensionality and capture hierarchy, they are constrained by tree inductive biases. Unlike ours, they lack the flexibility and generalization of neural networks. TANDEM addresses this by combining tree-based and neural encoders to capture complementary inductive biases.
[2]. These methods rely on fixed corruption schemes
Unsupervised Feature Selection for Tabular Data Unsupervised feature selection identifies in- formative features without requiring labels, often for tasks such as reconstruction or clustering. Reg- ularized autoeneoders learn global feature subsets for reconstruction [17]. More recent approaches use stochastic gating networks [20], which learn per-sample, input-dependent soft masks for context- aware selection. While prior work uses gating for sparsity or interpretability, we reinterpret them as a learnable augmentation mechanism. In TANDEM, gating networks are trained jointly with cross-reconstruccton: one applied globally to the neural encoder input, and others are hierar- chically applied across OSDT layers. This enables model-specific and sample-specis views that suppress nuisance features and emphasize the shared, informative structure for reconstruction and downstream learning.
3 Method
We address semi- and self-supervised representation learning for tabular data with searce labels. Let the unlabeled pool be Dunlah = ) C CRD and the labeled set Dua = ((Im) Dm ) M.M. where M & N. Our goal to learn useful embeddings from Dunlab that improve downstream classification or regression on Djab

---

Model. TANDEM is a hybrid masking autoencoder with complementary encoders and a shared decoder. Each input x R first passed through a sample-specific gating network (STG) that outputs feature mask g(x) E [0, 1 The masked view ã = à g()) is fed in parallel to: (i) a fully connected neural encoder producing NN E R and (ii) an ensemble of Oblivious Soft Decision Trees (OSDTs) producing 2OSDD' E Rt A shared decoder h reconstructs to &NN h(=AN) and jOSDT h(=0SDT), both in R.D. Over batch, this yields zNN, zoSDT Bxx and X XOSDT BBXD
Training and use. Pretraining is performed on Dunlab using the reakkstrrccon objective (with auxiliary consistency terms defined later). After pretraining, downstream heads are trained on Dlab- At inference, only the neural encoder and lightweight predictor are used. The training objective combines three components: reconstruction loss for each autoencoder, an alignment loss between their reconstructions, and similarity loss between the latent representations. Together, these losses regularize the model and promote consistency across encoders with different inductive biases. This design enables the neural encoder to leverage the structured, high-frequency inductive bias of the OSDT encoder, as further demonstrated in our spectral analysis
3.1 Oblivious Soft Decision Tree Encoder
The OSDT encoder consists of an ensemble of shallow, differentiable binary decision trees 27 with fixed depth L. Each tree follows the oblivious decision tree structure, where all internal nodes at the same level share single projection vector. This constraint ensures a consistent, hierarchical partitioning of the input space, thereby reducing variability in decision logic across paths. At each tree level & E {1...., L), a learned projection vector Wy E R. used to compute a soft split score from the (potentially gated) input TR. using the equation 8 (2) (we, Tt, where тę R is sigmoid to produce left/right
a learned threshold. The score passed through temperature-scale
routing probabilities: of()) (z) o = (z)
Each of the 2 leaf nodes corresponds a binary code
E {0, 1)4, and the probability of reaching given leaf computed as:
Pteaf (x) II [ot (x)] {=1
[o]
Each tree produces soft distribution over its 24 leaves, representing the probability of reaching each path. These leaf probabilities are concatenated to form the tree's latent output, denoted by (OSDT(2) E I8²". The final representation of the encoder is obtained by averaging these vectors across all 7' trees:
OSDT (x)
1 POSDT T
Σ Jt
(x) E R
This vector OSDT serves the latent representation of the tree encoder and is passed to the shared decoder. For further details on oblivious differentiable tree-based encoders, we refer readers to the NODE paper 113.
3.2 Training Objective
TANDEM's 's training objective is designed to align the two encoder streams while preserving their in- dividual inductive biases. It combines three loss components: a reconstruction loss for each encoder, alignment loss between the reconstructions, and term measuring the similarity of the latent rep- resentations. The reconstruction loss encourages each encoder to preserve semantic structure from its gated input, ensuring that both the neural and tree-based views remain informative:
N
Lyoonn
Σ m==
(1mm
5. 2OSD
I|
NN (13
N
where Im the input vector of the m-th sample, and ANN OOST are the reconstructed outputs of the neural and OSDT encoders, respectively. To promote consistency between the outputs of the two

---

encoders, we apply an alignment loss between their reconstructions:
Calign
NN
Finally, we enforce agreement in the latent space by minimizing the average cosine distance between the latent vectors produced by each encoder:
LRS
1 N
Σ
mm
.OSDT)
JNNNN
122
OSDT
Fm >NNN and ,0OSDT are the latent vectors of the m-th sample, produced by the neural and OSDT encoders, respectively. The combined objective balances encoder-specific reconstruction with cross- view consistency, encouraging complementary yet compatible representations.
where zm
3.3 Stochastic Gating Network as Sample-Level Masking
We implement a sample-specific gating mechanism that selects input features through stochastic, differentiable masks. Given an input RDD neural gating network fo(x) produces a parameter vector u(z) E R.D. A gating vector g(x) E [0, 1]_ is then sampled using a clipped Gaussian
perturbation:
g(x)) max(0, m1, + plxr +)),
<~N(000²).
We a 0.5 throughout training, as suggested values of ga(r) toward binary decisions, while preserving gradient flow.
23
The injected noise encourages the
i = x @ g(z).
In the neural encoder, single gating network fasw computes global mask. In contrast, the OSDT encoder uses a distinct gating network at each tree level E {1...., L}. producing a level-specific mask OSDT (I) E [0, 1). This mask is applied before computing the soft split score:
Te x O OSDT (x), 8x(2) = (wy, īz) TP-
This structure allows the OSDT encoder to learn different feature selections at different depths, supporting hierarchical and progressively refined decisions. Although the gating networks are pa- rameterized and trained independently within each encoder, they are jointly guided via the shared decoder and alignment objectives. This coordination implicitly aligns the gating behavior across encoders, while allowing each to exploit its architectural inductive bias.
4 Experiments and Results
We evaluate whether combining neural and tree-based encoders with sample-specific gating mech- anism improves performance on both classification and regression tasks in tabular datasets, partieu- larly when labeled examples scarce. We aim to assess the utility of self-supervised pretraining for generating effective representations under such low-supervision settings.
4.1 Dataset Selection and Preprocessing
Our experiments are conducted in the low-label regime where labeled budgets range from 50 to 1000 samples per dataset, enabling consistent evaluation under limited supervision. We used all relevant classification datasets from OpenML analyzed in widely cited studies on the limitations of deep learning for tabular data 99 18
We retained only datasets with at least 2,500 samples per class. For each class, 2,000 samples were allocated for self-supervised pretraining, and up to 1. 1,000 labeled samples in total (across classes) were reserved for downstream evaluation. This filtering resulted in 19 classification datasets. For regression, we extracted 13 datasets that satisfy the same filtering as in the classification benchmark from the same OpenML sources referenced above. All datasets were preprocessed using fixed pipeline: categorical features were one-hot encoded, numerical features were min-max normalized to [0, 1], and missing values were imputed with the column-wiss mean.
5

---

2000 for 8 1,...
02
(x m3)
(a(nn 2 3q ra
Pa(wa 3a
rar
ea(or]
ma r3?
Poon
Phin
Pain
p011
pino
Piot
Proo
2i11
Roas
Roo1
Rozo
Rost
Ryop
Riot
Ro30
R111
2000
go?
,.). L.). the input is gated by dedicated gating network ge, producing masked vector ă; 3 ge- This masked input is projected by a learnable vector wy and compared against a threshold тr, producing a soft binary decision. Probabilities propagate through the tree to define Pb. the soft path probability to leaf Rb. The final output of the encoder the concatenation of all leaf probabilities, ZOSDT R²² serving as a disentangled latent representation.
Figure 2: Architecture the OSDT encoder in TANDEM. each depth (1 (1....,
4.2 Training and Evaluation Setup
Pretraining was run for 100 epochs with batch size of 128, and this configuration was held constant across all experiments to ensure fair and consistent comparison. We used RMSprop as the optimizer both pretraining and fine-tuning across models. The training objective combined reconstruc- tion, output alignment, and latent similarity losses, described in Section 3 Hyperparameters, including learning rate, encoder depth, and weight decay, were selected using Optuna over trials based on the validation loss. Complete optimization, including computational details and parameter ranges, provided in the appendix.
For downstream evaluation, single-layer MLP classifier or regressor was trained on the labeled subset using the neural encoder. The encoder was frozen for the first 25 epochs, then fine-tuned for an additional 25 epochs at reduced learning rate. Early stopping was based on validation accuracy or MSE, appropriate. present, the gating network was kept frozen during fine-tuning and used as per-sample feature selector; it consisted of two-layer MLP with tanh activations and a hard- sigmoid output that produced binary masks.
We evaluate two groups of baselines under identical data budgets. Supervised-only methods are trained solely on the labeled subset and include multinomial logistic regression (classification), lin- car regression (regression). a standard MLP, XGBoost (5), CatBoost (14). TabM [7]. DeepTLF [4), and TabNet (supervised) (T] Self-supervised fine-tuning methods are pretrained on the same un- labeled pool we use (2,000 samples class) and then fine-tuned on the labeled subset; this group includes VIME (25). SCARF (2). SubTab 21 and TabNet with self self-supervised pretraining [I]. Following the authors' guidance, TabPFN 10 is used only for classification. For TabNet, we re- port the stronger of supervised and self-supervised variants for each task. Across all methods, hyperparameters are tuned using Optuna (50 trials), and model selection is based on validation per- formance (accuracy for classification and MSE for regression). For component analysis, we report ablations of our approach: (1) SS-AE, (2) SS-AE gating, (3) SS-AE with an OSDT encoder and

---

Table 1: Comparison across models on classification datasets (400 labeled samples). Best results
row are in bold.
Dataset
MeggReg
Tab
DeepTLF
TabM
TabPFN
XGBoost
CatBoost
MLP
TANDEM
CP MT
0.4927
0.6000
0.5438
0.5827
0.5998
0,5822
0.5401
0,5792
0.6779
0.7521
0.6781
0.6757
0.7720
0.7868
0.7929
0.7798
OG
0.6228
0.5736
0.3712
0.6228
0.6514
0.6136
0.6363
0.6321
0.6870
PW
0.9373
0.5906
0.9281
0.9384
0.9477
0.9353
0.9327
0.9315
0.9618
AD
0.7992
0.6583
0.7645
0.8115
0.8199
0.7875
0.8019
0.8006
0.8200
ALB
0.6065
0.5708
0.5512
0.6196
0.6494
0.6096
0.6354
0.5929
0.7038
BM
0,8325
0,5458
0.6306
0.8132
0.8241
0.7991
0,8171
0.7913
0,8233
CO
0.4624
0.4618
0.4881
0.5049
0,5485
0.5326
0.4975
0.4963
0,5491
0.6169
0.5458
0.6302
0.6451
0.6690
EL
0.6617
0.5391
0.6424
0.6610
0.7723
0.7668
0.7654
0.7525
0.6940
HE
0.4593
0.3779
0.3844
0.4811
0.4590
0.4853
0.4448
0,5462
0,5454
0.4984
0.4965
0.5532
0.6499
0.6096
0.6069
0.6035
0.6459
JA
0.5105
0,4930
0.4649
0.5743
0,5986
0,5409
0.5413
0,5417
0.5660
NU
0.4833
0.5016
0.4932
0.5221
0,4333
0.5308
0,5231
0.5517
0.6545
RS
0.6992
0,5063
0.6590
0.7886
0.7554
0.7057
0.7328
0.7243
0.7576
VO
0.4624
0.4389
0.3966
0.5082
0.4736
0,5400
0,5220
POL
0,8515
0.7511
0,5000
0.7480
0,9520
0.9425
0,9330
0.9321
0,9538
CA
0,8400
0.6667
0.6930
0.8107
0.8703
0,8416
0.8400
0,8330
0.8921
0.4862
0.3655
0.3753
0.5058
0.5811
0,5461
0,5400
0.5245
0.5928
Mean Accuracy
0.6380
0,5454
0,5626
0.6532
0,7012
0.6706
0.6731
0.6721
0.7124
Mean Rank
6.16
8.00
8.05
4.84
2.56
4.47
4.16
4.84
1.58
"TabPEN supports maximum of 10 classes and could not be evaluated on the helena datasct (13 classes); TabPEN's mcan/rank excludo that
dataset.
Fraction of daticett
Marphng
Soleet
5544
Figure 3: Classification accuracy distribution across models. Boxplot across baseline models and TANDEM: red lines denote the mean and black lines denote the median.
Figure 4: Classification Dolan-Moré pro- files. Model accuracy relative to the per-dataset best: higher curves indicate stronger perfor- mance across datasets.
neural decoder, (4) TANDEM without gating, and (5) TANDEM without the latent similarity and alignment losses; all ablations share the same decoder and architectural settings.
4.3 Experimental Setup
Each experiment was repeated 100 times with varying seeds and splits. We report mean accuracy (classification) or mean MSE (regression) per dataset, along with aggregated mean accuracy, mean MSE, and mean rank across all datasets. Additional statistics, including standard deviations and further details on datasets, are provided in the appendix.
4.4 Benchmarks (Classification Regression)
Performance against baselines at 400 labels. Table | and Figure (classification, 400 labels) show that TANDEM achieves the highest mean accuracy and the best average rank, with TabPFN as the closest competitor. For regression at the same label budget, Table 2 Figure innndicaee that TANDEM attains the lowest mean MSE and the best mean rank, with XGBoost as the nearest com- petitor. In both tasks, TANDEM is competitive or best on most datasets and delivers the strongest overall results; the Dolan-Moré curves [16] (Figures datasets.
66 further underscore robustness across

---

Table Comparison across models on regression datasets (400 labeled samples). Best results per
IOW are in bold.
Dataset
CatBoost
LogReg
MLP
SCARF
SubTab
VIME
TabM
TabNet
XGBoost
TANDEM
BSD
0.1506
0.6088
0.5395
0.7377
0.5778
0.5699
0,4922
0.7634
0.1742
0.0722
RH
0.0291
0.0020
0.0021
0.0382
0.0047
0.0347
0.0012
0.0703
0.0302
0.0011
MBGM
0.5206
2,0242
0.6508
0.7517
0.6400
0.7208
0.6666
1.0833
0.5612
0.3354
SGEM
0.0340
0.0041
0.0017
0.00388
0.0113
0.1848
0.0032
0.0231
0.0171
0.0025
0.0329
0.5474
0.0515
0.1331
0.3336
0.7107
0.0487
0.2463
0.0264
0.0297
0.6565
0.8731
0.8931
0.8461
0.8874
0.9215
0.8409
0.9067
0.6986
1.0057
CO
0.7047
0.8370
0.7711
0. 0.8030
0.7869
0.8152
0.8055
0.9399
0.7395
0,7334
0.0889
0.2330
0.0531
0.1252
0.1
0,1.54
0.0475
0.2908
0.0675
0.0498
0,46
0.5115
0.2290
0.2244
0.2331
0.1732
0.1549
0.2013
0.1549
0,1835
FY
0.4987
0.5237
0.5533
0.6110
0.5825
0.5591
0.5577
0.6253
0.4805
0.4143
HS
0.3761
0,3960
0.2664
0.3198
0.3519
0,4175
0.5212
0.3923
0.2950
0.1436
VS
0.0015
0.1682
0.0013
0.0094
0.0086
0.3711
0.0030
0.0483
0.0068
0.0024
1,0557
2.2639
1,0278
L,0804
1.1325
1.1236
1.0419
1,2104
L,1751
1,2300
Mean MSE
0.3318
0.69188
0.4354
0.5174
0,3405
0,3234
Mean Rank
4.00
8.38
4,38
7.23
7.08
8.46
4.85
9.38
4.15
3.38
Fraction of daratets: IMSE hait: beet)
1.0
0:4
02
3.75
2.00
3.25
2.50
2.35
3.00
Figure 5: Regression MSE distribution across models. Boxplot across bascline models and TANDEM; red lines denote the mean and black lines denote the median.
Figure 6: Regression Dolan-Moré profiles. Model MSE relative to the per-dataset best; higher curves indicate stronger performance across datasets.
Ablation studies (shared design). Tables (classification) and (regression) assess matched ab- lations under the 400-label setting. Across both tasks, removing gating omitting either encoder degrades performance, whereas full TANDEM, combining the neural and tree encoders with gating, consistently yields the strongest results.
Performance across label budgets (50-1000 labels). Figure 7 mean performance as label budgets increase from 50 to 1000 in both classification (left) and regression (right). Across both tasks, TANDEM is consistently either the top method or competitive with the best throughout the low-label regime (50-1000 labels).
5 Understanding Complementary Gating Behaviors via Frequency Decomposition
To better understand the role of sample-specific gating networks, we analyze how gating affects the spectral composition of the input within a single class. For each dataset, identify the class in which TANDEM achieves the highest classification accuracy and restrict our analysis to samples from that class. We then compute the unnormalized discrete Fourier transform (NUDFT) over the 50 most variant features, comparing the resulting frequency distributions across different gated and ungated versions of the input.
To analyze the effect of gating on the spectral composition the original input to its transformed versions under different gating mechanisms. Specifically, we @ NN (xr) produced by the stochastic gating network connected to the neural encoder (as defined in Section 3.3 and the input gated by the tree-based mechanism
input, we compare spectrum of
evaluate the gated input NNN

---

Table 3: Ablation study comparing TANDEM and its variants on classification datasets.
Datasel
SS-AE
SS-A Gating
OSDT AE Gating
TANDEM (no gule) TANDEM (no LRS Aligumentt TANDEM
CP
0.65U5
0.6602
0.6321
0.6740
0.6866
0.6779
MT
0.7996
0.8096
0.7639
0.8117
0.8173
0.8180
OG
0.6500
0.6740
0.6601
0.6689
0.6815
0.6870
PW
0,9353
0,9353
0.8259
0.9664
0.8673
0,9618
08111
0.8200
ALB
0.6497
0.6797
0.6652
0.6763
0.6993
0.7038
BM
0.814]
0.8
0.552
0.8
088198
0,8233
CO
0,5007
0.5207
0.5373
0.5220
0.5475
0.549
0.7331
EL
0.7429
0.7429
0.6149
0.7134
0.7078
0.6940
HE
0.5250
0.5350
0.4416
0.5315
0.5349
0.5462
HI
0.6179
0.6279
0.6621
0.6300
0.6529
0.6459
JA
0,5168
0.5468
0.5012
0.5529
0.5365
0.5660
0.6545
RS
0,7016
0,7016
0.7252
0.7311
0.7200
0.7576
VO
0,5154
0.5254
0.4334
0.4441
0.4840
0,5220
POL.
0,9325
0.9381
0.9102
0.9222
0.9420
0,9538
0.8721
BY 0.5148
0.5296
0.4901
0.5124
0.5180
0.5928
Mean Accuracy
0.6815
0.6941
0,6600
0.6966
0.6971
0,7124
2.79
1.74
Table 4: Ablation study comparing TANDEM and its variants regression datasets.
Dataset
SS-AE SS-AE Gating OSDT AE + Gating
TANDEM (mo gate)
TANDEM (no LRS + Alignment) TANDEM
0.7824
0.0722
BH
0.0026
0.0331
0. 1088
0.0462
0.0367
0.0011
MBGM
0.3325
0,7409
0,9492
0,7630
0.7369
0.3354
SGEM
0.0044
0.3291
0.1756
0,0025
AS
0.0270
0,9236
0,9438
0.4520
0.7320
0.0297
BF
I L0890
0.9622
0.9966
0.9010
0.9286
1.0057
CO
0.7849
018479
1,0007
0.8828
0.9214
0.7334
0.2010
EL
0.1865
0.2380
0.4752
0,2631
0.2971
0.1835
EY
0.4944
0,7872
1 1,4029
0.6590
0.6400
0,4143
VS
0.0023
0.6884
0.9759
0.5521
0.1157
0.0024
YP 1.3912
1,2134
1.1151
1.6014
1,3083
1.2300
Mean
0.6551
0.5832
Mean Rank
2.69
4.13
5.50
4.00
3.73
1.92
in the OSDT encoder. Since the OSDT encoder employs distinct gating network Dt. depth level & E {1....,Z, L} for each tree & E (1...., ,T,, compute the aggregated gating signal by averaging all gating masks across all levels and trees:
OSDT (x)) at each
OSDT (x)
OSDT g8,
(x),
T L
t=1
and apply it elementwise obtain OOST x @ g OSDT (x)). In addition, we analyze the spectrum of WNN produced by SS-AE with the same neural gating network as used in TANDEM. All spectral distributions are shown on a shared axis for comparison against the original input.
Across datasets, gating mechanisms function as adaptive frequency filters, suppressing high- frequency variation to varying degrees. We find that the neural gating transformation JNN x Q (x) in TANDEM consistently reduces high-frequency components more strongly than the tree- based transformation 2OSDT OSDT (z). This observation aligns with prior work the spectral bias of neural networks toward low-frequency representations (15), which can be limiting in tabular settings that often rely sharper, fighffrequency decision boundaries [
Notably, the neural and tree-based gating transformations within TANDEM, AN and OOSDT ex- hibit the strongest spectral contrast. The neural gating NN acts as a strong smoother, while the aggregated tree-based gating OSDT retains more high-frequency content. This reflects their com- plementary inductive biases, with each capturing different aspects of the input structure important for tabular decision boundaries. It is this consistent difference between g and g OSDI in TANDEM that contributes to the formation of complementary representations in the shared latent space.

---

Classification
Regression
Cchoel
012
032
Samples
306
60
20
Samplie
200
Figure 7: Learning curves across label budgets. Mean accuracy (left, classification) and mean
MSE (right, regression) vs. label budget.
covertype
PhishingWebsites
10
Poyio
Poy(k)
10
10
gated 8t (TANDEM)
galed 9 ITANDEM) gated by goser (TANDEM)
gated by gren ITANDEM)
(55-AE)
10
10
10a
10-
10
10a
10*
[k]
Figure 8: Frequency spectra of gated inputs. Comparison across datasets for each encoder- input using NUDFT.
Conclusion
We introduced TANDEM, a hybrid self-supervised framework designed for tabular data. This framework combines neural and tree-based encoders with sample-specific gating. TANDEM demonstrates impressive performance across various benchmarks in low-label scenarios, outper- forming both neural and tree-based baselines.
Ablation and spectral analyses indicate that the two encoders offer complementary inductive biases, favoring smooth patterns and high-frequency patterns, respectively. This design enables TANDEM to generalize effectively from limited amount of labeled data. The gating networks further enhance this by creating tailored, comple-specifie input views that leverage the strengths of each encoder.
While our study has limitations, such as small sample sizes and a focus on low-label settings, these constraints reflect woalwwrldd challenges in tabular data and facilitate meaningful comparisons with specialized methods, such TabPFN, which is also aimed few-shot classification. Despite these limitations, our results demonstrate that small-scale evaluations can offer valuable insights into model behavior and inform the development of scalable solutions.
For future work, we believe that the architectural principles behind TANDEM hybrid encoders, sample-specific gating, and model-based augmentation can be incorporated into transformer-bas tabular models. This integration could help create more robust and interpretable tabular foundation models, in line with the vision presented in the recent position paper [22].
Acknowledgment
OL was supported by the MOST grant No. 0007341.
10

---

References
[1] Sercan Ö. Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceed- ings of the AAAI Conference Artificial Intelligence, volume 35, pages 6679-66669 6687, 2021.
[2] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption, 2022.
[3] Ege Beyazit, Jonathan Kozaczuk, Bo Li, Vanessa Wallace, and Bilal Fadlallah. An inductive bias for tabular deep learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 43108-43135. Curran Associates, Inc., 2023.
[4] Vadim Borisov, Klaus Broelemann, Enkelejda Kasneci, and Gjergji Kasneci. Deeptif: robust deep neural networks for heterogeneous tabular data. International Journal of Data Science and Analytics, 16:85-100, 2023.
[5] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16), pages 785-794. ACM, 2016.
[6] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[7] Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learn- ing with parameter--fffiient ensembling, 2025.
[8] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data, 2023.
(9] Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still out- perform deep learning on tabular data? In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022.
[10] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A trans- former that solves small tabular classification problems in second. In Proceedings of the 11th International Conference on Learning Representations (ICLR 2023), 2023.
[1I] Soham Jana, Henry Li, Yutaro Yamada, and Ofir Lindenbaum. Support recovery with projected stochastic gates: Theory and application for linear models. Signal Processing. 213:109193, 2023.
[12] Jungkyu Kim, Kibok Lee, and Taeyoung Park. To predict or not to predict? proportionally masked autoencoders for tabular data imputation, 2024.
[13] Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural Oblivious Decision Ensembles for deep learning on tabular data. arXiv preprint 1909v 12, 2019.
[14] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and An- drey Gulin. Catboost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems 3/ (NeurIPS 201 pages 6649, 2018.
[15] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICMI. 2019), pages 5301-5310, 2019.
[16] Amit Rozner, Barak Battash, Henry Li, Lior Wolf, and Ofir Lindenbaum. Anomaly detection with variance stabilized density estimation, 2024.
[17] Uri Shaham, Ofir Lindenbaum, Jonathan Svirsky, and Yuval Kluger. Deep unsupervised feature selection by discarding nuisance and correlated features. Neural Networks, 152:34-43, 2022.
[18] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning not all you need. Infor- mation Fusion, 81:84-90, 2022.

---

[19] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Gold- stein. SAINT: Improved neural networks for tabular data via row attention and contrastive 2021.
pre-training. arXiv preprint
[20] Jonathan Svirsky and Ofir Lindenbaum. Interpretable deep clustering for tabular data. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024). pages 47314-47333 2024.
[21] Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabu- lar data for self-supervised representation learning. Advances in Neural Information Process- ing Systems, 34, 2021.
[22] Boris Van Breugel and Mihaela Van Der Schaar. Why tabular foundation models should be research priority. arXiv preprint arXIv:2405.01147,2222
[23] Yutaro Yamada, Ofr Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates, 2020.
[24] Junchen Yang, Ofir Lindenbaum, and Yuval Kluger. Locally sparse neural networks for tabular biomedical data. In Proceedings of the 39th International Conference Machine Learning (ICML 2022), 2022.
[25] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self- and semi-supervised learning tabular domain. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11033-11043. Curran Associates, Inc., 2020.
[26] Ozan Irsoy and Ethem Alpaydın. Autoencoder trees, 2014.
[27] Ozan Irsoy, Olcay Taner Yıldız, and Ethem Alpaydın. Soft decision trees. In Proceedings of the 21st International Conference Pattern Recognition (ICPR2012), pages 1819-1822, 2012.
12

---

Checklist
1. Did you state the main claims of the paper in the abstract and introduction? [Yes]
2. Did you discuss the limitations of your work? [Yes] We address this point in the conclusion section.
3. Did you state the full of assumptions of all theoretical results? [NA] The paper does not contain theoretical results.
4. Did you include complete proofs of all theoretical results? [NA]
5. If you ran experiments, did you include the code, data, and instructions needed to reproduce the main experimental results? [Yes] We include code to extract datasets and reproduce results as part of the supplementary material.
6. Did you include the full details of your experimental setup, including hyperparameters and training procedures? [Yes] Training setup is in Experiments and Results and hyperparam- eters are in Appendix
7. Did you report error bars and describe the variability of your results? [Yes] Error bars (stan- dard deviation over multiple runs) are reported in all performance tables. See Appendix B.
8. Did you provide information about compute resources used? [Yes] Details are provided in the appendix C.
9. Did you read the NeurIPS Code of Ethics and ensure your paper conforms to it? [Yes]
10. Did you discuss potential negative societal impacts of your work? [NA] Our work is purely
methodological and does not involve real-world applications at this stage.
11. Did you describe safeguards for responsible release of high-risk models? [NA] Our models are not considered high-risk or dual-use.
12. If you used existing assets, did you respect their licenses and cite them appropriately? [Yes] All datasets and libraries are properly cited, including license details in Appendix B.
13. Did you document any new assets (e.g., code or datasets) and include details on licensing and consent? [Yes] Code and dataset handling scripts included as part of the supple- mentary material
14. If you used crowdsoureing or conducted research with human subjects, did you include participant instructions and compensation details? [NA] No human participants or crowd- sourced data were used.
15. Did you obtain IRB approval (if required)? [NA] No human subjects were involved.
16. If you used LLMs in a non-trivial way, did you disclose their usage? [NA] No large lan-
guage models were used in non-trivial manner.
13

---

Appendix
A Results (400 labeled samples)
This section summarizes performance at fixed budget of 400 labeled samples: baseline classifica- tion results (Table A.T). classification ablations (Table A.2), and classification significance counts (Table A.3); followed by bascline regression results (Table A.4). regression ablations (Table A.5 and regression significance counts (Table A.6
A.1
Classification
Table A.I: Baseline classification accuracy at 400 labeled samples (mean + std).
Techu
DeyT1
34.75
07011 2 1185
0431 8001
14250
6813 21 21:00
24025
6.70 21 540
6776 2.0015
0.613 2 3013
5310
013100 TUAO
0180 186
03812 83114
3336 1133
E400 01-00
6108
0.3377 61.384
8410 4124
6.58 5.700
0400
(S71)
6.752
6479
0500 3010
1.50
5400
307
0230
4351
5.006- 4
5120 311
1203
02801 313.6
5110 11306
5100 11003
E.310 5.000
D1EER (1880
018772 88M1
038022 14T13
8005 241
2/4
0310122 14100
1979 (144)
4019 211
0.18
2.54
411%
5.5
Table A.2: Ablations classification accuracy at 400 labeled samples (mean + std).
Datascl
S9-AR
SS-AF. Gatnn
OSIT AF Garing
TANDEM (su prlc)
TANDEM (sul Alignment
TANDEM
CPP
0.300 +: 0.0818
0.6602
0.0918
0,6921 1: 0/99
06740 =: 0.034
0.6740 = 0.3344
0.6779 = 0.0325
MT
0.7766 :: 0.0255
0.8998
0.0255
0.7630
2 0.0240
03117 = 0.0214
0,5014 2 0.0266
0.8180
0.0224
DG
0.6440
+ 0.0120
0.640
0.0122
0.6631 0.0366
0.6659 = 0.0152
0.6671
0.0122
0.6670
0.0125
FW
0,0033 0.0131
0.3533
0.0131
0.8259 0.0310
0.9644 0.0129
0,9455
0.014]
0.9618
0.0145
AD
0.7755
0.0232
0.8055
0.0232
0,7658
0.0555
08111 0.0233
0,7950
0.0216
0.8200
0.0217
ALB
0.6497 + 0.0020
0.6797
0.0020
0,6652 0.0234
06763
0.224
0,6800 0.0277
0.7038
0.0263
BM
077411 + 0.0237
08141
0.0237
0,7552 0.0199
03128
00214
0,8059
0.0231
0.8233 0.0178
CO
0.9907
0,0170
852007
0.017U
0.5373 0.099
03220
000488
0.5104 2 0.0133
0.6491
0.0150
DC
00000
0.0264
0,0900
0.0264
0.6892 000233
0.7434
0.0254
0.118 0/006
0.3311 0,0211
13.
01122 0.0312
0,2429
0.0332
0.519
0/0044
07114
012532
030001 01276
0.800 0,040
HR
05000 0.0116
85150
0.01
0.4416 0.0016
03515 001.9
0.519% 00143
0.5462 0.01
0.0510
8.6779
0.0910
0.6821
0.0528
06:00 00799
= 0.0344
0.6499 0,093
05168
0.5012
0.5039-00000 00175
0.5541 - 0.0204
0.5860 0.0212
NU
0.5992 0.0842
8.002
0.0342
0.6140 0.0003
00318
= 0.6307
0.6545
RS
0.6766 0.0260
0
0.0250
0,7252
0.0274
07311 00252
0.7335 = 0.0270
0.7570
0.4954 0.0142
0.5254
0.0142
0.4334 0.0231
04741
0.026
0.4953 0.0140
0,5230 0.0138
0.0210
0.9102 0.0300
0.212 0.0055
0.0185
0.9538 0.0230
CA
0.0200 08853
03333
0,0190
0,9433
0.0270
08721
0.8921
0.0220
EY
0.0200
0.0200 0,9901
03124
0.0770
0.0828
0.0230
Mean Accuncc
0.6815
0.941
0,6000
0.9966
0.657
0.7124
Meon Kank
4.45
3.6l
4.71
242
2779
1.74
Table A.3: Number of datasets (out of 19; 18 for TabPFN) where TANDEM outperforms each baseline (classification). Parentheses indicate statistically significant wins (p < 0.05, 100 trials).
MLP
XGBoos
TabM
MLogReg
DeepTLF
Catooot
SS-AE
TOPPN
VIME
SCARF
SuhTis
TANDEM
(8(17)
(17)
(17)
(981
(9
(9
18(15)
(34)
16(17)
(19)
16(1
A.2 Regression
Dolan-Moré curves: Figure 4 in the main paper presents Dolan-Moré performance profiles, which show the proportion of datasets where each model achieves performance within factor 7 of the best-performing model. Unlike rank-based comparisons, these curves capture both accuracy and robustness, providing more complete view of model consistency across diverse tasks. Higher curves indicate models that maintain strong performance across a larger share of datasets, even if not ranked first.
14

---

Table A.4: Baseline regression MSE 400 labeled samples (mean + std).
Calleoe
Lagfts
SCA3T
VIME:
TahM
230
TANION
840
3,50
5400 0104
5.375
03203 0-82)
03774 6 0:375
E-002 4 200.0
1.3.4 032279
3130 00L1S
353W 2 1.0599
2/00 1.4112
63517 E3646
0640) 2 6011
0138 1 E3337
E1600 3 39150
1000
2.0155
32413 (0E5)
0.3564 22 0.0333
03007
6331
E323
à 03227
63167
0J0335
0333
LC442
3.338
19671 -00355
34805 L-0589
D4
L/G0 & 02212
0,38 6-0185
0000 -0088
0U223 1-100
01002 0.300
ELEGE 6 1-113
0,2415 3 0.0000
1,3.0 01033
218/3
ETV
2400
1.0000)
0 L.001
6.503 -0EE5
06639 03611
05825 3 03116
03881 1.0133
EA5T7 1 3007
14050
0000
348.5 0.0333
0:4143 21 0.0000
0.0055
234 g 11794
1.0078-2 01/0
11001 6300
1755 8236
1155 3 9600
1820 3 3094
171M 2 10/66
1115 2.09R
17001 115
Nor Jast
4.00
626
-36
125
725
1.46
4.4
3.8
4.5
3.56
Table A.5: Ablations regression MSE at 400 labeled (MSE mean + std).
Dunaet
Al Gating
Gating
TANDEM gate)
TANDEM
Alligmmen)
TANDEM
BSD
0.0734 = 0.0040
010773 # 0.0030
0.0744 #: 0.0037
0.0770 :: 0.0039
0.0758 #: 0.0035
0.0722 :: 0.00086
BH
(10012
0.0001
010012 # 01001
010011
0.0001
0.0012
0:000
0,0012 0.0001
0.0011 + -0.0001
MBGM
0.1689
0.084
0.3585
0.1199
0.3455
0.0173
0,3620
0.0181
0.3522 0.0176
0.3354
0.0168
SGEM
0,0025
0.0001
0.0027
0.0001
0.0001
0.0001
0,0026
0.0001
0.0025
0.0001
0.0016
0.0016
0.0155
0.0015
0,0312
0.0297 0.0015
11063
0.053
1.0751
0,033
1,0369
0.0515
1,0862
0.0543
1,0560 0.0528
1.0057 0.0503
08067
0040
07747
0.892
0.7554
00378
0,7921
0.0196
0.7701 1 0.0085
0.7334
0. 0.0367
DI
0.0853
0.0027
0,0833
0.0027
0.0813
0.0025
0,0538
0,0027
0,0523 1 0.0026
0,0498 0.0025
IL
0,2019 0.0101
0.19h2 1. 0.0096
0,1890
0.0095
0,1982 0:0099
0,1927 1 0.0096
0.1835 1 0.0092
EY
0.4557 0,0228
0.4433
010222
U.4267
0.0213
0,6474
010024
0.4350 0.0218
0.4143 + 0.0207
HS
(5000
0 0079
0.1537 : 0077
0.1479
0.0074
0,1551
0:0078
0.1508 0.0075
0.1436 : 0.0072
0.0001
0.0001
0.0001
0,0025
0.0024 0.0001
1,3530
0.0677
0.0688
0.0633
0,0664
1,2915
1.2300
Mean MSE
0.3622
0,6551
0,9207
0.3397
0,5832
0.3235
Mean Rank
2.69
4.35
3.50
4,00
3.73
1.92
B Extended spectral analysis
Extending the spectral analysis in the main paper (Section 6), we present additional visualizations across more datasets (Figure B. to further examine how the gating mechanisms in TANDEM shape the frequency content of the input. For each dataset, we focus on a single class and compute the unnormalized discrete Fourier transform (NUDFT) over the 50 most variant features. We compare the spectra of the original input æ; the neural-gated input N- = x @ g NN TANDEM and SS-AE with gating; and the tree-gated input OSDT @ g goSDT (æ), where goSDT is the average gating mask across all trees and depths.
(2)) as produced by both
These extended plots confirm the patterns observed in the main paper: neural gating acts as a strong low-pass filter, suppressing high-frequency components. In contrast, tree-based gating preserves more high-frequency variation.
C Gating activations and dataset profiling
To complement the spectral analysis (Section we present dataset profiling that motivates the choice of datasets used in the gating activation study, followed by summary diagnostics comparing the neural and tree gating behaviors. These diagnostics justify the focused per-dataset gating analysis reported in the Appendix.
C.1 Dataset profiling: categorical feature ratio
To better understand where TANDEM provides the largest gains, we grouped the benchmark datasets by their ratio of categorical features (i.e., the fraction of categorical input features). The breakdown in Table C.T was used to select datasets for the detailed gating analyses:
This characterization highlights that TANDEM performs exceptionally well on datasets with a mod- erate share of categorical features, while also showing strong results on fully numeric datasets whose continuous columns have low cardinality (i.e., behave like categorical inputs). The medium categor-
Table A.6: Number of datasets (out of 13) where TANDEM outperforms each baseline (regression). Values in parentheses indicate statistically significant wins (p < 0.05, 100 trials).
LogReg
MLP
VIME
SubTab
CatBoost
TabM
SCARF
XGBoost
SS-AE
TabNet
TANDEM
(111))
(1I)))
13 (10)
(11)
(2
11 (10)
10
(0)
9 (9)
12 (11)
15

---

109
10
Puu00
Paol0)
10-1
10
geteed 9t (TANDEM:
gated by g (TANDEM)
gated by goean (TANDEM)
X gated by griat (TANDEM))
gaited by 9tth (55-AE)
10-2
gatee by g (55-AE)
10
10
10a
10a
10
10
10a
10a
14
109
109
Poulk3
10
10-1
g (TANDEM;
goted by g (TANDEM;
geted gooat (TANDEM)
geled by gooo (TANDEM)
goted by 9 (55 AEI
goted by g (SS-AE)
10
10P
10
102
10
10
10
102
Figure B.1: Frequency spectra of gated inputs for the NN and OSDT encoders. Visualized datasets include OG, CP, VO, and CC. NN gating results in stronger suppression of high-frequency compo- nents compared to tree-based gating.
Category
Datasets (categorical ratio)
High (2 0.70)
PW (1 (1,00), BM (0.92), CO (0.85), AD (0.87)
Medium (0.30-0.69)
RS (0.69), ALB (0.39), BY (0.33), CC (0.33)
Low (< 0.30)
OG (0. 10), CP (0.27), VO (0.18), HL. (0.12), HI, HE, JA, NU, CA (0.00)
Table C.I: Dataset grouping by categorical feature ratio used select datasets for gating analyses.
ical group (RS, ALB, BY, CC), therefore, provides natural testbed for comparing gating behavior on categorical inputs.
C.2 Gating diagnosties (selected summaries)
Table C.2 reports interpretable summary statistics computed on the medium-categorical datasets. Definitions: BinActSim cosine similarity between binary activation vectors (i.e., {gate 0.5)); Corr is the Pearson correlation of gate values; VarRatio is the variance of the tree gate divided by Var(tree gate)
the variance of the neural gate, Var(ncural gate)* MeanActOSDT and MeanActNN are the mean gate activations for the tree and neural gating networks, respectively.
Table C.2: Gating similarity and activity statistics (selected datasets).
Dataset
BinActSim
Corr
VarRatio (OSDTINN)
MeanActOSDT
MeanActNN
RS
0.78
0.88
1.52
0.26
0.20
ALB
0.59
0.66
1.81
0.41
0.27
EY
0.33
0.34
1.61
0.30
CC
0.65
0.82
1.76
0.43
0.37
These statistics show that, on the selected medium-categorical datasets, the tree gating (OSDT) tends to be spikier (higher variance) and, on average, more active than the neural gating. At the same time, binary overlap and Pearson correlation indicate moderate agreement, suggesting that both encoders consider categorical features necessary.
16

---

C.3 Gating behavior on categorical features
For clarity, Table C.3 repeats the categorical-foussed comparison (used in the rebuttal) showing gate overlap. gate correlation, and the variance ratio (Tree/NN) restricted to categorical features only.
Table C.3: Comparison of gating behavior between the neural and tree encoders, evaluated only on the categorical features four medium-catee cetturrcateeoorical datasets.
Dataset
Gate Overlap
Gate Correlation
Gate Variance Ratio (Tree / /NN)
RS
0.78
0.88
1.52
ALB
0.59
0.66
1.81
EY
0.33
0.34
1.61
CC
0,65
0.82
1.76
D Dataset details
Tables D.I (classification) and D.2 (regression) summarize the datasets used in our experiments. For classification, we report the number of samples (N), features (F), and target classes (C). For regression, we report the number of samples (N) and features (F).
Dataset (Acronym)
#Samples (N)
#Features (F)
#Classes (C)
Click.predictionnsmaa
MagicTelescupe (MT)
19020
11
2
Otto Group Product Classification Classiication,Challnngg Challenge (OG)
16400
93
5
PhishingWebsites (PW)
9200
30
2
adult (AD)
9200
14
2
alberrtcategorical (ALB)
bank-marketing
45200
covertype (CO)
283300
6
Acfuull-ff-ceeit-erd---ii otw.:e/oorerreeeeiiiiiiooooooiaa (CC)
5200
23
2
eloctricity (EL)
19240
11
2
helena (HE)
36260
13
higgs (HI)
470080
2
jannis (JA)
83600
4
numerai28.6 (NU)
9200
119
2
road-safets caad-sefety.attegoriaal www.goricalaaaacoooooooooooo.cca (RS)
363240
67
2
volkret (VO)
14800
181
8
pol (PO)
15000
2
califomiaa (CA)
20634
eye_movemennss (EM)
10935
27
3
Table D.I: Dataset statisties (classification): number of samples (N), features (F), and target classes (C).
(Acronym)
#Samples
yprop_4_1 (YP)
7,331
251
analcatdata supreme (AS)
4,052
7
visualizing_sol (VS)
7,185
black.friday
102,093
diamonds (DI)
34,364
26
Mercedes Benz, Greener Manufaatrrn (MBGM)
4,209
563
Brazilian houses (BH)
8,416
18
Bike Sharing Demand (BSD)
12,428
PospplrewsPopullriyy (ONP)
25,787
house sales (HS)
14,688
21
SGEMM GPU kemel perfomaeee (SGEM)
146,960
17
electricity (EL.)
29.188
8
eyc_movements (EY)
8,562
27
covertype (CO)
350.608
54
Table D.2: Dataset statistics (regression): number samples (N) and features (F).
Note. For the regression benchmark, we do not require class labels; therefore, dataset selection required only minimum total of 3,000 (2,000 for pre-training and up to 1,000 for the downstream task) samples per dataset (not 2,500 samples per class as in the classification setting).
17

---

E
Compute cost
All timing measurements reported below were collected on an NVIDIA L4 GPU (cloud instance). For completeness, the local workstation used for development and other experiments is: Intel 17 CPU, 16GB RAM, NVIDIA RTX 3060 GPU GB).
Model
Mean pretraining time (s)
SS-AE
38.12
TANDEM
43.47
TabNet
50.22
SCARF
63.05
SubTab
264.89
VIME
309.76
Table E.I: Pretraining time per model (mean across datasets). Measured for 50 pretraining epochs with 2000 samples per label on an L4 GPU.
Model
Mean downstream time per batch (s)
XGBoost
MLP
0.08
TANDEM
0.08
DeepTFL
0.10
TabM
0.12
TahPFN
0.16
Table E.2: Downstream training time per batch (128 samples). Mcasurements taken on an L4 GPU.
Model architectures and training
The architectural components used in our experiments follow the design described in Sections 4.1 and 4.3 of the main paper.
Neural Encoder: 4-layer MLP with BatchNorm and Leaky ReLU activations. Hidden dimensions are chosen to match the embedding size dictated by the OSDT encoder.
OSDT Encoder: Ensemble of oblivious soft decision with fixed depth L; each tree outputs a soft assignment over 24 leaves. The mean-aggregated output defines the embed- ding dimension.
Neural Decoder:
Mirrors the architecture and dimensionality of the neural encoder.
Gating Network: 2-layer MLP with tanh activation and hard-sigmoid output; used for per-sample feature selection.
Fine-tuning MLP: A single-layer fully connected classifier trained on top of the encoder for downstream supervised evaluation.
G Hyperparameter tuning
We report mean pretraining time per model (Table E.1 batch (Table E.2
and mean downstream training time per
Table G.I
summarizes the hyperparameter search space used in Optuna-based tuning (50 trials per model, per dataset). Gating components, when applicable, were tuned separately.
H Optimization robustness
This section reports sensitivity results across key hyperparameters (Table H.1) and optimization cost curves as function of trial budget (Figure H.I
18

---

Table G.I: Hyperparameter search space for all models. Each model was optimized using Optuna with 50 trials. Gating components (when applicable) were tuned separately.
Model
Hyperparameter Search Space
Notes
TabNet
learning rale: [10 + 10 '
Preirained with TabNeePrettainer before supervised line tuning
type: sparsemax,
scheduler step size: [5-20], gamma: [0.8-0.95]
batch size: 1024, 2048
virtual hatch size: 64, 128, 256
epochs: [10-100). patience: [5-20]
MLP
learning rate: [10 10 (log scale)
4-layer MLP with ReLU activations
hidden [64-256]
batch size: 32, 64, 128
opochs: [10-50]
XGBoost
max depth: [3-10]
garmnn: [10 $ 1]
subsample, colsample bytree: [0.5,
reg alpha, reg lamhda: [10 $, 10
CatBoost
depth: [2-10]
verbose-0
learning [10-3, 0.3]
itcrations: [100-300]
Logistic Regression
[30 10] (log uniform)
nax, itter=0000
DFL
cstimators: [10-100]
Internal tree-based layers
max depth: [2-7]
dropout: [0.0-0.3]
layers: [1-3]
TabM
blocks: [2-4]
block: [64-256]
dropout: [0.0-0.3]
[8-61]]
learning rate: [10 10 (log)
TabPEN
n estimators: 4.8
Pretrained; limited tuning allowed
softmax temperature: [0.5, 1.0]
balance probabilities: Truc. Folse
average before softmax: Trae, False
SS-AE
learning rale: [10-1 10
Encoder uses powers-of-twe twe hidden sizes. Gating tuned sepa- rately.
weight decay: [1023
depth: [3-10]
TANDEM
learning [10 4, 10
Shared decoder. Gating network tuned separately.
weight decay: [10- 10
depih: [3 10
num trees: [2 16
gating optimizer, activation. leaming rate
Galing Network
hidden size: [32 128]
Used in SS AE and TANDEM for per sample musking.
learning rale: 110-1 10
weight decay: 110-5,100 10
19

---

H.1 Sensitivity analysis
In order assess model robustness, we vary key hyperparameters: alignment loss weight Aaligns learning rate, embedding dimension, number of trees, and gate temperature across five representa- tive datasets (CP, OG, PW, ALB, CC). As shown in Table H.I the best performance is maintained across a wide range of values, indicating that TANDEM is robust and not sensitive to narrow hyper- parameter settings.
Parameter
Value
PW
ALB
CC
Alignment weight (Aaig))
0.6702
0.6721
0.9352
0.6790
0.7165
0.6732
0.6789
0.9387
0.6781
0.7286
0,5
0.6775
0.6865
0,9613
0,7032
0.7326
0,9620
0.7231
10.0
0.6105
0.6230
0.8930
0.6211
0.6815
Learning rate (1r)
mean +: std
1e-4
0.6582
0.012
0.6380 1. 0.011
0.9521 1: 0.010
0.6912 1: 0.015
0.7201 1 0.013
5e-4
0.6705
0.013
0:012
0.009
+ 0.014
+ 0.7268
1e-3
0.6775
0.014
0.6560 0.013
0.9613
= 0.008
0.7032 + 0.013
0.7326 = 0.012
Se-3
0.6740
0.016
0.6472 0.017
0.9502
0.012
0.6890
+ 0.018
0.7190 + 0.015
1e-2
0.6601
0.018
0.6305
0.020
0.9310
0.020
0.6723 1. 0.021
0.7015 1 0.019
Embedding dimension (mean + std)
8
0.6681 0.012
0.6510 1: 0.014
0.9562
0.010
0.6950 1: 0.014
0.7280 1: 0.013
16
0.6724
0.013
0.6548
0.013
0.9591
: 0.009
0.6979 + 0.013
0.7302 + 0.012
0.6775
0.014
0,6560 0.013
0.7032
0.7326
64
0.6760 0.015
0.6550 0.015
0.9600 : 0.010
0.7020
+ 0.015
0.7310 + 0.014
Number of trees (# trees) mean + std
50
0.6680 0.013
0.6480 1: 0.014
0.9580 1 0.009
0.6950 1 0.013
0.7288 : 0.012
0.013
0.013
0.009
1
200
0.6776
0.014
0.6560 0.013
0.9613
= 0.008
0.7032 + 0.013
0.7326 = 0.012
500
0.6764
0.015
0.6558 0.015
0,9610 0.010
0.7025 + 0.015
0.7320 + 0.014
Gate temperature (softmax temperature)
0.1
0.6720 0.013
0.6380 0.014
0.9608
0.009
0.7001 + 0.013
0.7302 + 0.012
0.5
0.6765
0.013
0.6555
0.013
0.9612
0.008
0.7030 1 0.013
0.7320 1 0.012
1.0
0.6775
0.014
0.6560
0.013
0.9613 : 0.008
0.7032 1 0.013
0.7326 + 0.012
0.015
Table H.l: Concatenated sensitivity results for selected hyperparameters and datasets (CP, OG, PW,
ALB, CC). Values are classification accuracy; entries marked "mean + std" show the mean and standard deviation across runs.
H.2 Cost-based optimization (trial budget analysis)
To empirically assess optimization robustness,
ran Optuna hyperparameter searches and mea- sured performance as function of trial budget. Figure H. reports mean accuracy and standard deviation computed over moving window of the last five trials at each budget size, across different random seeds and hyperparameter samples.
Smelt Onal Arcopmy
Seadteed Deltarys Sinke
0.42
2.00
6413
329
202
6330
Figure H.l: Mean accuracy over trials (moving window of last 5 trials) on the CP (left) and HI (right) datasets function of trial budget. Curves show model mean accuracy, and shaded regions indicate the corresponding standard deviation across seeds/hyperparamete samples.
20


Reviews:
[{"review_id": "GB0WmpO5xw", "reviewer": "Reviewer_Meqn", "summary": "This paper introduces TANDEM (Tree-And-Neural Dual Encoder Model), which combines neural networks with oblivious soft decision trees in a hybrid autoencoder for tabular classification under limited labels. The key idea is using two complementary encoders - a standard neural network and a tree-based encoder - each with sample-specific gating networks that act as learnable feature selectors. The encoders share a decoder and train jointly via reconstruction, alignment, and latent similarity losses. At test time, only the neural encoder is used. The authors test this on 19 OpenML datasets using 400 labeled examples each, reporting improvements over various supervised baselines.", "questions": "1. Why exclude SCARF, VIME, and SubTab from comparisons? These are the most relevant recent tabular SSL methods and their absence is puzzling given they're discussed in related work.\n\n2. Can you explain the TabNet baseline results? The 0.54 mean accuracy seems much lower than expected for this method. What hyperparameters were used and how was it implemented?\n\n3. What about statistical significance? With 100 runs, you should be able to provide p-values for the reported improvements. Are these differences statistically meaningful?\n\n4. How does this scale? What happens with larger datasets (>10K samples) or different label ratios? Is the approach computationally practical for realistic problem sizes?\n\n5. Which datasets benefit most? Is there any pattern in terms of dataset characteristics (number of features, categorical vs numerical, etc.) that determine when TANDEM helps most?", "limitations": "The authors mention small sample sizes and low-label focus, but several other limitations deserve attention:\n\nEvaluation scope: Results from 19 carefully filtered datasets with 400 examples each may not generalize to the broader space of tabular problems.\n\nBaseline coverage: Missing comparisons to the most relevant existing methods limits our understanding of true performance gains.\n\nHyperparameter sensitivity: The method introduces multiple hyperparameters (loss weights, gating noise, tree parameters) but there's limited analysis of sensitivity to these choices.\n\nReal-world applicability: The controlled experimental setting may not reflect challenges in practical deployment.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nOriginality: Genuinely novel architectural contribution.\nThe hybrid neural-tree autoencoder for tabular SSL appears unprecedented in the literature. While NODE uses oblivious decision trees and various methods combine neural/tree approaches, no prior work integrates them within a self-supervised autoencoder framework. The \"model-based augmentation\" concept represents a paradigm shift from data corruption approaches (SCARF, VIME, SubTab) to architectural diversity, which could influence future research directions.\n\nQuality: Technically sound with well-motivated design.\nThe three-component loss function is well-designed: reconstruction loss preserves semantic information, alignment loss (L2 distance between reconstructions) prevents encoder divergence, and cosine similarity in latent space maintains consistency despite architectural differences. The stochastic gating mechanism using clipped Gaussian noise (σ=0.5) provides differentiable feature selection. The spectral analysis provides concrete evidence that neural gating acts as a low-pass filter while tree-based gating preserves high-frequency content, validating the complementary inductive biases claim.\n\nClarity: Well-structured presentation with clear motivation.\nThe paper effectively motivates the problem (neural networks' spectral bias on tabular data), clearly explains the hybrid architecture, and provides intuitive explanations for design choices. Mathematical notation is consistent, figures effectively illustrate the approach, and the spectral analysis adds valuable theoretical insight. The writing flows logically from problem identification through method description to experimental validation.\n\nSignificance: Addresses important limitations in tabular SSL.\nThe work tackles a real problem - neural networks often underperform trees on tabular data, and existing tabular SSL methods struggle with effective augmentation strategies. The approach could help bridge the gap between tree-based and neural methods for tabular learning, potentially improving performance in low-label regimes where labeled data is expensive.\n\n\nWeaknesses\n\nQuality: Experimental validation is severely limited.\nThe evaluation scope is problematically narrow - only 19 datasets with artificially constrained settings (≥2,500 samples per class, exactly 400 labeled examples) cannot support general effectiveness claims. Missing comparisons to key tabular SSL methods (SCARF, VIME, SubTab) mentioned in related work make it impossible to assess true advancement. Baseline results are concerning: TabNet achieving 0.54 mean accuracy while TANDEM gets 0.71 seems implausible without implementation details. No statistical significance testing, despite 100 repetitions, undermines confidence in reported improvements.\n\nClarity: Key experimental details are underexplained.\nWhile the method description is clear, critical experimental aspects lack transparency. Baseline implementation details are missing, making suspicious results (TabNet's poor performance) hard to evaluate. The choice of hierarchical gating for trees versus single gating for neural encoders is not well justified. Computational cost analysis is absent, leaving scalability questions unanswered.\n\nSignificance: Limited demonstrated impact.\nThe narrow experimental scope (400 labeled examples on filtered datasets) severely limits practical relevance. Many real-world tabular problems involve thousands of samples, not hundreds. Without broader evaluation or comparison to recent SSL methods, it's unclear whether this approach offers meaningful advantages over existing techniques. The improvements, while consistent in the constrained setting, may not generalize to realistic scenarios.\n\nOriginality: Individual components largely exist.\nWhile the specific combination is novel, core components are incremental extensions of existing work: oblivious decision trees (NODE), stochastic gating (similar to TabNet's attention), autoencoder SSL (standard), and multi-component losses (common in contrastive learning). The novelty lies primarily in integration rather than fundamental algorithmic innovation. The model-based augmentation concept, while interesting, needs broader validation to demonstrate its effectiveness beyond this specific architecture.", "was_revised": true, "final_justification": "I thank the authors for the detailed responses! After evaluating the responses to all of the reviews, I have decided to increase the rating due to the added value and insight through the additional experiments.", "rebuttal": "We thank the reviewer for their thoughtful assessment and constructive feedback. We are particularly grateful for your recognition of TANDEM’s “genuinely novel architectural contribution” and its “paradigm shift” in self-supervision. Your appreciation of our well-motivated design, including the alignment and latent similarity losses, as well as the spectral analysis that validates the complementary inductive biases, is deeply encouraging. We also value your acknowledgment of the vital challenge our work addresses by bridging the gap between neural and tree-based models in tabular learning.\n\nWe found your comments and questions extremely helpful, and we address each of them below, providing additional experiments, clarifications, and analyses.\n\n\n**W1 + W3 + Q1: Experimental validation and scope**\n\n\nWe thank the reviewer and want to clarify that our method’s contributions extend beyond specific dataset sizes or label counts. Nonetheless, we have significantly expanded the experiments in three key directions:\n\n1. **Recent SSL baselines**:  \nWe added SCARF, VIME, and SubTab to our benchmark, evaluating them on the same 19 classification datasets with 400 labeled samples per class.  \nAs shown in **Table 1**, TANDEM outperforms all three methods in both mean accuracy and mean rank across datasets.  \nThis confirms that our hybrid approach with dual inductive biases provides stronger representations than existing SSL alternatives in low-label regimes.\n\n*Table 1: mean accuracy and mean rank per over all datasets*\n\n*Due to space constraints, detailed results for all evaluations in this response will be included in the appendix*.\n\n| Dataset           | SCARF      | VIME       | SubTab     | TANDEM     |\n| ----------------- | ---------- | ---------- | ---------- | ---------- |\n| **Mean Accuracy** | 0.6696     | 0.6489     | 0.6735     | **0.7157** |\n| **Mean Rank**     | 3.00       | 3.11       | 2.58       | **1.32**   |\n\n2. **Extension to regression tasks**:  \nTo test the generality of TANDEM beyond classification, we evaluated it on 13 regression datasets using 400 samples per task, all from the benchmark from [1].  \nThe results, shown in **Table 2**, demonstrate that TANDEM also achieves the best average performance (lowest mean MSE and mean rank) compared to all baselines. We believe that results could be improved by using other losses for NN (as shown in [2]), and consider this an important area for future work. \n\n***Table 2**:  mean MSE (MMSE)  and mean rank of various models across 13 regression datasets. Best results are bolded.*\n\n| Dataset       | CatBoost | LogReg |  MLP   | SCARF  | SubTab |  VIME  | TabM   | TabNet | XGBoost | SS AE  | TANDEM  |\n|---------------|----------|--------|--------|--------|--------|--------|--------|--------|---------|--------|---------|\n| **Mean MSE**  |  0.33  | 0.69 | 0.38 | 0.43 | 0.43 | 0.51 | 0.40 | 0.51 |  0.34 | 0.36 | **0.32** |\n| **Mean Rank** |   4.00   |  8.38  |  4.38  |  7.23  |  7.08  |  8.46  |  4.85  |  9.38  |   4.15  |  4.69  | **3.38**   |\n\n\n3. **Varying the number of labels**:  \nWe conducted additional experiments ranging from **50 to 1000** labeled samples per dataset on the classification task.\nAs shown in **Table 3**, TANDEM consistently ranks at the top or remains competitive across all sample sizes.\n\n***Table 3**: mean classification accuracy for each model trained on different numbers of labeled samples. Bold values indicate the best performance for that sample size (we report only the top 5 models here).*\n\n| Model    |     50 |    100 |    200 |    400 |    600 |    800 |   1000 |\n|:---------|-------:|-------:|-------:|-------:|-------:|-------:|-------:|\n| TabM     | 0.5572 | 0.6081 | 0.6281 | 0.6566 | 0.6847 | 0.6896 | 0.6920 |\n| SubTab   | 0.5787 | 0.6145 | 0.6348 | 0.6735 | 0.6788 | 0.6825 | 0.6846 |\n| SS-AE    | 0.6027 | 0.6298 | 0.6532 | 0.6799 | 0.6901 | 0.6735 | 0.6897 |\n| TabPFN   | 0.6150 | 0.6487 | 0.6734 | 0.6966 | 0.7183 | **0.7312** | **0.7374** |\n| TANDEM   | **0.6284** | **0.6591** | **0.6920** | **0.7151** | **0.7237** | 0.7284 | 0.7237 |\n\n\nThese additions substantially strengthen the empirical foundation of the paper and highlight the significance of TANDEM’s contributions. The model consistently outperforms alternatives in both classification and regression tasks, across a wide range of supervision levels. While many public tabular datasets are large, real-world domains such as healthcare and finance often face a shortage of labeled data. Our results demonstrate strong performance precisely in these challenging settings, underscoring TANDEM’s relevance and practical value (see e.g., [3],[4]). We thank the reviewer for raising this point, which prompted a more comprehensive and impactful evaluation.\n\n**W1 + Q3: Statistical significance comparison**\n\n***Table 4**: how many datasets (out of 19; 18 for TabPFN) TANDEM outperforms each baseline. Parentheses indicate **statistically significant** wins (p < 0.05, 100 trials).*\n\n|           | MLP     | XGBoost | TabM    | MLogReg | DeepTLF | CatBoost | SS-AE   | TabPFN   | VIME     | SCARF    | SubTab   |\n|-----------|---------|---------|---------|---------|---------|----------|---------|----------|----------|----------|----------|\n| TANDEM    | 18 (17) | 18 (17) | 18 (17) | 19 (18) | 19 (19) | 19 (17)  | 18 (18) | 13 (14)  | 16 (17)  | 17 (19)  | 16 (16)  |\n\n\n**W2 + Q2: TabNet baseline results**\n\nWe understand the reviewer’s concern regarding the TabNet results. We used the **pytorch-tabnet** package and tuned its full pipeline (pretraining + classification/regression) via Optuna. In addition, we reproduced the official TabNet implementation from GitHub, including its embedding tuning. Both setups were tuned independently; the best achieved ~0.57 mean accuracy, which is below the performance of TANDEM.\nPrior work [5] also reported suboptimal TabNet performance on tabular data, including datasets from our benchmark, likely due to its sensitivity to tuning and limited handling of categorical features. We thank the reviewer for raising this concern.\nAll implementation and optimization details appear in the appendix.\n\n\n**W2: Justification for hierarchical vs. shared Gating**\n\nIn the **tree encoder**, each layer operates directly over the **original input feature space**. Because of this, **gating must be applied at every depth**, to determine which features are relevant for that layer’s decisions. Moreover, to preserve the **oblivious** structure where layers operate independently, we use a **separate** gating network for each layer. Each of these gating networks is tailored specifically to its corresponding layer, making the overall mechanism inherently hierarchical.\n\nIn contrast, the **neural encoder** applies gating once at the input, since subsequent layers operate **sequentially on the resulting feature embeddings**.\n\n**Q4: Computational cost and scalability**\n\nDuring inference, TANDEM is applied as a fully connected MLP, making it highly efficient and lightweight, suitable even for home CPU environments. Pretraining is under 2× the cost of the self-supervised Autoencoder, with the primary overhead from the tree encoder, which scales modestly with depth. Despite this, TANDEM remains significantly faster than SubTab, VIME, TabNet, and SCARF, both in training and inference.\n\n**Q5: Categorical feature ratio breakdown**\n\nTo better understand where TANDEM excels, we grouped the datasets by their categorical feature ratio.\n\n**High (≥ 0.70):** PW (1.00), BM (0.92), CO (0.85), AD (0.87)  \n**Medium (0.30–0.69):** RS (0.69), ALB (0.39), EY (0.33), CC (0.33)  \n**Low (< 0.30):** OG (0.10), CP (0.27), VO (0.18), EL (0.12), HI, HE, JA, NU, CA (0.00)\n\nThis characterization highlights that TANDEM performs particularly well on datasets with a **moderate share of categorical features** and also shows strong results on entirely numeric datasets with **low-cardinality continuous features**, which often behave like categorical inputs.\nStudying this effect further is an interesting direction for future work.\n\n**L4: Sensitivity analysis of TANDEM hyperparameters**\n\nWe conducted a sensitivity analysis over five datasets, examining various ranges of loss weights, gating hidden dimension, tree depth, and number of trees. TANDEM showed **stable performance** across a wide range of values. Optimization trials using Optuna on two datasets (one where TANDEM excels and one where it does not) demonstrated **stable performance**.\nAdditional results will be included in the appendix. The alignment and latent representation similarity loss weights sensitivity analysis is discussed in the response to reviewer **uEwZ**, and the optimization behavior is addressed in response to reviewer **km3R**. (Omitted here due to space constraints).\n\n**We hope this addresses all your concerns, and we would be happy to further clarify open issues.**\n\n[1] Grinsztajn  et al. Why do tree-based models still outperform deep learning on typical tabular data?\n\n[2] Stewar et al. Building Bridges between Regression, Clustering, and Classification.\n\n[3] Arazi at al. TabSTAR: A Foundation Tabular Model With Semantically Target-Aware Representations.\n\n[4] Hollmann et al. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second.\n\n[5] Gorishniy et al Revisiting deep 318 learning models for tabular data."}, {"review_id": "bHaVGu8Mf2", "reviewer": "Reviewer_Fcpp", "summary": "This paper proposes a new approach of learning from tabular data in low-label settings. Shortcomings of neural networks on tabular data are proposed to be mitigated by supplementing the modeling process with a tree-based model. This model intends to capture sharp variations commonplace in tabular data that neural networks have been shown to struggle on.", "questions": "1.  Are there any features that are known to be irrelevant/nuisance features in datasets from OpenML? Do we know which features are irrelevant in $\\texttt{covertype}$ and $\\texttt{PhishingWebsites}$?\n\n2. Are irrelevant features and nuisance features the same? Is it possible that some features are a nuisance (to neural networks), but are not irrelevant? The way things are currently worded imply that OSDT is learning irrelevant features.\n\n3. Is the gating network for OSDT selecting a higher proportion of categorical features compared to the gating network for the neural encoder?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n•\tThis work bridges the research gap between modern deep learning techniques such as self-supervised learning, and applications that rely on tabular data.\n\n•\tPaper is well-written and organized.\n\n•\tExperiments to show that the gating networks lead to complimentary spectra is useful in making the case that the two models are not learning from the same set of features.\n\n•\tThe dashed box to show the combined inductive biases an in Figure 1 is appreciated.\n\nWeaknesses:\n\n•\tLack of examples and experiments to test if existing feature selection and/or elimination methods fail on tabular data. If we can partition these feature sets, we can train a network and a tree separately, and then ensemble them.\n\n•\tIt would be nice to have more explanation about how the complementary feature sets differ in terms of magnitude of dataset-specific overlap, and if models prefer certain types of features.", "was_revised": true, "final_justification": "The authors have sufficiently addressed my concerns in their responses and have communicated them clearly. I am increasing the rating to accept.", "rebuttal": "Thank you for the thoughtful and well-engaged review. We appreciate your interest in the core motivation behind our work, especially your questions around how the gating networks behave and whether the two encoders prefer different types of features. We're also glad you found the paper clear and technically solid, and we appreciate your recognition of the strong empirical results. We're happy to address the insightful points you raised.\n\n**W1: Evaluating Feature Selection in Tabular Data**\n\nWe thank the reviewer for raising the important question of whether existing feature selection or elimination methods are sufficient for inducing complementary representations in tabular learning. We address this in two parts: by evaluating **global feature selection with trees**, and by comparing TANDEM (our method) to **local masking-based self-supervised (SSL) approaches**.\n\n1. **Global Feature Selection with Trees**:\n\nTo explore the reviewer’s suggestion that partitioned feature sets could support effective ensemble, we evaluated whether applying explicit feature selection before training improves performance in a tree-based setup. Specifically, we trained a decision tree (DT) alongside a simple one-layer softmax model under two key settings: one using all input features, and one using only a subset selected by a SOTA NN-based model for feature selection (STG [1]).\nIn both cases, ensembling improved performance over the tree alone. Notably, the variant with explicit FS yielded **further gains**, showing that even simple models benefit from selecting relevant features. While this setup achieved promising results competitive with strong classifiers like XGBoost and CatBoost, it still fell short of TANDEM.These findings support the reviewer’s intuition and highlight the value of **combining diverse inductive biases** through architectural integration.\n\n***Table 1**: Accuracy of Decision Tree, FS-gated MLP, and their ensemble, along with Jaccard similarity between their top-10 selected features.*\n\n| Setting          | Tree Acc | Model Acc | Ensemble Acc | FeatSim |\n|:----------------:|---------:|----------:|-------------:|--------:|\n| with FS   |   0.630  |   0.644   |     **0.653**    |  **0.455**  |\n| without FS     |   0.630  |   0.622   |     0.641    |  0.405  |\n\n\n2. **Evaluating Local Masking-Based Feature Selection Methods**:\n\nWe aimed to evaluate how existing local feature selection methods in SSL compare to our approach. To this end, we considered two representative SSL methods that rely on masking: **VIME**, which applies noise to random subsets of features and trains the model to predict the corruption mask [2], and **SubTab**, which reconstructs inputs from masked subsets [3].\nTo include a stronger baseline that incorporates learned feature selection, we also evaluated a self-supervised autoencoder with feature selection (SS-AE with FS using STG [1]), which learns to suppress irrelevant features during pretraining.\n\nAs shown in **Table 2**, both VIME and SubTab underperform compared to TANDEM, demonstrating the limitations of random or task-agnostic masking used for local feature selection in SSL. In contrast, SS-AE with FS performs significantly better than the other local feature selection methods, highlighting the value of learned, sample-specific gating in self-supervised settings. \n\nStill, TANDEM consistently outperforms all alternatives, suggesting that its **combination of gating with diverse encoder types** is key to learning strong, complementary representations.\n\n***Table 2**: Mean accuracy and mean rank across datasets for VIME, SubTab, SS-AE with STG, and TANDEM.  \nAll methods were evaluated on the same datasets and under the same evaluation protocol as used in the paper, with identical pretraining data.*\n*Due to space constraints, detailed results for all evaluations in this response will be included in the appendix*.\n\n| Model        | Mean Accuracy | Mean Rank |\n|--------------|----------------|------------|\n| TANDEM       | **0.7157**     | **1.40**   |\n| SS-AE with FS  | 0.6941         | 2.20       |\n| SubTab       | 0.6735         | 3.10       |\n| VIME         | 0.6489         | 3.70       |\n\n**W2 + Q3: Complementary gating behavior**:\n\n To further analyze how the neural and tree components differ in their learned feature preferences, we computed several metrics across four datasets with a moderate proportion of categorical features (**Table 3** , further details on the datasets categorization are shown in the response to reviewer **Meqn**). This analysis focuses specifically on how the gating network of the decision tree and the gating network of the neural network differ in their feature selection behavior over categorical features. We report three interpretable metrics. **Gate Overlap** measures the proportion of categorical features that are simultaneously activated by both gating networks. **Gate Correlation** captures the alignment between the gate strength values assigned by the two networks using Pearson correlation. **Gate Variance Ratio (Tree / NN)** reflects how sharply each gating network modulates feature importance by comparing the variance in their gate strengths. We observe moderate agreement between the gating networks in terms of overlap and correlation. Furthermore, the tree’s gating network consistently shows **higher variance**, indicating it makes more decisive distinctions between categorical features. In contrast, the neural gating network applies smoother, more conservative gating. These differences support our core design goal in TANDEM: to encourage the two components to attend to **complementary aspects** of the input.\n\n ***Table 3**: Comparison of gating behavior between the neural and tree encoders, evaluated only on the categorical features of the datasets.*\n\n\n| Dataset | Gate Overlap | Gate Correlation | Gate Variance Ratio (Tree / NN) |\n|---------|--------------|------------------|----------------------------------|\n| RS      | 0.78         | 0.88             | 1.52                             |\n| ALB     | 0.59         | 0.66             | 1.81                             |\n| EY      | 0.33         | 0.34             | 1.61                             |\n| CC      | 0.65         | 0.82             | 1.76                             |\n\n\n**Q1: Feature relevance in OpenML datasets**\n\nWe did not find official documentation labeling irrelevant or nuisance features in the OpenML datasets. Globally, most features appear task-relevant; however, we believe some features act as local nuisances, i.e., unhelpful for specific samples even if not globally irrelevant. This is supported by the behavior of our gating mechanism, which selectively downweights such features during inference.\nSimilar observations have been made in prior work, where both global and local nuisance features were identified and mitigated([1], [4]).\n\nFurther analysis of feature quality is crucial to ensure these benchmarks remain both informative and realistic.\n\n**Q2: OSDT robustness to nuisance features**\n\n We refer to nuisance features as variables that may be irrelevant for the prediction task. For example, in genetics, when trying to predict a medical condition, often only a small subset of variables carries information, while the rest are nuisance features. In our current notation, we treat nuisance and irrelevant features interchangeably.\n\nTo clarify our claim that tree-based methods are more resilient to nuisance features, we refer to prior work [5], which shows that decision trees are inherently less sensitive to uninformative features due to their greedy, axis-aligned splits and implicit regularization. \nTo further illustrate this property empirically, we injected 100 purely irrelevant Gaussian features into the CO dataset. As shown in Table 4, XGBoost’s performance remained stable, whereas the MLP experienced a significant drop in accuracy. This validates the notion that nuisance features can degrade the performance of NNs and underscores the robustness of tree-based approaches, like OSDT, in attenuating the influence of nuisance features.\n\n***Table 4**: Impact of adding 100 Gaussian nuisance features to the CO dataset.*  \n\n| Dataset | Model   | Accuracy (original)     | Accuracy (with noise)     |\n|---------|---------|-------------------------|----------------------------|\n| CO      | XGBoost | 0.5326 ± 0.016          | 0.5317 ± 0.018            |\n| CO      | MLP     | 0.4963 ± 0.014          | 0.4437 ± 0.029            |\n\n**We appreciate the feedback and would be happy to further clarify any open issues.**\n\n[1] Yamada et al. Feature Selection using Stochastic Gates. \n\n[2] Yoon et al. VIME: Extending the success of self- and semi-supervised learning to tabular domain.\n\n[3] Ucar et al. SubTab: Subsetting features of tabular data for self-supervised representation learning.\n\n[4] Yoon et al. INVASE: Instance-wise Variable Selection using Neural Networks\n\n[5] Grinsztajnet al. Why do tree-based models still outperform deep learning on typical tabular data?"}, {"review_id": "Bu2rpLfGfp", "reviewer": "Reviewer_km3R", "summary": "The authors propose TANDEM, a hybrid method utilising two different encoder types: a deep neural network and an ensemble of oblivious soft decision tree, both sharing the same decoder. \n\nDuring training, the method enforces agreement between the latent embeddings of the two encoders, allowing the OSDT to incentivise the neural network into capturing high-frequency signals and conditional feature interplay, overcoming the bias of neural networks towards smooth, low frequency functions, thereby addressing a core weakness of deep models when used to tasks of learning tabular data. Additionally, the proposed system utilises stochastic gating networks for each encoder. The gating networks act as sample-dependent feature selectors, providing different components of the data for the neural network and for the OSDT. The original input is reconstructed from the two embeddings using a shared decoder, with the dual encoders trained in sync through two components of the loss function: reconstruction alignment and latent representation similarity. The OSDT is discarded at inference time, attaching a simple MLP classifier to the pretrained neural encoder and gating module.\n\nThe proposed method and the experimental setup shows an interesting and effective way of fusing tree-based inductive biases into self-supervised pretraining of deep learning methods, achieving improved results on a suite of classification tasks.", "questions": "* Could the authors please report how training time and memory footprint compare to that of the baselines? Including approximate training times and hardware details would help contextualize the computational efficiency of the proposed method.\n* Could the authors clarify why the experiments are confined to only 400 labeled samples, considerably fewer than typical benchmarks (e.g. 10k to 50k in [1])? Additionally, would it be possible to evaluate the model in fully supervised and moderately labeled regimes to provide a more comprehensive assessment of its effectiveness?\n* It would be helpful to describe how hyperparameter sensitive the proposed method is in comparison to tree-based methods (or other baselines considered), as mentioned above in the Weaknesses section. If possible, please include hyperparameter search space and the normalised test accuracy across budget size. \n* Could the authors please provide information on why regression tasks have not been considered?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n* Instead of relying on dataset-level interventions like data augmentation, which can be potentially problematic for tabular data, the proposed method enhances the model by integrating complementary inductive biases\n* The complementarity of the inductive biases of the two components is thoroughly demonstrated in the experiments, with spectral analysis revealing a pronounced contrast\n* The experimental setup demonstrate consistent gains over a diverse range of methods, highlighting the benefits of training a neural network and a tree-based encoder with synchronized representations\n\n**Weaknesses**:\n* The proposed method involves several hyperparameters, such as temperature, number of trees, tree depth, and mask ratio, that could potentially introduce tuning challenges. While the reported results based on the best-performing run over 50 trials show substantial improvements, the paper would benefit from a more thorough analysis of the method's sensitivity to hyperparameters. This aspect is particularly relevant given that one of the key strengths of tree-based models is their strong out-of-the-box performance that requires minimal tuning. I recommend including an analysis of the normalized test accuracy across trial budget sizes (by accounting for random permutations of runs), similar to Figures 1 and 2 in [1] that would further strengthen the evaluation of the proposed method.\n* The hyperparametrs search ranges, as well as memory footprint and computational complexity estimates are not provided. Including these efficiency metrics is essential in order to estimate the practical feasibility and scalability, as well as potential limitations of the proposed method when comparing to tree-based, neural networks, and hybrid approaches.\n* The evaluation is limited to classification tasks. The most highly utilised benchmarks for tabular datasets also include regression tasks ( [1], [2]). Additionally, evaluation is conducted only on low label regimes (400 labeled samples), different from the standard evaluation procedures on the same benchmarks.\n* TANDEM's self-supervised signal is based on feature masking and reconstruction. This objective implicitly relies on correlation among the dataset columns that allows the prediction of the hidden columns from the remaining ones. On datasets where features are largely independent or weakly correlated, the reconstruction loss might carry little information.\n\n[1] Grinsztajn, L., Oyallon, E. and Varoquaux, G., 2022. Why do tree-based models still outperform deep learning on typical tabular data?. _Advances in neural information processing systems_, _35_, pp.507-520.\n\n[2] Shwartz-Ziv, R. and Armon, A., 2022. Tabular data: Deep learning is not all you need. _Information Fusion_, _81_, pp.84-90.", "was_revised": true, "final_justification": "The authors responses, to my questions as well as those raised by the other reviewers, clarify several aspects of the proposed method. After carefully reading the other reviews and the rebuttal, I am raising my initial score for the following reasons:\n\n* The authors have expanded the experiments to regression tasks, for the setup of up to 400 labeled examples. The results show improvement of TANDEM over the baselines, achieving the best mean rank. However, I recommend including further details on what regression datasets were used and how did the authors select them as this aspect has not been included in the discussions.\n* A new evaluation varying the number of labeled data samples shows that TANDEM remains competitive even outside the scarce regime.\n* Baselines referenced in the related work but previously missing from the comparisons (SCARF, VIME, SubTab) have been added, with TANDEM exceeding their performance. VIME and SubTab in particular add two representative SSL methods relying on masking.\n* Deeper analysis of the gating mechanism is provided. Complementarity between the two gates is demonstrated through (i) targeted experiments on gating behavior on the categorical features regarding overlap, correlation and variance ratio in activations (Table 3, response to reviewer Fcpp) and (ii) a variance per sample and feature analysis revealing dynamic adaptation (Table 4, response to reviewer uEwZ). These results substantiate the claim of the authors on component complementarity.\n* New ablation on the weight of the alignment loss component, addressing concerns regarding potential representations collapse.\n* Hyperparameter sensitivity analysis shows stable accuracy among the top performing runs. While compute efficiency results show similar orders with other methods, I still recommend including the full parameter ranges for the optimization.", "rebuttal": "We thank the reviewer for their thoughtful assessment and constructive feedback. We're particularly grateful for your recognition of the key strengths of our work, including the overall clarity and motivation of TANDEM’s architecture, the effective integration of tree-based and neural inductive biases via dual encoders, and the use of stochastic gating networks for per-sample feature selection. We're also grateful for the acknowledgement of our spectral analysis as a convincing tool to demonstrate the representational differences between the encoders and for noting the consistent performance improvements across a broad set of classification benchmarks. These points reinforce the motivation behind our design choices and the validity of our experimental conclusions.\nThe reviewer raises important points about evaluation scope, efficiency, and hyperparameter sensitivity. We appreciate the opportunity to address them below.\n\n**W1 + Q3: Hyperparameter sensitivity and stability**\n\nWe fully agree that one of the core strengths of tree-based models is their strong out-of-the-box performance, and we designed TANDEM (our method) to retain this practical usability. We clarify that although TANDEM includes several configurable parameters (e.g., number of trees, tree depth), in practice we observed **stability across a wide range of values** (see table and further analysis on the loss weights in response to **uEwZ**). Furthermore, **TANDEM does not utilize a mask ratio parameter**, and we did **not perform tuning on the temperature** parameter, which was fixed to a default value throughout all experiments. This further minimizes the tuning burden in practice.\n\nTo empirically assess optimization robustness, we include a cost analysis of performance over increasing trial budgets. The following tables report the mean accuracy and standard deviation computed over a moving window of the last five trials at each budget size, across different random seeds and hyperparameter samples drawn using Optuna. This experiment is based on the reviewer’s suggestion.\n\nAs shown in **Table 1** and **Table 2**, TANDEM demonstrates stable accuracy across multiple hyperparameter trials, indicating that the method is not highly sensitive to hyperparameter tuning. This result holds for both datasets, where TANDEM is the best, and for those where it is not.\n\n***Table 1**: mean Accuracy Over Trials on CP dataset (window of last 5 trials)*\n| Trial Budget | TabPFN        | XGBoost       | MLP           | TANDEM        |\n|--------------|---------------|---------------|---------------|---------------|\n| 5      | 0.592 ± 0.004 | 0.580 ± 0.004 | 0.575 ± 0.016 | 0.675 ± 0.011 |\n| 10     | 0.593 ± 0.002 | 0.574 ± 0.005 | 0.563 ± 0.012 | 0.678 ± 0.009 |\n| 20     | 0.597 ± 0.005 | 0.587 ± 0.005 | 0.564 ± 0.019 | 0.682 ± 0.005 |\n| 30     | 0.597 ± 0.004 | 0.580 ± 0.007 | 0.587 ± 0.007 | 0.682 ± 0.003 |\n\n***Table 2**: mean Accuracy Over Trials on HI dataset (window of last 5 trials)*\n\n| Trial Budget | TabPFN        | XGBoost       | MLP           | TANDEM        |\n|--------------|---------------|---------------|---------------|---------------|\n| Trial 5      | 0.654 ± 0.006 | 0.628 ± 0.005 | 0.554 ± 0.020 | 0.640 ± 0.010 |\n| Trial 10     | 0.650 ± 0.002 | 0.633 ± 0.004 | 0.561 ± 0.017 | 0.642 ± 0.006 |\n| Trial 20     | 0.659 ± 0.004 | 0.633 ± 0.009 | 0.562 ± 0.006 | 0.644 ± 0.005 |\n| Trial 30     | 0.656 ± 0.005 | 0.632 ± 0.007 | 0.566 ± 0.014 | 0.644 ± 0.006 |\n\n**W2 + Q1: Hyperparameter transparency and computational efficiency**\n\nWe clarify that all hyperparameter search ranges for TANDEM and the baselines are provided in the Appendix (Table E.1), ensuring transparency and reproducibility.\nTo assess practical feasibility, we provide two tables below. **Table 3**  shows the mean pretraining time across all datasets for 50 epochs and 2000 samples per label on an L4 GPU.  TANDEM is faster than all self-supervised baselines evaluated, except for the self-supervised autoencoder (SS-AE), which happens because of the added OSDT encoder and gating net, which are both smaller than the SS-AE (and TANDEM’s) encoder-decoder network. \n\n***Table 3**:  Pretraining Time per Model*\n| Model   | Mean Time (s) |\n|:--------|--------------:|\n| SS‑AE   | 38.12         |\n| TANDEM  | 43.47         |\n| TabNet  | 50.22         |\n| SCARF   | 63.05         |\n| SubTab  | 264.89        |\n| VIME    | 309.76        |\n\n\n**Table 4**  presents downstream training time per batch (128 samples) on an L4 GPU, where TANDEM behaves like a lightweight MLP.\nTANDEM's downstream training cost is minimal, reducing to a **simple MLP plus one-time usage of a gating network**, making it highly scalable and practical for real-world deployment, even on CPUs (in contrast to TabPFN). Its performance gains and representational power justifies the modest training overhead (just slightly more than the MLP).\n\n***Table 4**:  Downstream Training Time per Batch*\n \n| Model             | Mean Time (s) |\n|-------------------|---------------:|\n| XGBoost           | 0.03           |\n| MLP               | 0.08           |\n| TANDEM | 0.08           |\n| DeepTFL           | 0.10           |\n| TabM              | 0.12           |\n| TabPFN            | 0.16           |\n\n\n**W3 + Q2 + Q4: Evaluation beyond classification and across labeling regimes**\n\nWe thank the reviewer for pointing out the limited scope of the initial evaluation. Following this valuable feedback, we expanded our experiments in two significant directions:\n1. **Extension to regression tasks**\n\nTo test the generality of TANDEM beyond classification, we evaluated it on 13 regression datasets using 400 labeled samples per task, all from the benchmark from [1].  \n\nThe results, shown in **Table 5**, demonstrate that while less decisive than in the classification evaluation, TANDEM also achieves the best performance (lowest MSE and mean rank) compared to all baselines. We believe that results could be improved by using other losses for NN (as shown in [2]), and consider this an important area for future work. \n\n***Table 5**:  mean MSE (MMSE)  and mean rank of various models across 13 regression datasets. Best results are bolded.*\n\n*Due to space constraints, detailed results for all evaluations in this response will be included in the appendix*.\n| Dataset       | CatBoost | LogReg |  MLP   | SCARF  | SubTab |  VIME  | TabM   | TabNet | XGBoost | SS AE  | TANDEM  |\n|---------------|----------|--------|--------|--------|--------|--------|--------|--------|---------|--------|---------|\n| **Mean MSE**  |  0.33  | 0.69 | 0.38 | 0.43 | 0.43 | 0.51 | 0.40 | 0.51 |  0.34 | 0.36 | **0.32** |\n| **Mean Rank** |   4.00   |  8.38  |  4.38  |  7.23  |  7.08  |  8.46  |  4.85  |  9.38  |   4.15  |  4.69  | **3.38**   |\n\n2. **Varying labeled sample sizes**:  \n\nWe conducted additional experiments ranging from 50 to 1000 labeled samples per dataset on the classification task.\nAs shown in **Table 6**, TANDEM consistently ranks at the top or remains competitive across all sample sizes. TANDEM achieves **state-of-the-art performance** with fewer than 800 labeled samples, and remains competitive even beyond that scale.\n\n***Table 6**: mean classification accuracy for each model trained on different numbers of labeled samples. Bold values indicate the best performance for that sample size.*\n\n| Model    |     50 |    100 |    200 |    400 |    600 |    800 |   1000 |\n|:---------|-------:|-------:|-------:|-------:|-------:|-------:|-------:|\n| TabM     | 0.5572 | 0.6081 | 0.6281 | 0.6566 | 0.6847 | 0.6896 | 0.6920 |\n| SubTab   | 0.5787 | 0.6145 | 0.6348 | 0.6735 | 0.6788 | 0.6825 | 0.6846 |\n| SS-AE    | 0.6027 | 0.6298 | 0.6532 | 0.6799 | 0.6901 | 0.6735 | 0.6897 |\n| TabPFN   | 0.6150 | 0.6487 | 0.6734 | 0.6966 | 0.7183 | **0.7312** | **0.7374** |\n| TANDEM   | **0.6284** | **0.6591** | **0.6920** | **0.7151** | **0.7237** | 0.7284 | 0.7237 |\n\nThese additions substantially strengthen the empirical foundation of the paper and highlight the significance of TANDEM’s contributions. The model consistently outperforms alternatives in both classification and regression tasks, across a wide range of supervision levels. While many public tabular datasets are large, real-world domains such as healthcare and finance often face limited labeled data. Our results demonstrate strong performance precisely in these challenging settings, underscoring the relevance and practical value of TANDEM. We thank the reviewer for raising this point, which prompted a more comprehensive and impactful evaluation.\n\n**W4: Feature correlation and reconstruction objectives**\n\nWe agree that when features are weakly correlated, reconstruction-based pretraining may carry limited signal. However, in real-world tabular data, dependencies often exist between features. To demonstrate this, we computed the sum of absolute Pearson correlations between a feature and all other features (on numerical features only) within each dataset. In the table below, we present the mean of these values, which indicates substantial inter-feature dependencies in practice. We note that there may be additional nonlinear dependencies that have not been evaluated here.\n\n***Table 7**: Mean and median of sum of absolute correlations per feature averaged across all datasets (numerical features only).* \n| Metric                         | Value (mean ± std)      |\n|-------------------------------|--------------------------|\n| Mean Sum Abs Corr             | 3.08 ± 1.61              |\n| Median Sum Abs Corr           | 2.03 ± 1.48              |\n\n**We hope this addresses all your concerns, and we would be happy to further clarify open issues.**\n\n[1] Grinsztajnet al. Why do tree-based models still outperform deep learning on typical tabular data?. \n\n[2] Stewart et al. Building Bridges between Regression, Clustering, and Classification."}, {"review_id": "Ckub2yeE0E", "reviewer": "Reviewer_uEwZ", "summary": "This paper proposes Hybrid Autoencoders, a neural architecture for tabular data that combines two distinct encoding paths: a vector encoder that operates on the entire feature vector and a set encoder that processes each feature independently via a shared MLP. A learnable gating mechanism fuses the two representations in a dimension-wise manner, and a consistency loss encourages alignment between the two branches. The model is evaluated on several classification benchmarks, including semi-supervised settings, and shows competitive or superior performance compared to MLP-based baselines.", "questions": "- Have you evaluated the method on regression tasks?\n- Does the consistency loss improve performance consistently, or are there datasets where it is detrimental?\n- Do the gating values collapse to a single branch across training, or are both branches used meaningfully throughout?\n- Why were existing unsupervised or semi-supervised baselines (e.g., VIME, contrastive methods) not included in the experimental comparison?", "limitations": "- The proposed method is only tested on classification benchmarks. Its applicability to regression or structured prediction is not validated.\n- While the gating mechanism is central to the architecture, its behavior is only superficially analyzed via averaged values per dimension.\n- The paper lacks comparison with other unsupervised or self-supervised representation learning methods for tabular data, limiting the assessment of its effectiveness in the semi-supervised regime.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The architecture is modular and conceptually simple, combining complementary local (feature-wise) and global (vector-wise) representations.\n- The use of a per-dimension gating mechanism provides interpretable fusion and adaptivity across datasets.\n- The method does not rely on complex pretraining or specialized architectures, making it easy to implement and extend.\n\n**Weakness**\n\n- The architectural novelty is limited; the method essentially combines a standard MLP and a shared-feature encoder.\n- The model is evaluated only on classification tasks. It remains unclear how it performs on regression, missing values, or under distribution shift.\n- The consistency loss, while conceptually appealing, may suppress diverse representations between branches; this risk is not analyzed in detail.\n- The paper does not clarify the role of the gating mechanism beyond showing average values; it is unclear how gating varies across samples or features.\n- The relationship between fusion behavior (Figure 5) and generalization performance is not well established.\n- The paper adopts a semi-supervised setting with limited labeled data, yet does not compare against standard self-supervised learning approaches followed by linear probing or fine-tuning, which are widely used for few-label regimes. This includes methods like VIME, masked-feature modeling, or contrastive learning, all of which are relevant baselines for evaluating representation quality.", "was_revised": true, "final_justification": "I remain unconvinced, as not all of my concerns have been addressed, and the additional results provided do not appear to be reliable. As an active researcher working on tabular learning and its industrial applications, and someone who has been continuously using models such as TabPFN, I find the relatively low performance reported in this submission even more questionable. Furthermore, I do not believe this work provides important insights to the field. Therefore, I will maintain my current score.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We appreciate the recognition of the modular and conceptually simple architecture, the interpretability and adaptivity enabled by the per-dimension gating mechanism, as well as the model’s ease of implementation and extensibility. We appreciate that the core contributions of integrating complementary local and global feature representations in a principled and interpretable way are viewed as valuable and technically sound.\n\nWe appreciate the reviewer’s concerns and address them through new regression experiments, further analysis of gating behavior, and clarification of the consistency loss. We also extend our evaluation to include additional SSL baselines such as VIME, SubTab, and Scarf. These additions broaden the empirical scope and reinforce the method's effectiveness and generalization.\n\n**W1: Architectural contribution - parallel fusion of neural networks and tree-based encoders**\n\nWe appreciate the reviewer’s point regarding architectural simplicity. While the neural and tree-based components are individually standard, the core novelty lies in their **parallel** integration within the same **representation space**. Unlike prior works that combine trees and neural networks in a **sequential manner** , for example, by feeding the output of one into the other ([1]), our approach processes features **independently** through both encoders and then fuses the resulting representations directly. This parallel fusion enables the model to learn from the complementary inductive biases of trees and neural networks without one dominating or reshaping the other's input space.\n\n**W2 + Q1 + L1: Extension to Regression Tasks**\n\nTo test the generality of TANDEM beyond classification, we evaluated it on 13 regression datasets using 400 samples per task, all from the benchmark from [2].  \n\nThe results, shown in **Table 1**, demonstrate that while less decisive than in the classification evaluation, TANDEM also achieves the best average performance (lowest mean MSE and mean rank) compared to all baselines, We believe that results could be improved by using other losses for NN (as shown in [3]), and consider this an important area for future work. \n\n***Table 1**:  mean MSE (MMSE)  and mean rank of various models across 13 regression datasets. Best results are bolded.*\n\n*Due to space constraints, detailed results for all evaluations in this response will be included in the appendix*.\n\n| Dataset       | CatBoost | LogReg |  MLP   | SCARF  | SubTab |  VIME  | TabM   | TabNet | XGBoost | SS AE  | TANDEM  |\n|---------------|----------|--------|--------|--------|--------|--------|--------|--------|---------|--------|---------|\n| **Mean MSE**  |  0.33  | 0.69 | 0.38 | 0.43 | 0.43 | 0.51 | 0.40 | 0.51 |  0.34 | 0.36 | **0.32** |\n| **Mean Rank** |   4.00   |  8.38  |  4.38  |  7.23  |  7.08  |  8.46  |  4.85  |  9.38  |   4.15  |  4.69  | **3.38**   |\n\n\n\n**W2 + Q3: Effect of consistency loss on representation diversity**\n\nWe appreciate the concern regarding the potential collapse of diverse representations due to the consistency loss. To investigate this, we conducted a sensitivity analysis over a wide range of values for both components of the consistency term: 1) alignment loss weight and 2) latent representation similarity (LRS) loss weight. Results on five representative datasets are summarized below.\n\n***Table 2-3**: Accuracy for varying consistency hyperparameter values in parentheses represents feature dim.*\n\n***Table 2**: Sensitivity to Alignment Loss Weight*\n\n| Hyperparameter | CP(11) | OG(93) | PW(30) | ALB(25) | CC(23) |\n|----------------|--------|--------|--------|---------|--------|\n| 0.0            | 0.6702 | 0.6721 | 0.9332 | 0.6790  | 0.7165 |\n| 0.1            | 0.6732 | 0.6789 | 0.9387 | 0.6781  | 0.7286 |\n| 0.5            | 0.6775 | 0.6865 | 0.9613 | 0.7032  | 0.7326 |\n| 1.0            | 0.6753 | 0.6860 | 0.9620 | 0.7038  | 0.7231 |\n| 10.0           | 0.6105 | 0.6230 | 0.8930 | 0.6211  | 0.6815 |\n\n***Table 3**: Sensitivity to latent representation similarity weight*\n\n| Hyperparameter | CP     | OG     | PW     | ALB    | CC     |\n|----------------|--------|--------|--------|--------|--------|\n| 0.0            | 0.6742 | 0.6520 | 0.9547 | 0.6802 | 0.7179 |\n| 0.1            | 0.6739 | 0.6587 | 0.9583 | 0.6976 | 0.7281 |\n| 0.5            | 0.6783 | 0.6860 | 0.9615 | 0.7034 | 0.7327 |\n| 1.0            | 0.6692 | 0.6870 | 0.9618 | 0.7038 | 0.7254 |\n| 10.0           | 0.5859 | 0.5845 | 0.8808 | 0.5807 | 0.6312 |\n\nThese results show that across a **wide range**, the consistency terms either improve or preserve performance without collapsing the dual encoders. Only at **excessively large** values do we observe degradation due to dominance of one branch, but this is rare and easily avoided with light tuning.\n\n**W4 + L2: Clarifying the role of the gating mechanism**\n\nTo provide a deeper analysis of the gating mechanism, we computed summary statistics across four datasets with a moderate proportion of categorical features (**Table 4** , further details on the datasets categorization are shown in the response to reviewer **Meqn**) to capture how gate values fluctuate across both features and samples. Specifically, we report the mean and variance of the activations from the two gating networks, one connected to the neural encoder and the other to the decision tree.\n\nAs shown in Table 4, both gating networks exhibit relatively high variance, highlighting that gate values are not static but **dynamically adapt per sample and feature**. The tree-based gating shows higher variance, reflecting its more discrete, thresholded behavior, while the neural gating is smoother yet still responsive.\n\n***Table 4**: Mean and variance of gate activations across samples and features.*\n\n| Dataset | MeanNNGate | VarNNGate | MeanDTGate | VarDTGate |\n|--------------------------------------------|------------|-----------|------------|-----------|\n| ALB| 0.604 | 0.294 | 0.4544 | 0.338 |\n| CC| 0.487| 0.248 | 0.4342 | 0.301 |\n| RS| 0.646| 0.282 | 0.5801 | 0.345 |\n| EY | 0.594| 0.236 | 0.7075 | 0.380 |\n\n**W5 + Q3 + L2: Complementary gating behavior and its role in generalization**\n\nTo examine how the complementary gating behavior of the neural and tree components contributes to generalization, we computed several metrics on the same datasets analyzed previously.\n\n This analysis focuses specifically on how the gating network of the decision tree and the gating network of the neural network differ in their feature selection behavior over categorical features, potentially leading to improved generalization. We report three interpretable metrics. **Gate Overlap** measures the proportion of categorical features that are simultaneously activated by both gating networks. **Gate Correlation** captures the alignment between the gate strength values assigned by the two networks using Pearson correlation. **Gate Variance Ratio (Tree / NN)** reflects how sharply each gating network modulates feature importance by comparing the variance in their gate strengths.  \n\nWhile the two gating networks show moderate agreement across all datasets in terms of overlap and correlation, the tree’s gating network consistently shows **higher variance**, indicating it makes more decisive distinctions between categorical features. In contrast, the neural gating network applies smoother, more conservative gating. These differences support our core design goal in TANDEM: to encourage the two components to attend to **complementary aspects** of the input, which helps with generalization.\n\n\n ***Table 5**: Comparison of gating behavior between the neural and tree encoders, evaluated only on the categorical features of the datasets*\n\n| Dataset | Gate Overlap | Gate Correlation | Gate Variance Ratio (Tree / NN) |\n|---------|--------------|------------------|----------------------------------|\n| RS      | 0.78         | 0.88             | 1.52                             |\n| ALB     | 0.59         | 0.66             | 1.81                             |\n| EY      | 0.33         | 0.34             | 1.61                             |\n| CC      | 0.65         | 0.82             | 1.76                             |\n\n**W6 + Q4 + L3: Comparison with SSL baselines**\n\nWe've added SCARF, VIME, and SubTab to our benchmark, evaluating them on the same 19 classification datasets with 400 labeled samples per class.  \n\nAs shown in **Table 6**, TANDEM outperforms all three methods in both mean accuracy and mean rank across datasets.  \nThis confirms that our hybrid approach with dual inductive biases provides stronger representations than existing SSL alternatives in low-label regimes.\n\n***Table 6**: mean accuracy and mean rank per over all datasets.*\n\n| Dataset           | SCARF      | VIME       | SubTab     | TANDEM     |\n| ----------------- | ---------- | ---------- | ---------- | ---------- |\n| **Mean Accuracy** | 0.6696     | 0.6489     | 0.6735     | **0.7157** |\n| **Mean Rank**     | 3.00       | 3.11       | 2.58       | **1.32**   |\n\n**We appreciate the feedback and would be happy to further clarify any open issues.**\n\n[1] Luo et al. NCART: Neural Classification and Regression Tree for Tabular Data. \n\n[2] Grinsztajn et al . Why do tree-based models still outperform deep learning on typical tabular data?.\n\n[3] Stewart et al. Building Bridges between Regression, Clustering, and Classification."}]

Author rebuttal:

Meta-review: The paper provides a solution for tabular data tasks in low-label settings. It is mentioned to be well written, and provide a practical solution to an important problem (Meqn, “The work tackles a real problem - neural networks often underperform trees on tabular data, and existing tabular SSL methods struggle with effective augmentation strategies.” uEwZ ”The method does not rely on complex pretraining or specialized architectures, making it easy to implement and extend.”). 

In addition to the bottom-line results showing a competitive performance compared with SOTA techniques, the reviews appreciated the studies motivating the different design choices. km3R “The complementarity of the inductive biases of the two components is thoroughly demonstrated in the experiments,” , Meqn “The spectral analysis provides concrete evidence that neural gating acts as a low-pass filter while tree-based gating preserves high-frequency content, validating the complementary inductive biases claim.”, Fcpp “Experiments to show that the gating networks lead to complimentary spectra is useful in making the case that the two models are not learning from the same set of features.”. 

The paper initially had a borderline rating by the reviewers mostly due to incomplete experiments, requiring an extension of the study to regression tasks in addition to classification tasks, and various ablation studies. The authors provided these experiments in the rebuttal stage and the reviewers Fcpp and km3R raised their score, mentioning that their concerns were fully mitigated and in particular these experiments provide sufficient justification to the design choices and proof of the competitiveness of the method compared to existing art. 

A remaining issue without consensus is the novelty and insights provided by the paper. Despite the rebuttal, the reviewers had mixed opinions about the subject, leading to different overall scores. This is a more subjective issue, and given that more than one reviewer did find the work to have interesting insights and novel components, I believe that the paper will be of value to participants of NeurIPS.
Decision: Accept (poster)