Title: Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settings
Venue: NeurIPS 2025
Accepted: yes
Keywords: Tabular data

Self-supervised learning

Autoencoders

Hybrid models

Feature selection

Oblivious soft decision trees

Model-based augmentation
Abstract: Deep neural networks often under-perform on tabular data due to their sensitivity to irrelevant features and a spectral bias toward smooth, low-frequency functions. These limitations hinder their ability to capture the sharp, high-frequency signals that often define tabular structure, especially under limited labeled samples. While self-supervised learning (SSL) offers promise in such settings, it remains challenging in tabular domains due to the lack of effective data augmentations. We propose a hybrid autoencoder that combines a neural encoder with an oblivious soft decision tree (OSDT) encoder, each guided by its own stochastic gating network that performs sample-specific feature selection. Together, these structurally different encoders and model-specific gating networks implement model-based augmentation, producing complementary input views tailored to each architecture. The two encoders, trained with a shared decoder and cross-reconstruction loss, learn distinct yet aligned representations that reflect their respective inductive biases. During training, the OSDT encoder (robust to noise and effective at modeling localized, high-frequency structure) guides the neural encoder toward representations more aligned with tabular data. At inference, only the neural encoder is used, preserving flexibility and SSL compatibility. Spectral analysis highlights the distinct inductive biases of each encoder. Our method achieves consistent gains in low-label classification and regression across diverse tabular datasets, outperforming deep and tree-based supervised baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Hybrid Autoencoders, a neural architecture for tabular data that combines two distinct encoding paths: a vector encoder that operates on the entire feature vector and a set encoder that processes each feature independently via a shared MLP. A learnable gating mechanism fuses the two representations in a dimension-wise manner, and a consistency loss encourages alignment between the two branches. The model is evaluated on several classification benchmarks, including semi-supervised settings, and shows competitive or superior performance compared to MLP-based baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The architecture is modular and conceptually simple, combining complementary local (feature-wise) and global (vector-wise) representations.\n- The use of a per-dimension gating mechanism provides interpretable fusion and adaptivity across datasets.\n- The method does not rely on complex pretraining or specialized architectures, making it easy to implement and extend.\n\n**Weakness**\n\n- The architectural novelty is limited; the method essentially combines a standard MLP and a shared-feature encoder.\n- The model is evaluated only on classification tasks. It remains unclear how it performs on regression, missing values, or under distribution shift.\n- The consistency loss, while conceptually appealing, may suppress diverse representations between branches; this risk is not analyzed in detail.\n- The paper does not clarify the role of the gating mechanism beyond showing average values; it is unclear how gating varies across samples or features.\n- The relationship between fusion behavior (Figure 5) and generalization performance is not well established.\n- The paper adopts a semi-supervised setting with limited labeled data, yet does not compare against standard self-supervised learning approaches followed by linear probing or fine-tuning, which are widely used for few-label regimes. This includes methods like VIME, masked-feature modeling, or contrastive learning, all of which are relevant baselines for evaluating representation quality.\n\nQUESTIONS:\n- Have you evaluated the method on regression tasks?\n- Does the consistency loss improve performance consistently, or are there datasets where it is detrimental?\n- Do the gating values collapse to a single branch across training, or are both branches used meaningfully throughout?\n- Why were existing unsupervised or semi-supervised baselines (e.g., VIME, contrastive methods) not included in the experimental comparison?\n\nLIMITATIONS:\n- The proposed method is only tested on classification benchmarks. Its applicability to regression or structured prediction is not validated.\n- While the gating mechanism is central to the architecture, its behavior is only superficially analyzed via averaged values per dimension.\n- The paper lacks comparison with other unsupervised or self-supervised representation learning methods for tabular data, limiting the assessment of its effectiveness in the semi-supervised regime.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose TANDEM, a hybrid method utilising two different encoder types: a deep neural network and an ensemble of oblivious soft decision tree, both sharing the same decoder. \n\nDuring training, the method enforces agreement between the latent embeddings of the two encoders, allowing the OSDT to incentivise the neural network into capturing high-frequency signals and conditional feature interplay, overcoming the bias of neural networks towards smooth, low frequency functions, thereby addressing a core weakness of deep models when used to tasks of learning tabular data. Additionally, the proposed system utilises stochastic gating networks for each encoder. The gating networks act as sample-dependent feature selectors, providing different components of the data for the neural network and for the OSDT. The original input is reconstructed from the two embeddings using a shared decoder, with the dual encoders trained in sync through two components of the loss function: reconstruction alignment and latent representation similarity. The OSDT is discarded at inference time, attaching a simple MLP classifier to the pretrained neural encoder and gating module.\n\nThe proposed method and the experimental setup shows an interesting and effective way of fusing tree-based inductive biases into self-supervised pretraining of deep learning methods, achieving improved results on a suite of classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n* Instead of relying on dataset-level interventions like data augmentation, which can be potentially problematic for tabular data, the proposed method enhances the model by integrating complementary inductive biases\n* The complementarity of the inductive biases of the two components is thoroughly demonstrated in the experiments, with spectral analysis revealing a pronounced contrast\n* The experimental setup demonstrate consistent gains over a diverse range of methods, highlighting the benefits of training a neural network and a tree-based encoder with synchronized representations\n\n**Weaknesses**:\n* The proposed method involves several hyperparameters, such as temperature, number of trees, tree depth, and mask ratio, that could potentially introduce tuning challenges. While the reported results based on the best-performing run over 50 trials show substantial improvements, the paper would benefit from a more thorough analysis of the method's sensitivity to hyperparameters. This aspect is particularly relevant given that one of the key strengths of tree-based models is their strong out-of-the-box performance that requires minimal tuning. I recommend including an analysis of the normalized test accuracy across trial budget sizes (by accounting for random permutations of runs), similar to Figures 1 and 2 in [1] that would further strengthen the evaluation of the proposed method.\n* The hyperparametrs search ranges, as well as memory footprint and computational complexity estimates are not provided. Including these efficiency metrics is essential in order to estimate the practical feasibility and scalability, as well as potential limitations of the proposed method when comparing to tree-based, neural networks, and hybrid approaches.\n* The evaluation is limited to classification tasks. The most highly utilised benchmarks for tabular datasets also include regression tasks ( [1], [2]). Additionally, evaluation is conducted only on low label regimes (400 labeled samples), different from the standard evaluation procedures on the same benchmarks.\n* TANDEM's self-supervised signal is based on feature masking and reconstruction. This objective implicitly relies on correlation among the dataset columns that allows the prediction of the hidden columns from the remaining ones. On datasets where features are largely independent or weakly correlated, the reconstruction loss might carry little information.\n\n[1] Grinsztajn, L., Oyallon, E. and Varoquaux, G., 2022. Why do tree-based models still outperform deep learning on typical tabular data?. _Advances in neural information processing systems_, _35_, pp.507-520.\n\n[2] Shwartz-Ziv, R. and Armon, A., 2022. Tabular data: Deep learning is not all you need. _Information Fusion_, _81_, pp.84-90.\n\nQUESTIONS:\n* Could the authors please report how training time and memory footprint compare to that of the baselines? Including approximate training times and hardware details would help contextualize the computational efficiency of the proposed method.\n* Could the authors clarify why the experiments are confined to only 400 labeled samples, considerably fewer than typical benchmarks (e.g. 10k to 50k in [1])? Additionally, would it be possible to evaluate the model in fully supervised and moderately labeled regimes to provide a more comprehensive assessment of its effectiveness?\n* It would be helpful to describe how hyperparameter sensitive the proposed method is in comparison to tree-based methods (or other baselines considered), as mentioned above in the Weaknesses section. If possible, please include hyperparameter search space and the normalised test accuracy across budget size. \n* Could the authors please provide information on why regression tasks have not been considered?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach of learning from tabular data in low-label settings. Shortcomings of neural networks on tabular data are proposed to be mitigated by supplementing the modeling process with a tree-based model. This model intends to capture sharp variations commonplace in tabular data that neural networks have been shown to struggle on.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n•\tThis work bridges the research gap between modern deep learning techniques such as self-supervised learning, and applications that rely on tabular data.\n\n•\tPaper is well-written and organized.\n\n•\tExperiments to show that the gating networks lead to complimentary spectra is useful in making the case that the two models are not learning from the same set of features.\n\n•\tThe dashed box to show the combined inductive biases an in Figure 1 is appreciated.\n\nWeaknesses:\n\n•\tLack of examples and experiments to test if existing feature selection and/or elimination methods fail on tabular data. If we can partition these feature sets, we can train a network and a tree separately, and then ensemble them.\n\n•\tIt would be nice to have more explanation about how the complementary feature sets differ in terms of magnitude of dataset-specific overlap, and if models prefer certain types of features.\n\nQUESTIONS:\n1.  Are there any features that are known to be irrelevant/nuisance features in datasets from OpenML? Do we know which features are irrelevant in $\\texttt{covertype}$ and $\\texttt{PhishingWebsites}$?\n\n2. Are irrelevant features and nuisance features the same? Is it possible that some features are a nuisance (to neural networks), but are not irrelevant? The way things are currently worded imply that OSDT is learning irrelevant features.\n\n3. Is the gating network for OSDT selecting a higher proportion of categorical features compared to the gating network for the neural encoder?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TANDEM (Tree-And-Neural Dual Encoder Model), which combines neural networks with oblivious soft decision trees in a hybrid autoencoder for tabular classification under limited labels. The key idea is using two complementary encoders - a standard neural network and a tree-based encoder - each with sample-specific gating networks that act as learnable feature selectors. The encoders share a decoder and train jointly via reconstruction, alignment, and latent similarity losses. At test time, only the neural encoder is used. The authors test this on 19 OpenML datasets using 400 labeled examples each, reporting improvements over various supervised baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nOriginality: Genuinely novel architectural contribution.\nThe hybrid neural-tree autoencoder for tabular SSL appears unprecedented in the literature. While NODE uses oblivious decision trees and various methods combine neural/tree approaches, no prior work integrates them within a self-supervised autoencoder framework. The \"model-based augmentation\" concept represents a paradigm shift from data corruption approaches (SCARF, VIME, SubTab) to architectural diversity, which could influence future research directions.\n\nQuality: Technically sound with well-motivated design.\nThe three-component loss function is well-designed: reconstruction loss preserves semantic information, alignment loss (L2 distance between reconstructions) prevents encoder divergence, and cosine similarity in latent space maintains consistency despite architectural differences. The stochastic gating mechanism using clipped Gaussian noise (σ=0.5) provides differentiable feature selection. The spectral analysis provides concrete evidence that neural gating acts as a low-pass filter while tree-based gating preserves high-frequency content, validating the complementary inductive biases claim.\n\nClarity: Well-structured presentation with clear motivation.\nThe paper effectively motivates the problem (neural networks' spectral bias on tabular data), clearly explains the hybrid architecture, and provides intuitive explanations for design choices. Mathematical notation is consistent, figures effectively illustrate the approach, and the spectral analysis adds valuable theoretical insight. The writing flows logically from problem identification through method description to experimental validation.\n\nSignificance: Addresses important limitations in tabular SSL.\nThe work tackles a real problem - neural networks often underperform trees on tabular data, and existing tabular SSL methods struggle with effective augmentation strategies. The approach could help bridge the gap between tree-based and neural methods for tabular learning, potentially improving performance in low-label regimes where labeled data is expensive.\n\n\nWeaknesses\n\nQuality: Experimental validation is severely limited.\nThe evaluation scope is problematically narrow - only 19 datasets with artificially constrained settings (≥2,500 samples per class, exactly 400 labeled examples) cannot support general effectiveness claims. Missing comparisons to key tabular SSL methods (SCARF, VIME, SubTab) mentioned in related work make it impossible to assess true advancement. Baseline results are concerning: TabNet achieving 0.54 mean accuracy while TANDEM gets 0.71 seems implausible without implementation details. No statistical significance testing, despite 100 repetitions, undermines confidence in reported improvements.\n\nClarity: Key experimental details are underexplained.\nWhile the method description is clear, critical experimental aspects lack transparency. Baseline implementation details are missing, making suspicious results (TabNet's poor performance) hard to evaluate. The choice of hierarchical gating for trees versus single gating for neural encoders is not well justified. Computational cost analysis is absent, leaving scalability questions unanswered.\n\nSignificance: Limited demonstrated impact.\nThe narrow experimental scope (400 labeled examples on filtered datasets) severely limits practical relevance. Many real-world tabular problems involve thousands of samples, not hundreds. Without broader evaluation or comparison to recent SSL methods, it's unclear whether this approach offers meaningful advantages over existing techniques. The improvements, while consistent in the constrained setting, may not generalize to realistic scenarios.\n\nOriginality: Individual components largely exist.\nWhile the specific combination is novel, core components are incremental extensions of existing work: oblivious decision trees (NODE), stochastic gating (similar to TabNet's attention), autoencoder SSL (standard), and multi-component losses (common in contrastive learning). The novelty lies primarily in integration rather than fundamental algorithmic innovation. The model-based augmentation concept, while interesting, needs broader validation to demonstrate its effectiveness beyond this specific architecture.\n\nQUESTIONS:\n1. Why exclude SCARF, VIME, and SubTab from comparisons? These are the most relevant recent tabular SSL methods and their absence is puzzling given they're discussed in related work.\n\n2. Can you explain the TabNet baseline results? The 0.54 mean accuracy seems much lower than expected for this method. What hyperparameters were used and how was it implemented?\n\n3. What about statistical significance? With 100 runs, you should be able to provide p-values for the reported improvements. Are these differences statistically meaningful?\n\n4. How does this scale? What happens with larger datasets (>10K samples) or different label ratios? Is the approach computationally practical for realistic problem sizes?\n\n5. Which datasets benefit most? Is there any pattern in terms of dataset characteristics (number of features, categorical vs numerical, etc.) that determine when TANDEM helps most?\n\nLIMITATIONS:\nThe authors mention small sample sizes and low-label focus, but several other limitations deserve attention:\n\nEvaluation scope: Results from 19 carefully filtered datasets with 400 examples each may not generalize to the broader space of tabular problems.\n\nBaseline coverage: Missing comparisons to the most relevant existing methods limits our understanding of true performance gains.\n\nHyperparameter sensitivity: The method introduces multiple hyperparameters (loss weights, gating noise, tree parameters) but there's limited analysis of sensitivity to these choices.\n\nReal-world applicability: The controlled experimental setting may not reflect challenges in practical deployment.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}