Title: ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zero-shot learning

clustering

representation learning

in-context learning

tabular data
Abstract: Clustering tabular data remains a significant open challenge in data analysis and machine learning. Unlike for image data, similarity between tabular records often varies across datasets, making the definition of clusters highly dataset-dependent. Furthermore, the absence of supervised signals complicates hyperparameter tuning in deep learning clustering methods, frequently resulting in unstable performance. To address these issues and minimize the need for per-dataset tuning, we adopt an emerging approach in deep learning: zero-shot learning. We propose ZEUS, a self-contained model capable of clustering new datasets without any additional training or fine-tuning. It operates by decomposing complex datasets into meaningful components that can then be clustered effectively. Thanks to pre-training on synthetic datasets generated from a latent-variable prior, it generalizes across various datasets without requiring user intervention. To the best of our knowledge, ZEUS is the first zero-shot method capable of generating embeddings for tabular data in a fully unsupervised manner. 
Experimental results demonstrate that it performs on par with or better than traditional clustering algorithms and recent deep learning-based methods, while being significantly faster and more user-friendly.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an unsupervised approach to learn embeddings for zero-shot clustering of the tabular data. The main ingredients of their approach are: 1) pre-train using synthetic data as in TabPFN 2) Transformer architecture for the backbone 3) loss based on GMM. In a series of experiments, the authors demonstrate the applicability of their approach to the\n\nSTRENGTHS AND WEAKNESSES:\n## Context\nDeep learning for tabular (heterogeneous) data has seen large number of works, which can be roughly grouped into architecture improvements (TabNet, NODE), unsupervised learning (SwitchTab, Scarf), retrieval-augmented models (TabR, ModernNCA), and foundational models (TabGPT, TabPFN). The latter, TabPFN, is the latest work on the foundational models for tabular data that has demonstrated an almost magical ability to generalize to previously unseen datasets in a zero-shot fashion via in-context learning. This work builds on the idea of TabPFN and extends it to the tabular clustering problem. \n\n## Strength\n1. **Empirical results.** The method clearly outperforms other approaches based on shallow models (e.g. GMM) and autoencoders (e.g. AE-GMM, G-CEALS). The authors provide an ablation study to elucidate design choices of the loss. In addition, they evaluate model calibration, which is useful in many settings. \n2. **Impact in practice.** The authors formulate the problem as a GMM. Therefore, the predicted embeddings are organized spherically in the latent space, which makes them directly amenable to application with downstream algorithms such as KNN or K-Means with minimal adaptation. \n3. **Novelty.** To my knowledge, this is the first zero-shot embedding algorithm for tabular data. \n\n## Weaknesses\n1. **Incomplete comparison.** **Firstly**, the TabPFN repo has extensions, one of which is the unsupervised learning of embeddings: https://github.com/PriorLabs/tabpfn-extensions/tree/main/src/tabpfn_extensions/embedding. I'm not sure which work has appeared first, but nevertheless, I feel it would be a valuable comparison to understand the difference between TabPFN and ZEUS embeddings. **Secondly**, there are prior works on the unsupervised learning for tabular data such as Scarf or SwitchTab -- these are not zero-shot, but it would be nice to include some reference (or benchmark) to them to understand how they differ from the current work.\n2. **Failure cases not explained.** If we take this work apart, the technical novelty lies in prior dataset generation and loss formulation. Both are formulated / generated using GMM. My complaint is that the section on the results may not accurately depict the real-world performance: 1) simulated / transformed datasets also follow GMM -- all the model needs to learn here is how to learn identity or inverse (of ResNet) transformation 2) real datasets are chosen to have ARI > 0.4 by at least one of the methods. Related to 2), there are multiple  problems: a) the threshold for cutoff appears a bit arbitrary b) all of the methods (AE, GMM) build on top of Gaussian assumption -> therefore selected real datasets are likely to have the Gaussian bias as well c) the problem in (b) could be addressed if the authors include other methods e.g. TabPFN or SwitchTab. In general, it would be great to see a stress test of the model to understand when it fails. Let's say there are no clusters, or if there is a severe class imbalance, or there is a mode connecting, or the geometry of clusters is very difficult.\n\nQUESTIONS:\n## Understanding loss\n1. From your final objective given in (3), I have doubts about the difficulty of the optimization and the importance of good initialization. Do you keep updating $\\hat c_k$ at each iteration? \n2. What role do $\\hat \\pi$'s play? In unsupervised settings, we use EM to fit GMM, so $\\hat \\pi$'s are updated each E-step. In your case, they seem fixed, defined by the training label. \n\n## Architecture\n1. There is little that is said about the architecture in the paper. The authors mention that they re-use TabPFN backbone in Section 3.1. However, for TabPFN the training label is passed together with the features. There's also cross-attention between training and test sets. Could the authors elaborate more on the specific architectural adjustments? \n2. Major drawback of TabPFN is the context length, which is currently limited at around 10,000 samples for TabPFN V2. Recent works such as TabFlex or Mixture of ICP have explored addressing this concern via subsampling or compressing the data. Are these approaches portable to ZEUS?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-19"}

comments: SUMMARY:
The paper presents a transformer-based approach (called ZEUS) where a transformer is trained on datasets with ground truth cluster labels and learns to assign cluster labels to new datasets. Zeus has been compared against k-means, GMMs and deep clustering methods on several families of real and synthetic datasets.

STRENGTHS AND WEAKNESSES:
Strengths
S1. ZEUS implements a novel idea on how to train a transformer using annotated clustering datasets in order to assign cluster labels on new datasets.
S2. The approach can accommodate mixed datasets with both real and categorical features.

Weaknesses
W1. Since supervised training is used, clustering performance strongly depends on the annotated datasets used for clustering.
W2. There are limits imposed by the input dimensionality of the transformer (dataset size, number of features)

QUESTIONS:
Q1. It is not clear how the real datasets have been separated into training and testing.
Q2. In several cases, ZEUS performance is very low (much lower that k-means and GMMs). This reduces confidence in the method’s validity.
Q3. It would be interesting to compare performance on datasets with discrete features only. Discrete features cause difficulties in k-means and GMMs.
Q4. For many real datasets the number of clusters is 2, which is quite small.
Q5. It should be emphasized that k-means is applied on the transformer output in order to obtain the final cluster labels.
Q6. Some results on the influence of noise and outliers would add value to the experimental study.

LIMITATIONS:
The major limitations of the method are discussed in the paper: dependence on the synthetic datasets used for training and restrictions on the scale of the dataset.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework for unsupervised representation learning for tabular data, such that the resulting representations can be used for clustering datasets in a zero-shot manner. To achieve this, the proposed method (ZEUS) leverages a novel method for generating synthetic data for pre-training, and a set of losses that promote the desirable properties in the learned embedding space. The resulting model leads to strong clustering results in the evaluations conducted.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed methodology is novel, both regarding the synthetic sample generation method and the proposed losses.\n2. ZEUS indeed outperforms alternative methods in terms of the quality of the generated clusters.\n3. The paper is well written and its methodology and motivation is clear.\n\nWeaknesses:\n1. The results presented in Tab. 1 & 2 and Appendix Tab. 4-7 show that, on the aggregate, ZEUS does better than the compared methods. However, it is by small margins that undermine the impact of the proposed method.\n2. The evaluation comparisons are lacking. Most methods compared are naive (e.g. K-Means), and more advanced methods (e.g. G-CEALS) may be at a disadvantage given the size of the models involved: my understanding is that the autoencoder used by G-CEALS is much smaller and less powerfull than the transformer architecture used by ZEUS, which means one cannot be sure how much of ZEUS' gains are related to the architecture as opposed to the method. Finally, it is unclear to my why methods such as DEPICT and IDC are not included in the comparisons.\n3. The ablations should be more thorough. Most importantly, I found no study of the impact of using different synthetic data (Gaussian/Categorical/NN transformed) in the training mix, despite that being a key contribution/component of the paper.\n\nQUESTIONS:\nMy pripary concern is, as stated in the Weaknesses, related to the lack of experiments and comparisons in the paper. I would urge the authors to include more compared methods and ablations in the paper to strengthen the paper's findings (e.g. if more methods were included it might highlight the sifnificance of ZEUS' results). Alternatively, I would urge the authors to justify why works such as the ones I mentioned earlier were not included.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggests ZEUS as a paradigm to pre-train a network for tabular data to allow for general purpose transformations into an embedding space, in which tabular representations are more separated and thus easier to cluster. The chosen pre-training method is extensively explained and theoretically justified and performance benefits are underlined with experimental evaluation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is easy to follow and presents a clear motivation and context for the suggested approach. Furthermore, the experimental evaluation shows overall a clear performance advantage of the ZEUS representations over other classification approaches.\n\nWeaknesses:\n\nIt is not really clear how the pre-training exactly affects the model performance. It seems that ZEUS is the only model that is trained on data other than the downstream task, even though similar pre-training may also be conceivable for some of the baseline models. A comment or even additional experiments on that end may be beneficial.\n\nFurthermore, it would be interesting to see how pre-training on real datasets would affect the ZEUS representations. It also doesn't really become clear how many samples or epochs the model has been trained on. The paper could clearly benefit from additional details and discussion of the data selection.\n\nAt many instances the paper refers to appendices for more details, which are, however, not present in the paper. An evaluation of these crucial aspects is thus not possible in the presented version, which hinders several aspects of understanding and reproducibility.\n\nRegarding the structure, it is not clear why the related work, introducing all relevant baseline models for comparisons is introduced so late and even abbreviations are only introduced at this point.\n\nThe role of the underlying transformer model is not very clear and how its importance measures against that of the data basis. The paper would benefit from additional experiments in this regard.\n\nminor comments.\nthere are some typos that need to be addressed.\nl. 107: --> needs to be performed.\nl. 123:  the introducing --> the introduction of\n\nQUESTIONS:\nWhat dimensional output does Zeus have and what role does the dimensionality play?\n\nHow exactly does the nature of zero-shot learning come into play here and how does it differ from more general transfer learning through feature extraction settings? The reference experiments seem to employ the pre-trained ZEUS network to transform features from the raw data domain to a different embedding space, in which the clustering can be performed, which seems to be a common technique under the umbrella term of transfer learning.\n\nAs ZEUS effectively only transforms the features, why weren't other approaches than k-means employed for the clustering to allow a better understanding of how valuable the transformed ZEUS representations are?\n\nLIMITATIONS:\nThe paper lacks some experimental validation with respect to its data basis and model architectures. Further, some details cannot be extracted due to a missing appendix.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-15"}