ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Data
Authors: Patryk Marszałek
Tomasz Kuśmierczyk
Witold Wydmański
Jacek Tabor
Marek Śmieja
Venue: NeurIPS 2025 poster ()
Keywords: Zero-shot learning
clustering
representation learning
in-context learning
tabular data
Primary area: deep_learning
TL;DR: We introduce ZEUS, a transformer pretrained on synthetic datasets to perform zero‐shot, fully unsupervised clustering of tabular data—no fine‐tuning or hyperparameter search needed—and show it matches or outperforms classical and deep baselines.

Abstract: Clustering tabular data remains a significant open challenge in data analysis and machine learning. Unlike for image data, similarity between tabular records often varies across datasets, making the definition of clusters highly dataset-dependent. Furthermore, the absence of supervised signals complicates hyperparameter tuning in deep learning clustering methods, frequently resulting in unstable performance. To address these issues and minimize the need for per-dataset tuning, we adopt an emerging approach in deep learning: zero-shot learning. We propose ZEUS, a self-contained model capable of clustering new datasets without any additional training or fine-tuning. It operates by decomposing complex datasets into meaningful components that can then be clustered effectively. Thanks to pre-training on synthetic datasets generated from a latent-variable prior, it generalizes across various datasets without requiring user intervention. To the best of our knowledge, ZEUS is the first zero-shot method capable of generating embeddings for tabular data in a fully unsupervised manner. 
Experimental results demonstrate that it performs on par with or better than traditional clustering algorithms and recent deep learning-based methods, while being significantly faster and more user-friendly.

ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Data
Patryk Marszalek! 1.2
Tomasz Kušmierezyk*1
Witold Wydmański 1.2
Jacek Tabor!
Marek Smieja".1
Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland
{t. kusmierczyk, witold. wydnakd.wydd.aa.kkk jacek. tabor, marek. sateeeakmiiejaauu edu. pl patryk..aarzaallkoooctroaaa uj edu.p2
Abstract
Clustering tabular data remains a significant open challenge in data analysis and machine learning. Unlike for image data, similarity between tabular records often varies across datasets, making the definition of clusters highly dataset-dependent. Furthermore, the absence of supervised signals complicates hyperparameter tuning in deep learning clustering methods, frequently resulting in unstable performance. To address these issues and reduce the need for per-dataset tuning, adopt emerging approach in deep learning: zero-shot learning. We propose ZEUS, a self- contained model capable of clustering new datasets without any additional training or fine-tuning. It operates by decomposing complex datasets into meaningful components that can then be clustered effectively. Thanks to pre-training on synthetic datasets generated from a latent-variable prior, it generalizes across various datasets without requiring user intervention. To the best of our knowledge, ZEUS is the first zero-shot method capable of generating embeddings for tabular data in a fully unsupervised manner. Experimental results demonstrate that it performs par with or better than traditional clustering algorithms and recent deep learning-ban based methods, while being significantly faster and more brer-frinndl.
Introduction
Clustering remains a fundamental yet challenging task in unsupervised learning. It particularly hard for tabular data, which inherently lacks the structured spatial semantic properties of images or Unlike for image clustering, where intrinsic visual similarities can guide cluster formation, defining meaningful similarities in tabular data is highly dataset-specifie, complicating the generalization of clustering methods across diverse applications. Recent developments leveraging deep learning have demonstrated promise generating richer representations for clustering tasks. However, these methods frequently suffer from instability due to their sensitivity to hyperparameter selection, challenge exacerbated by the absence of supervised signals to guide optimization. Consequently, practitioners working with tabular data often resort to simpler, classical algorithms like k-means [ despite their limited capacity for capturing complex underlying data structures, simply to avoid extensive manual tuning.
'Joint contribution to project conception, design, and research supervision.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Gaussian mixture
[o, 2, 1,1, Labels
Real data
Generated data
Datasets generator
Model
Neural network
Model
transformation
Optimized
Loss
Representation Inference
Representation
Pre-training
Datasets generation
Figure Schematic characterization of ZEUS: (left) synthetic datasets generation; (middle) pre- training on datasets with known labels; (right) deployment of frozen model for real-world tasks.
We address these challenges with ZEUS- zero-shot transformer-bast model for embedding new tabular datasets in a form convenient for unsupervised separation (=clustering), without the need for additional fine-tuning. Given new dataset, ZEUS returns its transformed representation, where clusters can be easily discovered using simple methods, like k-means. Since works as a zero-shot learner, it significantly reduces hyperparameter tuning complexity and computation time, enabling effective clustering in seconds. It is a plug-and-play solution that requires no fine-tuning and runs in single forward pass for new datasets.
Inspiration for ZEUS stems from the Prior-data Fitted Networks (PFNs) [16] a recently introduced framework highlighting the potential of in-context learning for tabular data. PFNs operate by feeding a transformer [31] contexts representing complete tasks, i.c., training data along with query samples for which the model predicts labels. Despite their impressive performance, they are limited to supervised problems. ZEUS extends this basic idea for unsupervised tasks by addressing the two key challenges of how to: (1) generate prior synthetic data with clear but non-trivial clustering structures for pre-training, and (2) encode prior clustering knowledge for new unlabeled datasets. Additionally, in contrast to TabPFN, ZEUS is zero-shot model and during inference does not rely any context labels.
Unlike traditional methods that optimize arbitrary heuristics (e.g., DEC (32), ZEUS approaches pre-trained on
clustering by learning how to invert data generation processes. In particular, synthetic datasets to learn how to infer cluster assignments. The datasets have known latent structures, which enables supervised guidance for the training, and by generating diverse datasets we supply with the prior knowledge about what can constitute a possible clustering structure, enabling effective generalization to new real-world datasets during inference. Figure Tillustrates the key concepts of the method while Figure presents representations generated by ZEUS.
Experiments demonstrate that ZEUS consistently matches or surpasses both classical clustering algorithms and recent deep learning-basse based approaches, establishing it as powerful and practical tool for the unsupervised analysis of tabular data. Beyond its strong empirical performance, we also provide theoretical justification by showing that ZEUS fits the framework of Prior-data Fitted Networks, thereby reinforcing its theoretical soundness (see Section 2.3
We supplement the paper with an appendix that includes background prior-data fitted networks, specifics of the synthetic data generation process, details of the experimental results, and additional experimental studies. The code used in this paper is available at httpps:/ggthbbb.com https///glhubb.oom/mmm////uu gmmmm/euus
2 Pre-training representation for clustering
In this section, we introduce our approach to zero-shot representation pre-training for clustering. First, we characterize the proposed probabilistic model and its training objective. Then, we explain the process of generating synthetic datasets for pre-training. Finally, we highlight the connection between our model, Bayesian learning and the framework for prior-data fitted networks.

---

ZEUS
Figure 2: t-SNE visualization for a sample synthetic dataset. The representation from ZEUS (right panel) significantly improves consistency with ground-truth classes and reveals clearer data structure.
2.1 ZEUS method
Zero-shot pre-training. Traditional machine learning models fo are trained by optimizing parame- ters 8 to solve specific fixed task, such classification clustering,
fixed dataset. On the other hand, in-context learning (ICL) [ 21 14] enables pre-trained models fo- to adapt to new tasks or datasets without ever updating the pre-trained parameters 0*, Similarly, zero-shot learning allows models to address problems they were never explicitly trained on, without requiring any examples of the target task. Unlike ICL, which adapts through demonstration examples, zero-shot approaches rely entirely on knowledge encoded during pre- pre-training. The pre- vre-training of zero-shot models typically involves massive and diverse datasets that encourage the model learn rich, generalizable represen- tations of underlying patterns and relationships. This is usually done using transformers, as they both can process set-valued inputs (c.g.. accept a whole dataset once) and model complex dependencies.
Our approach involves pre-training transformer fe on diverse collection of datasets such that D = (x.y). y), where y {fr denotes ground-trute -rrth labels (e.g., clusters) and x {xra are feature vectors. A loss L encourages fo to assign accurate labels to all data points in the target datasets D:
et = arg min Ep(D) [C fo(x))] = arg arg min
Σ Cly, fo(x)),
(1)
ND Dru((D)
where p(D) process generating prior datasets which implicitly characterize a space of possible target tasks (here approximated with Np Monte-Carle samples). Note, the loss L may factorize over individual points as C(y, P) = >; E(y,, p.). where P. denotes probabilistic prediction for an individual input zi, but unlike for classification, for tasks such as clustering or anomaly detection, the predictions á must
made jointly for all inputs. Hence, fo(x) can not be decomposed into predictions for individual inputs z4. Furthermore, in contrast typical supervised ICI., training labels are not included in the context, as they are unavailable during inference. This makes the unsupervised learning problem significantly more challenging, since cannot explicitly guide the model toward the desired structure given dataset.
Representation learning for probabilistic clustering. We pre-train encoder fo to map inputs x to representation vectors =(x;) fo(x); ; E RDD =; S. We aim at positioning data points S, so that they naturally form probabilistic clusters, and our goal is to make the encoder find better representations for target tasks. This objective we frame as maximizing the log-likelihood correct cluster assignments. Since the ground-truth cluster assignments y available during pre- pre-training, this is expressed as:
Lpros =-
E log Pys (xi)-
For specifying (x),,, we draw inspiration from Gaussian Mixture Models (GMMs) 15] where each cluster is represented by a Gaussian distribution centered at Cz. and data points are probabilis- tically assigned to clusters through probabilities pı (Irz), based on their proximity to these centroids. Consequently, alongside the encoded representations, we need to define a set of K cluster centroids (c.g., prototypical representations) {cx kK which reside in the same representation space S. We

---

then then,jointe jointly optimize L w.r. and {c+} such that the representations 2(x;)) become structured in way undisclosing underlying probabilistic structures in datasets.
We aim for higher probabilities p for points closer to these centroids. To achieve this, we first define scores Oue (a;) (x.) cg 2 to quantify the compatibility between representations and centroids ck. and then transform them into cluster membership probabilities using softmax, yielding soft cluster exp(es (m:)) Note that the ground-truth cluster assignments y/ are known assignments p& (2:) Σ exp(a,(#=))
during pre-training, and hence, the maximum likelihood centroids can be straightforwardly estimated as Ex = No Σ [ z(r;); where N₂ denotes the number of data points assigned the k-th cluster. Following this formulation, the optimization needs to be performed only w.r. the parameters the neural network fo, the centroids are already specified by them.
Besides the centroids, we additionally consider another concept from GMMs, namely cluster priors sk- Similar to centroids, the prior cluster frequencies & can be estimated from the training labels as the proportions of examples in the k-th cluster, and then incorporated directly into the probability calculation, resulting in the final form of our probabilistic scoring:
th exp()u((())
Pt(x,)) Σj=1 itj (x:()
where ag(2;)) = -11x(<x)) c 12.
(3)
Remark 2.1. ZEUS cluster assignments correspond to the membership probabilities in GMMs SNN(c((,) E., but with fixed diagonal covariance matrices Eg I, defined as: Tsk Σ (, effectively forcing the encoder to learn circular clusters rather than elliptical ones.
Inference. The structural relations learned during pre-training leveraged during inference when the representations produced by the model fa- are used to create predictive distribution for new, previously unseen inputs x* as p p(y" fe. (x*) )). At this point, the labels y* are unknown and therefore cannot be used to estimate the centroids nor the priors. Hence, to structure the obtained representation into clusters, we use traditional learning algorithm, for example, similar to GMMs relying on Expectation- Expectation-Maximizaaa Maximization or simply k-means.
Regularization. Although theoretically Epros is sufficient to build clustering representation, experimentally verified that the introduction additional regularizers further improves the predictions in the inference (see Section 3.4
First, we aim to ensure that the representations z(x;) associated with a particular cluster are compactly distributed around their corresponding centroid to enhance intra-cluster cohesion. We define point concentration regularizer that explicitly minimizes the distance between representations and the centroids, analogous to the k-means objective of minimizing within-cluster sum of squares:
Cep Σ
Σ (xx)) 1:
(4)
Second, to prevent the cluster centroids Cj from collapsing towards similar points in the space, we add centroid separation regularizer. We achieve this by maximizing the sum of squared distances between all distinct pairs of centroids. However, to avoid this term dominating the loss if centroids were pushed infinitely far apart, we cap the contribution of each pair's squared distance at predefined threshold T. The term to be minimized is thus:
Csep Σ
Σ
min(l - ,
(5)
k=1
The final loss combines the main clustering objective with two regularization terms as
L Epros depLep + Asep
(6)
where the hyperparameters Aep 0 and Asep 2 promote point concentration and control the relative importance of enforcing centroid separation, respectively. We used simply App Aeep = 1.

---

2.2 Prior data for pre-training ZEUS
The key component of ZEUS the data- aata-aene prior p(D). As primarily affects the general- ization ability of the pre-trained model fe. it must be designed to cover diverse data distributions and various cluster configurations. In particular, we construct it as a latent variable model (LVM), assuming that each dataset 'D is sampled from a K K-component mixture of distributions. Formally, the probability of data point x 'D under this model
p(x) = - p(x y= k)
p(y=
where p(y) is a categorical distribution, and p(æ y k) represents the k-th (continuous) component. Although the categories y remain latent for real datasets, they are known during synthetic data generation and can serve as ground truth labels - a property integral to pre-training ZEUS. This procedure, coupled with Theorem (in Appendix) ensures that sufficiently expressive model trained on synthetic datasets can handle real datasets drawn from arbitrary mixtures of distributions.
The number of categories K we sample uniformly between 2 and 10, and the observations are sampled from multivariate Gaussian distributions. To control the complexity of datasets, we introduce a constraint (Eq. 7 in Appendix) that ensures sufficient separation between each pair of components. For each component, we generate between 50 and 800 samples. Since real data rarely consist of Gaussian clusters, we additionally transform the data points from each category using randomized ResNet-like neural networks to produce more realistic cluster shapes. We selected ResNets due to their properties. they define invertible transformations (2). and this ensures that clustering structures will be preserved in outputs. Finally, we append a certain fraction of categorical features in one-hot encoding scheme to selected datasets, which is followed by an optional PCA reduction to keep the data dimension at the requested maximum level. Complete details of the prior-data generating process
be found in Appendix B.2 Theoretical justification of this procedure is presented in Appendix B.I
The above strategy for generating clustering datasets illustrates the key idea behind ZEUS. Instead of clustering each dataset individually using an arbitrary loss criterion (as in k-means or DEC), ZEUS learns to perform clustering by inverting the data generation process. Although our synthetic datasets are limited by the selected family of distributions specified above, the proposed paradigm for zero-shot unsupervised learning for clustering is general.
2.3 Relation to Bayesian learning and Prior-Data Fitted Networks
The recently introduced framework of Bayesian inference through transformers demonstrates that neural networks pre-trained on synthetic datasets implicitly approximate Bayesian inference without explicitly computing posterior distributions (25 [6). By framing the approach described in Section as a Prior-Data Fitted Network (PFN), we show ZEÜS implicitly performs approximate Bayesian averaging.
Given a prior distribution p(D) over (synthetic) datasets D, a PFN parameterized by 0 is trained to min- imize the negative likellioddo of predicting held-out labels within datasets sampled from this prior.
The associated loss function is defined as: CPFN ()) = EP ((t.))))(<( (- log (u(a, 2 Deta)). Minimizing the Prior-Data Negative Likellool is then equivalent to minimizing the expected Hellback-Leibles divergence between the network's predictive distribution and the true Posterior Predictive Distribution (PPD) p(y|æ, Dotz.) = Jo p(y]2,, (Dtz ())(((()((()))))))))))))))))))) ()) (see Corollary 1.1 in 251). A pre-trained PFN approximates this integral implicitly, yielding distribution qe ( ax D Desw) directly from forward propagation of the network. In Section a more detailed explanation of PFNs was provided.
ZEUS instead of directly outputting go(y)æ, Detx), maps inputs x to latent representation vectors 2xx). The probabilistic assignments (z) are then constructed from these vectors according to Eq 3 This equation specifies a PPD for inference, but also defines a probability mass function for training:
Remark 2.2. Eqs. 7 2 and constitute a valid Prior-Data Negative Log-Likelihodd, equivalent to ctz W.
Eg. from with
This follows by mapping (x)) == go 2r 0) and noting that Eq. corresponds to the cross-entropy between the true labels {y} and the probabilistic assignments Py (z). Intuitively, this formulation
5

---

reinterprets probabilisticc clustering with known labels classification task. Then, pre- vre-training the transformer by minimizing the cross-entropy loss over PFNs, and thus, we can conclude that ZEUS implicitly learns Bayesian approximation through prior fitting.
datasets remains identical to
Our method, however, deviates from traditional PFNs by imposing an explicit mixture-like structure on latent representations (Eq. 3). unlike more general PFNs:
Remark 2.3. By enforcing the clustering structure, ZEUS may impose stronger assumptions on the PPD compared to vanilla PFNs. This structure might be suboptimal for classification tasks and the true PPD may not belong the family of attainable solutions.
As explained above, clustering based extension is theoretically sound, nonetheless, the assumption could limit representational expressivity compared to the more flexible transformers employed for the original PFNs [16]. On the other hand, the enforced structures shall be more appropriate for unsupervised tasks.
3 Experiments
This section presents an experimental evaluation of the clustering performance of our method. Due to space constraints, further results are provided in Appendix G
3.1 Experimental setup
consists of 12 attention blocks, each with 6 heads and a token dimension of 512, with GeLU activation employed. Following the TabPFN design, each data point is first embedded using a linear transformation and then passed as a token to the transformer. For the unsupervised setting, no label embeddings are ever created or presented to the model. Similarly, for the zero-shot case, no query set is used either, and consequently, attention is computed solely over the support set. Finally, unlike TabPFN, we omit any additional MLP decoder after the transformer.
Model architecture:
ZEUS relies on a transformer architecture similar to TabPFN [6.
Pre-trainings In the gre-training phase, we sample datasets from the mixture of Gaussians and transformed mixtures equal proportions (1:1), generating 1000 unique dataset batch samples for each epoch. For training. we employ the Adam optimizer along with a cosine learning rate scheduler with warm-up, using a learning rate of 2e-5. The plot illustrating model improvements during the pre-training process is available in Section
Inference:
During inference, preprocessing of each dataset involves standardizing numerical features, followed by scaling them to the range 1], whereas categorical features are transformed using one-hot encoding. The input size of our model is fixed to 30. For datasets with lower dimensionality, datasets, we reduce the number of input features via Principal Component Analysis (PCA). Unless stated otherwise, at inference we use k-means applied to the normalized (=scaled to [- 1, 1J) transformer output in order to obtain clusters from our learned representation.
we pad the missing positions with zeros, while for
For evaluation, we consider three groups of datasets: real datasets from OpenML B (Real), synthetic mixtures of Gaussians (Syn. Gauss.), and synthetic mixtures of Gaussians transformed by ResNet-like neural networks (Syn. Transf.). Both types of synthetic datasets are augmented with categorical variables. The process of generating synthetic datasets is described in Section 2.2
Datasets:
Each dataset contains at most 2000 samples as per model design and due to memory limitations. The study covers 34 real datasets, selected based their clustering fcasibility, defined as ARI 2 0.4 achieved by at least one of the methods. Additionally, 20 synthetic datasets of each type were generated from the same prior in the pre-training phase, but with a different random seed to ensure a fair comparison. Detailed statistics of the datasets, such as the number of numerical and categorical features, are provided in Appendix
We compare ZEUS against a wide spectrum of state-of-the-an clustering methods used for tabular data. It includes k-means (KM), Gaussian Mixture Model (GMM), and deep-learning methods based on autoencoder (AE) architectures, including DEC [32], IDEC [3], IDC (30), and G- CEALS 27
Baselines:
We additionally consider k-means and GMM performed in the autoencoder latent space

---

Table 1: Clustering quality (ARI) of ZEUS versus competing methods (higher is better).
GMM
AE-GMM
IDEC
G-CEALS
TabPEN
SCARE
ZEUS
Real
55.54
48.49
51.43
53.56
55.93
$4.57
52.28
40.37
31.32
26.95
57.43
Gauss.
89.90
76.93
81.26
81.40
89.35
82.57
66.43
62.84
55.97
8.32
89.03
Syo. Transf.
75.04
75.88
00:45
71.29
79.94
61.26
66.78
49.17
15.66
2.48
86,33
Table 2: Average rank of the methods used the benchmark (lower better).
KM
GMM
AE-KM
AB-GMM
DEC
IDCC
IDC
G-CEALS
TabPEN
SCARF
ZEUS
Real
5.65
5.72
Syn. Gauss.
2,65
3.65
5.65
5.60
3.23
5.70
7.95
8.90
8.75
11.00
2.92
Syn. Transf
3.50
6.85
3.20
6.35
7.80
11.00
(respectively referred to AE-KM and AE-GMM), along with k-means applied to representations obtained from TabpFN--Unsuperised 17 (TabPFN) and SCARF 1. to assess the clustering quality of different feature representations.
For AE-based baselines, we employ the standard configuration used in the prior literature, i.c., an architecture comprising of hidden layers with sizes [500, 500, 2000]. We used the latent dimension of 20, which we verified experimentally as the best value. Further details on the hyperparameters and code repositories for the baselines are provided in Appendix
Reporting: In the main text, report only the results aggregated for each group of datasets (Real, Syn. Gauss, Syn. Transf)), while detailed results for individual datasets can be found in Section For the reader's convenience, bold the best results and underline the second-best ones.
3.2 How effective is ZEUS for clustering?
To evaluate the performance of the clustering methods, we employ standard evaluation procedure, where clusters identified by models are expected to correspond to undisclosed ground-truth classes. We use the Adjusted Rand Index (ARI) for quantitative evaluation, and improve readability, we scale the ARI values by factor of 100, where 100 indicates perfect clustering and values near represent random grouping. Table presents summary of ARI scores averaged over random seeds and all datasets within each dataset group. Additionally, Table displays the average rankings for each of the methods.
We observe that ZEUS achieves competitive performance for all three groups of datasets. In particular, it achieves the best average ARI and the best rank for both the OpenML and Synthetic Transformed datasets, c.g., for the most challenging clustering scenarios. No- tably, ZEUS outperforms the second-best method, DEC, by more than percentage points, and the classical baselines by over 10 percentage points on the Synthetic Trans- formed datasets. The results the Real datasets further demonstrate that ZEUS ef- fectively generalizes the knowledge acquired during the pre- pre-training stage on synthetic data to general data distributions. For the simplest Gaussian datasets, ZEUS ends up in top-3, with results only slightly below those of k- means and DEC. In terms average rank it gets the second place.
KMMANS
AE-KGMEANS
DEC
G-CEALS ZEUS
GMM
AE GMM
IDEC
10a
Time (s)
10a
10a
128
256
512
1024
2048
4034
Number of samples
Among the AB-based methods, DEC per- forms the best, while G-CEALS consistently demonstrates the weakest performance across all datasets. K-means and GMM, representatives of the classical methods, perform reasonably well.
Figure 3: Clustering time vs. input size.
7

---

Table 3: Soft clustering quality of ZEUS vs. baselines, measured by Brier score (lower is better).
KM
GMM
AE-KM
AE-GMM
DEC
IDEC
G-CEALS
ZEUS
Real 0.4366
0.4799
0.4643
0.4679
0.3941
0.3671
0.4722
0.3817
Syn. Gauss.
0.0970
0.3110
0.2073
0.2484
0.3943
0.2308
0.3946
0.1269
Syn. Transf.
0.2803
0.2566
0.4535
0.3140
0.4638
0.3892
0.4951
0.1796
In particular, k-means consistently lands among the best-performing methods, which justifies its broad adoption among practitioners, despite being one of the most basic approaches. On the other hand, approaches such as SCARF or TabPFN are well-suited for tasks like classification, thanks to their strong performance in feature extraction. However, without additional regularization during training or specialized post-hoc fine-tuning (such as with DEC), effective separation of their representations difficult. The detailed results, including scores for individual datasets corresponding to the averages in Tables [ andd can be found in Appendix F..T
Finally, Figure BiareBjillssss iillssratte how the examined clustering methods scale with the increasing number of input data points. ZEUS maintains almost constant time while being only slightly slower than the basic k-means. It shows that the overhead from creating the representations by ZEUS is minimal. On the other hand, the remaining deep clustering algorithms require significantly more time and scale poorly with the increasing input size.
3.3 Are ZEUS's predictions calibrated?
For applications where uncertainty quantification is as crucial as the clustering decisions themselves, assessing calibration is particularly relevant. Having demonstrated the strong performance of ZEUS for hard clustering, we now examine whether its probabilistic foundations yield well-calibrated soft assignments. We compare against competing approaches using the Brier score, which measures the accuracy of predicted probabilities. Unlike the previously used ARI, it penalizes both incorrect cluster assignments and poorly calibrated confidence scores, thereby providing a more comprehensive evaluation.
The Brier score is supervised metric, meaning its direct computation for unsupervised clustering tasks is not straightforward. However, when ground-truth classes are available and the number of clusters matches the number of classes, one-to-one mapping between clusters and classes can be established using the Hungarian algorithm [19 20) which aims to maximize the total agreement between cluster-class pairs. The cost matrix A for this assignment problem is Aje M where Yic- if data point belongs to class c, and Y,,- = 0 otherwise.
i= Pij Yie.
Table repprts the average Brier score computed over 5 seeds for all datasets in respective groups For the analysis, we used the vanilla variant of ZEUS, which is paired with the GMM clustering since k-means does not provide soft assignments. The covariances were constrained to be identities as implied by Eq. 3 All competing baselines, except for k-means, provide probabilistic cluster assignments, making the Brier score calculation straightforward for them. For k-means, we used one-hot encoding to represent its assignments as probabilities.
ZBUS demonstrates outstanding performance, achieving the best results for the Synthetic Transformed datasets while ranking second both the OpenMI. and Synthetic Gaussian collections. The benefits of using ZEUS representations are especially visible in comparison with the vanilla GMM. Although the GMM achieves the second-best score on transformed data, its performance across the remaining datasets is merely modest. Among the remaining baselines, IDEC exhibits an interesting pattern: despite relatively weak clustering performance in Table it achieves the top position on real datasets according to the Brier score and shows marked improvement on synthetic data. DEC displays the opposite tendency, with calibration results significantly inferior to its strong ARI performance. While G-CEALS remains generally weaker than other methods, its calibration performance consistently exceeds its clustering results presented Table I The K-means algorithm, both in its standard implementation and when applied to autoencoder embeddings, yields impressive scores on Gaussian- categorical mixture datasets likely due to the inherent separability of these data structures, which allows even simple not probability estimates to produce well-calibrated predictions.
8

---

3.4 How helpful is regularization for ZEUS?
Representations obtained by minimizing the basic loss Cprob can be further improved by regularizing the optimization process structure the latent representations. In Table we examine how different combinations of the Lprab loss with the regularizers Cop and Exep affect the final performance across various datasets. To ensure a fair comparison, all models were evaluated using a data generator with fixed (identical for all) settings.
Table 4: Regularisation impact, measured by ARI (higher is better).
Cprob
Lyrob + Leep Cprob + Lep Sprob + Lsep + Cep
Real 44.80
51.60
48.65
$7.43
Syn. Gauss, 83.37
81.88 79.29
90,59
89.03
Syn. Transf.
79.85
88.58
86.33
Our main model, which during pre-training incorporates both regularizers, exhibits a clear perfor- mance advantage on real datasets and consistently holds second place for the synthetic ones. Table shows that model pre-trained using only Fprob and its variant with Seep included are insufficient for properly separating the transformer's representations for the clustering task, as they both struggle with clustering the synthetic datasets encountered during pre- pre-training. On the other hand, the results for Lyros - Cep imply a positive impact by the compact loss component, Lep, especially for synthetic data, which, however, does not translate to strong performance the OpenML benchmark. We conclude that all three loss components are necessary for good performance, with Lep being crucial for pre-training and Leep important for real data.
This mismatch between performance on real and synthetic datasets suggests a potential prior mis- specification, which is then mitigated by the regularizers. In particular, among the real datasets, there may be some that are not well explained by the data generation process used during pre-training. Hence, for future work, one may want to explore alternative mata-generating priors that may be more appropriate for these outliers.
3.5 What data-generating prior is optimal?
As mentioned in Section 2.2] constructing an appropriate data-generating prior p(D) is crucial for maximizing performance and generalizability of ZEUS. To validate this claim, we conduct an ablation study evaluating four prior designs: Gaussian Categorical, NN-transformed + Categorical, Gaussian NN-transformmed and Gaussian + NN-transformed Categorical (the standard ZEUS model). Each configuration name directly reflects the combination of priors used for pre-training (see Section B.2 in the Appendix for further details). For a fair comparison, we keep the default hyperparameter setup unchanged, except for the prior itself.
Table 5: ARI scores (averages; in rows) for ZEUS variants pre-trained on different priors (columns).
Gauss. + Cat.
NN-transf. Cat.
Gauss. NN-transf.
Gauss. + NN-transf. + Cat.
Real 40.59
50.90
52.00
57.43
Syn. Gauss. 92.61
89,90
75.25
89,03
Syn. Transf.
73.34
87.04
71.29
86.33
The results of the study are summarized in Table 5 As expected. the Gaussian + Categorical model achieves top performance on the Synthetic Gaussian datasets, while the NN-transformed Categorical model excels for the Synthetic Transformed data. Notably, the latter also performs well on the Gaussian sets, as its prior built upon transformed Gaussian mixtures. In contrast, the Gaussian + NN-transformed model yields relatively poor average ARI scores across both synthetic benchmarks. This drop is primarily caused by a significant decline in ARI performance on categorical datasets. Nonetheless, its average rank remains consistently competitive. Finally, for the OpenML. datasets, the best performance achieved when pre-training includes all three types of priors, underscoring the complementary importance of each prior in the training process.

---

4 Related work
Tabular data clustering. Traditional clustering algorithms, like k-means 223 GMM 7 or
hierarchical methods (26), have widespread applications across data mining, bioinformatics, customer segmentation, and anomaly detection. However, these methods often rely on predefined distance metrics and fail to capture complex, non-linear relationships, making them suboptimal for high- dimensional and heterogeneous tabular datasets.
pioneering Deep Embedded Clustering (DEC) 32 improves the target data representation by training the autoencoder and computing soft assignments in its latent space via Student's ' t-distribution. In a concurrent work [33]. the authors perform joint dimensionality reduction using AE and k-means clustering in the latent space. Then, improved DEC (IDEC) KK] extends DEC by jointly optimizing reconstruction and clustering objectives, while spectral variants replace k-means steps with graph- based updates [8] G-CEALS (27) replaces the t-distribution assumption with multivariate Gaussian clusters. DEPICT [10] attaches a softmax layer on top of an embedding network and trains with cross-entropy loss eliminate the assumption of an explicit distribution prior. Finally, IDC (30
predicts interpretable cluster assignments at the instance and cluster levels.
Most of these deep learning models require careful hyperparameter tuning and early stopping which
is unrealistic in the fully unsupervised setting due to the lack of labels 228] 29 Moreover, the optimization process has to be performed for each dataset, which is often time-consuming. Although multiple deep clustering approaches are currently in use, most of them are designed for texts or images 37 and cannot be directly adapted for tabular data due the lack of dominant neural architecture for heterogeneous tabular inputs [1, (24).
Representation learning for tabular data. Self-supervised learning (SSL) has been transformative in domains like vision and language [ but has struggled show similar success in tabular data
34 36.. A large diversity of data and a lack of pre-defined correlation between features hinder the
design of universal pretext tasks or augmentations, as well as the transfer learning between domains
On the other hand, in-context and zero-shot learning 4 enable the use of a single pre-trained model for multiple tasks out of the box, without any additional tuning. In particular, TabPFN treats small tabular datasets contexts consisting of features along with labels, and achieves state-of-the-art classification in a single forward pass [16]. Although thcoretical analyses reveal that transformers can implicitly implement algorithms such as gradient descent in context, their use is currently restricted to supervised problems [9]
35
5 Conclusion
In this paper, we presented ZEUS, a zero-shot transformer-base model that enables effective and efficient clustering of tabular data without the need for fine-tuning or extensive hyperparameter search. By pre-training on synthetic datasets with known latent structures, ZEUS learns generalizable representations that help simple clustering algorithms to uncover meaningful structures. Our experi- ments show that ZEUS consistently matches or outperforms both classical and deep learning based clustering methods, offering practical and theoretically grounded solution for unsupervised analysis of tabular data.
Limitations. Since ZEUS is technically based on the TabPFN architecture, it inherits some of its drawbacks: a maximum number of input features and samples. However, the recently introduced TabPFN v2 (17) showed that tabular transformers can process larger datasets with negligible increase in computational time. Moreover, our experimental results demonstrate that even the dimension of input data exceeds the fixed value of 30, applying PCA does not significantly hurt the clustering performance.
The final performance of ZEUS heavily relies on the synthetic data used in pre-training While we release a basic version of ZEUS, which encodes certain assumptions about data clusters, one can adjust the pre-training stage using different datasets. Finally, ZEUS does not cluster data itself, but constructs a convenient embedding space in which clustering can be performed using basic algorithms.
10

---

Acknowledgments and Disclosure of Funding
We are grateful to the Reviewers for their effort and insightful comments on the paper.
This research is part of the project No. 2022/45/P/ST6/0290 co-funded by the National Science Centre and the European Union Framework Programme for Research and Innovation Horizon 2020 under the Marie Sklodowska-Curie grant agreement No. 945339. For the purpose of Open Access, the authors have applied CC-BY public copyright licence to any Author Accepted Manuscript (AAM) version arising from this submission.
The research of Marszalek and M. Smieja was supported by the National Science Centre (Poland), grant no. 2023/50/ESTT6/0019 The research of Witold Wydmański was supported by the Ministry of Science grant no. PN/01/0195/2022 and NCN Sonata BIS grant number 2020/33////22/000586 The research of J. Tabor was supported by the National Science Centre (Poland), grant no. 2023/49/B/ST660113.. Some experiments were performed on servers pur- chased with funds from the flagship project entitled "Artificial Intelligence Computing Center Core Facility" from the DigiWorld Priority Research Area within the Excellence Initiative Research University program at Jagiellonian University in Krakow. We also gratefully acknowledge Polish high-pes perforerformecce computing infrastructure PLGrid (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PL.G/224/017899.
References
[1] D. Bahri, H. Jiang, Y. Tay, and D. Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. International Conference on Learning Representations, 2022.
[2] J. Behrmann, W. Grathwohl, R. T. Chen, D. Duvenaud, and J.-H. Jacobsen. Invertible residual networks. In International Conference Machine Learning, pages 573-582, PMLR, 2019.
[3] B. Bischl, G. Casalicehio, T. Das, M. Feurer, S. Fischer, Gijsbers, S. Mukherjee, A. C. Müller, L. Németh, L. Oala, L. Purucker, S. Ravi, J. N. van Rijn, P. Singh, J. Vanschoren, J. van der Velde, and M. Wever. Openml: Insights from 10 years and more than a thousand papers. Patterns, 101):101
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
[5] S. Cai, L. Qiu, X. Chen, Q. Zhang, and L. Chen. Semantic-enhanced image clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1869-6878, 2023.
[6] Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607, PMLR, 2020.
[7] A. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: series B (methodological), 39(1):1-22, 1977.
[8] L. Duan, C. Aggarwal, S. Ma, and Sathe. Improving spectral clustering with deep embedding and cluster estimation. In 20/9 IEEE International Conference on Data Mining (ICDM), pages 170-179, 2019.
[9] S. Garg, D. Tsipras, P. Liang, and G. Valiant. What can transformers learn in-context? a case study simple function classes. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, 2022.
[10] K. Ghasedi Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE International Conference on Computer Vision, pages 5736-5745, 2017.
11

---

[11] J.-B. Grill, F. Strub, F. Altché. C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new ap- proach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271-21284, 2020.
[12] I.. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Advances in Neural Information Processing Systems, volume 35, pages 507-520, 2022.
[13] X. Guo, L.. Gao, X. Liu, and J. Yin. Improved Deep Embedded Clustering with Local Structure Preservation. In Proceedings of the Twenty-Sixth International Joint Conference Artificial Intelligence, IJCAI-17, pages 1753-1759, 2017.
[14] X. Han, D. Simig, T. Mihaylov, Y. Tsvetkov, A. Celikyilmaz, and T. Wang. Understanding in-context learning via supportive pretraining data. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12660-12673. Association for Computational Linguistics, 2023.
[15] L. P. Hansen. Large sample properties of generalized method of moments estimators. Econo- metrica: Journal of the Econometric Society, pages
1982.
[16] N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. The Eleventh International Conference on Learning Representations, 2023.
[17] N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, S. B. Hoo, R. T. Schirrmeis- ter, and F. Hutter. Accurate predictions small data with tabular foundation model. Nature, 637(8045):319-324
2025.
[18] Q Jingang. D. Holzmüller, G. Varoquaux, and M. Le Morvan. TabICL: tabular foundation model for in-context learning on large data. In Forty-second International Conference on Machine Learning, 2025.
[19] H. W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-97, 1955.
[20] H. W. Kuhn. Variants of the Hungarian method for assignment problems. Naval Research Logisties Quarterly, 3(4):253-255) 258, 1956.
[21] J. Lee, A. Xie, A. Pacchiano, Y. Chandak, C. Finn, O. Nachum, and E. Brunskill. Supervised pretraining can learn in-context reinforcement learning. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[22] Y. Li, Hu, D. Peng, J. Lv, J. Fan, and Peng. Image clustering with external guidance. In International Conference on Machine Learning, pages 27890-27902. PMLR, 2024.
[23] MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Statistics, volume 5, pages 288-298. University of California press, 1967.
[24] D. McElfresh, S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White. When neural nets outperform boosted trees on tabular data? In Advances in Neural Information Processing Systems, volume 36, pages 76336-76369, 2023.
[25] Müller, N. Hollmann, S.P. Arango, Grabocka, and Hutter. Transformers can Bayesian inference. In International Conference on Learning Representations, 2022.
[26] Murtagh and Contreras. Algorithms for hierarchical clustering: an overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1):86-97, 2012.
[27] S. Rabbani, V. Medri, and M. D. Samad. Deep clustering of tabular data by weighted Gaussian distribution learning. Neurocomputing, 623:129359, 2025.
12

---

[28] M. Sadeghi and N. Armanfard. IDECF: Improved deep embedding clustering with deep fuzzy supervision. In 2021 IEEE International Conference on Image Processing (ICIP), pages 1009-1013, 2021.
[29] M. Sadeghi, S. Soleimani, and N. Armanfard. Deep clustering with self-supervision using pairwise similarities. JEEE Access, 2025.
[30] Svirsky and O. Lindenbaum. Interpretable deep clustering for tabular data. In Proceedings of the s1 International Conference on Machine Learning, volume 235, pages 47314-47330. PMLR, 21-27 Jul 2024.
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[32] J. Xie, R. Girshick, and A. Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, pages
PMLR, 2016.
[33] B. Yang, X. Fu, N. D. Sidiropoulos, and M. Hong. Towards k-means- --means-riendll spaces: Simulta- neous deep learning and clustering. In International Conference on Machine Learning, pages 3861-3870. PMLR, 2017.
[34] J. Yoon, Y. Zhang, J. Jordon, and M. Van der Schaar. Vime: semi-supervised learning tabular domain. In Advances in Neural Information Processing Systems, volume 33, pages 11033-1 1003, 2020.
Extending the success of self-and
[35] Z. Zhao, L. Alzubaidi, J. Zhang, Y. Duan, and Y. Gu. A comparison review of transfer learning and supervupe learning: Definitions, applications, advantages and limitations. Expert Systems with Applicutions, 242: 122807, 2024.
[36] B. Zhu, X. Shi, N. Erickson, M. Li, G. Karypis, and M. Shoaran. XTab: cross-table pretraining for tabular transformers. In Proceedings of the 40th International Conference on Machine Learning, pages 43181 142204,, 2023.
[37] M. Znalezniak, Rola, P. Kaszuba, J. Tabor, and M. Šmieja. Contrastive hierarchical clustering. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 627 647.643.660 Springer, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions: (1) the introduction of ZEUS, a transformer-based model for zero-shot clustering of tabular data; (2) the use of synthetic datasets with known latent-variable structure for pre-training; and (3) strong empirical results on both real and synthetic datasets. These claims are consistently substantiated in the methodology, experiments, and analysis sections. The scope and limita- tions (e.g., dimensionality constraints, reliance on synthetic priors) are also acknowledged in the conclusion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes dedicated Limitations paragraph in the conclusion, where it explicitly discusses constraints related to model capacity (e.g.. fixed number of features and samples inherited from TabPFN), reliance on synthetic data for pre-training, and the fact that ZEUS generates embeddings but does not perform clustering end-to-end. It also notes the potential for prior misspecification and suggests that the performance may vary depending on how well the synthetic priors match real-wor wolld data. These limitations are realistic, clearly acknowledged, and contextualized in terms of potential extensions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
14

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not present formal theorems, lemmas, or propositions with accompanying proofs. While it includes a theoretical framing (c.g., the connection to Prior-
Data Fitted Networks and Bayesian inference), these presented conceptual motivations rather than formal theoretical contributions. There are no formal mathematical results stated proven.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides comprehensive details necessary for reproducing the main results. It specifies the model architecture, the training procedure (data generation process, optimizer, learning rate, batch size, number of epochs), the inference pipeline (standardization, PCA, normalization, clustering algorithm), and the datasets used (OpenML datasets, synthetic datasets with known priors). also includes evaluation metries, and explicit descriptions of ablation setups. We also provide link to an anonymized code repository.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
15

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The paper includes a link to an anonymized code repository. The repository provides all necessary components to reproduce the main results: the model definition, synthetic dataset generator, training pipeline, preprocessing scripts, and evaluation code. Real-world datasets are sourced from OpenML. (publicly accessible), and synthetic datasets can be generated using the provided scripts.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public//uides/Codeen Coadddess/oodeeSumsisiiiiooo.o
stt.uuummmiiiioooooooooooooooiil for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides extensive details about the experimental setup in Section 3 It specifies:
Model architecture
16

---

Optimizer and learning rate, as well as use of cosine scheduling with warm-up
Dataset preprocessing steps (standardization, PCA for high nigh-dimensional inputs, one-hot for categorical features)
Evaluation metrics (ARI and Brier score)
These details are enough interpret the results and, needed, reproduce them, Additional implementation details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While we mention that all results are averaged over 5 random seeds (c.g., in Tables I
we do not report any error bars, standard deviations, or confidence intervals. The source variability (random seed) is implicitly mentioned, but no statistical significance tests or variability estimates are provided in figures or tables
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification; While the paper presents extensive experimental results, does not specify the total compute cost.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hattps://neur/ httpe//eerris.ccppppeiieeeeeeeee Ceuublic/EticsGuaie Etwhiccuudeeieeee
Answer: [Yes]
The research focuses on unsupervised learning for tabular data using synthetic datasets and publicly available real-world datasets from OpenML. It does not involve any human subjects, sensitive personal data, or models with known societal or environmental risks. The synthetic data generated in controlled, fully artificial manner, and real datasets are used strictly for benchmarking in accordance with their licenses. The research is aligned with the NeurIPS Code of Ethics regarding reproducibility, transparency, and responsible development of machine learning models.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes a Broader Impact section in the appendix, which discusses both the potential benefits and risks of using ZEUS.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
18

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pre-trained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: ZEUS's pretraining utilizes exclusively synthetic datasets, and thus doesn't introduce any risk of misuse. It is not capable of generating human-likk content, nor does process natural language, images, or other modalities associated with misuse risk. As such, there is no high-risk component requiring safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The paper cites and builds upon the publicly available TabPFN v1 codebase, which is correctly credited and stated to be released under the Apache 2.0 License. All real-world datasets are obtained from OpenML.org and are explicitly described as open data, with the acknowledgment that individual datasets may have licenses such as CC-BY. This information clearly documented in the paper in the "Licensing and Third-Party Assets" section of the appendix. The code released by the authors is also provided under the Apache 2.0 License, and appropriate instructions for reproduction are included.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, snversswihcode com/clccooaoooooo com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
19

---

The paper introduces new model, ZEUS, and synthetic data generation pipeline used for its pre-training. Both are made available in the anonymized code repository linked in the main text. The repository includes:
Justification:
code for the model architecture, training loop, and inference,
scripts for generating synthetic datasets,
configuration files for hyperparameters and preprocessing.
usage instructions (e.g., via a README file).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
The paper does not involve any crowdsourcing or research with human subjects. All experiments are conducted on synthetic datasets or publicly available tabular datasets from OpenML, with no human annotation, survey, or data collection involved.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does involve neither crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
20

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The research presented in this paper does not involve large language models (LLMs) as part of the methodology. experiments, or any original contribution. ZEUS is transformer-based model trained synthetic tabular data for zero-shot clustering, and all components are specific to structured data. No LLMs were used in model design, training. inference, evaluation, or theoretical framing. Therefore, no declaration under the NeurIPS LLM policy is required.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
21

---

ZEUS: Zero-shot Embeddings for Unsupervised Separation of Tabular Data supplementary material
A
Background on Prior-Data Fitted Networks
The recently introduced framework of Bayesian inference through transformers demonstrates that neural networks pre-trained on synthetic datasets implicitly approximate Bayesian inference without explicitly computing posterior distributions [25 [6] The pre-trained transformers are known as Prior- Data Fitted Networks (PFNs). The pre-training involves synthetic prior fitting, wherein a transformer network is trained offline on numerous datasets generated from a predefined prior distribution over tasks. This pre-training procedure allows the transformer to implicitly encode Bayesian posterior predictive distribution by optimizing a cross-entropy loss between its predictions and the synthetic data labels.
Formally, pre-training is carried out as follows: given prior distribution p(D) over (synthetic) datasets D, a PFN network parameterized by is trained to minimize the negative log-like oggliielihood of predicting held-out labels within datasets sampled from this prior. The associated loss function, known as the Prior-Data Negative Log- Log-Likelihood (Prior-Data NLL), is explicitly defined as:
(PF = (E=.,u((ew) log go (ula, Detx));
where each dataset Det and data point (x, v) are sampled from the predefined prior distribution p(D). As shown formally in (25), minimizing this loss is mathematically equivalent to minimizing the expected cross-entropy between the predictive distribution qo(v) (,, Deta) and the true posterior predictive distribution (PPD) derived from the prior. Specifically, minimizing the Prior-Data NLL is equivalent minimizing the expected Kullback-Leiblee divergence between the network's 's predictive distribution and the true PPD p(yl Deta):
(0) (D
Do Dot,)))
where H denotes the cross-entropy. Thus. optimality the predictive distribution qe implies matching it exactly to the true Bayesian posterior predictive distribution, provided that the parametric family of distributions defined by the transformer is sufficiently expressive (25].
Within this in-context learning as Bayesian inference framework, transformers approximate Bayesian averaging implicitly. Given training dataset Dat = {x;, y/) y1))
input æ, and prior over hypotheses o, the Bayesian posterior predictive distribution is formally expressed as:
1 )
/ p(u x
ply x. Detz)
integrating all hypotheses e t. PFNs approximate this integral implicitly through pre-training on prior-generated datasets, yielding a distribution (y(a,, Dat.) directly from forward propagation
of the network conditioned on dataset Pata 25 16
B Details of synthetic data generation process
B.1 Theory for synthetic-to-real data generalization
Theorem 1 (Univeral Approximation Theorem for Mixture Distributions). Let P E T.P: be mixture of distributions on Rd, where each P, is a probability distribution, and n. > 0. -
Then there exists mixture of Gaussians Q = (HN E,) and neural network F such that we can approximate with arbitrarily small error by the pushforward measure F4Q. Additionally, we can select F so that Fg N(Ja, Ex) approximates P; with arbitrarily small error.
Proof. We proceed in several steps.
Step 1: Approximation of individual components.
Making use of the universal approximation theorem, for each component P; there exists neural network F. which transforms arbitrary Gaussian Q2 A(4, E,) into P., i.c. Fi se P..
22

---

Step 2: Constructing unified neural network.
We now define single neural network that can represent all the F; networks. Define F Rd x {1, .k) - IR such that F(z,i) i) R Fz(z). This can be implemented by encoding the index i as one-hot vector or embedding and concatenating to 2, enabling F to learn the behavior of each
F, conditioned on the index i. The network F can be extended to the domain Rd x IR in an arbitrary way.
Step 3: Approximation of the full mixture.
Since each ( 2,) approximates P, the mixture F#g ++Q= - TTPyN(1s E,) approximates P 1 T.P.
Thus, we have constructed a mixture of Gaussians Q and a neural network F such that Fg Q approximates P, and each component TtNN(ns, Ex) approximates Pt arbitrarily well. This completes the proof.
B.2 Insights into data-generating priors
We use three types of probabilistic procedures to generate the prior data:
Gaussian. The goal to create a continuous data type where each cluster follows a multivariate Gaussian distribution with carefully designed means and covariance matrices. This construction process incremental. Starting with a mixture containing & components, the addition of a new cluster involves initially placing it at position 0 and then shifting it in randomly chosen direction. The covariance structure for each Gaussian is generated through cigendecomposition, with eigenvalues sampled from predefined range of (0.005, 0.05] to control the shape and orientation of the clusters. Additionally, to guarantee the presence of both full-rank Gaussians in higher dimensions and degenerate ones in lower dimensions, extra conditions are introduced to narrow the range of eigenvalues. These constraints are activated with probabilities of 0.25 and 0.2, respectively. To ensure adequate separation between clusters and to prevent trivial overlap, we apply a Wasserstein-2 distance constraint, requiring that the means μ4 and μj of any two clusters, along with their corresponding covariance matrices D; and Mij are separated by at least threshold T,, as follows:
||4 - 1 tr(E, + E, و
The exact minimum distance value T varies between 0.5 and 1.0 and is randomly selected for each component independently in order promote data diversity. This entire approach allows us to model a variety of cluster geometries, from spherical to highly elongated elliptical shapes.
2. Categorical. To enrich the generated data, we incorporate categorical features alongside continuous ones by sampling from categorical distributions that are biased toward certain categories for each cluster. To produce these varied categorical probability patterns, we use the Dirichlet distribution. The probability of including categorical features is controlled parameter, set to 0.3. Up to 3 categorical variables may be max__aaaaeaaaae categories possible values, defined as 5. The resulting categorical variables are then converted to one-hot encoding and combined with the continuous features, producing mixed-type datasets that more closely resemble real-world tabular data.
by a ciategorical_hhanee added, each having between 2 and max
3. NN transformed. To create more complex, non-linearly separable cluster structures, we apply transformations the numerical features using random neural networks with 3 layers. These transformations map the original data through several non-linear operations while preserving cluster identity information, producing datasets with more challenging decision boundaries. Following the approach of Invertible Residual Networks 22) we constrain the spectral norm of each transformation layer to be less than In addition, standardization is applied between residual layers to ensure more stable transformations. To help preserve cluster separability, we append one-hot vectors indicating the component identity to the numerical variables before passing them through the random neural network. A After the transformation, these extra dimensions are removed using the PCA algorithm, restoring the data to its original dimensionality. This comprchensive approach enables the
23

---

simulation of intricate, non-linear data manifold structures often presented in real-world clustering problems without introducing degenerate configurations.
All features undergo standardization and scaling to ensure numerical stability during training. Contin- uous features are normalized to the range [-1, I]. while ensuring that the relative separation between clusters is preserved.
By training on this diverse collection of synthetic datasets-ranging from well-separated Gaussian clusters to complex, transformed manifolds with mixed feature types ZEUS learns to identify meaningful cluster structures across wide speetrum of data distributions. This enables adapt to previously unseen datasets at inference time without additional training.
C Pre-training plots
Synthetic Gaussian
Synthetic Transformed
2
83
70
Epoch
Epoch
Figure 4: Visualization pre-training process
Figure presents two plots of average ARI over 200 synthetic validation datasets throughout 300 pre- nee-training epochs. The first plot corresponds to the crussian-categorical datasets referred as Synthetic Gaussian, while the second illustrates their NN transformed variants, named Synthetic Transformed. The data generation procedures are described more detail Appendix [B.2] Both plots illustrate that the quality of the ZEUS representation improves over time, indicating that the model learning new patterns, as evidenced by the increasing ARI during the pre-training process.
D Statistics of datasets used in experimental study
Tables and 8 provide detailed information about the datasets used in the experimental analysis. Each table includes the number of set instances, the number of categorical and numerical features, the total dimensionality, and the overall number of categories, represented by the length of the one-hot encoded vectors. Additionally, the tables report the number of classes that each dataset contains. In Table the ID column corresponds the OpenML ID, while in the remaining tables functions solely as an index.
24

---

Table Real/OpenMI. datasets statisties.
ID
# Instances
Numerical features
Categorical features
Dimension
Categories (one-hots)
Classes
14
2000
76
76
10
699
2
16
2000
64
64
0
10
18
2000
O
10
22
2000
47
47
0
10
35
366
33
130
129
6
51
294
25
19
2
53
270
13
13
0
2
56
435
32
32
2
61
150
4
4
0
3
187
178
13
13
3
377
600
60
60
0
6
458
841
0
4
481
209
14
7
2
694
310
0
9
721
200
10
10
0
2
0
733
209
2
745
159
14
20
6
2
2
756
159
15
15
796
209
6
36
30
2
2
820
235
12
840
205
17
68
51
2
854
158
14
9
2
1462
1372
4
2
1495
250
18
18
2
1499
210
7
0
3
1510
569
30
30
0
2
1523
310
3
4153
180
66
66
0
6
40496
500
10
40682
215
5
5
0
3
40705
959
44
2
42261
150
4
4
0
3
42585
344
3
25

---

Table 7: Synthetic Gaussian datasets statistics.
ID
Instances
# Numerical features
# Categorical features
Dimension
# Categories (one-hots)
Classes
0 1337
16
0
16
8
1
1383
23
0
23
2
1421
8
2
14
7
3
992
12
4
1314
8
0
8
0
5
1497
23
6
1370
2
0
2
0
6
7
1646
3
16
8
1520
11
0
11
8
9
1537
0
18
10
825
26
0
26
2
11
1112
0
9
12
1093
15
0
15
8
13
742
0
6
14
1595
11
2
18
15
1417
14
0
14
6
16
1787
28
28
17
764
19
0
19
4
18
889
25
25
19
1660
28
0
28
9
Table 8: Synthetic Transformed datasets statistics.
ID
Instances
# Numerical features
# Categorical features
Dimension
# Categories (one-hots)
Classes
0 1337
16
0
16
8
1
1627
30
0
30
2 1631
11
O
11
7
3
1891
17
10
4
1142
3
O
3
0
4
5
1222
24
0
24
6
953
6
0
6
6
7
1508
0
9
10
8
840
7
3
15
8
5
9
1745
14
10
1618
23
0
23
6
11
1432
0
13
9
12
1860
9
0
9
9
13
563
10
0
10
14
1033
6
2
13
15
750
0
2
16
1451
14
2
20
10
17
679
30
0
30
2
18
859
22
25
2
19
1493
11
0
11
7
26

---

E
Baselines
The evaluation of baseline models is based on the following libraries and GitHub repositories:
1. scikit-learn used for k-means and GMM,
2.
https///fthbbbaoomyyyyaaaaannnee imm.edeeetiiimlllennttttttiioo of the DEC,
3. http:://thhaa.coomddddw/ng////gt source code for the IDEC,
4.
https:///gihubb.com///ivrrriddd- official inplennntoonnnnnnnnnntttttoo of the IDC method,
5. https:/ghhhbb.ooomnnddaaatttrdl- http:///ssaaaauooooooaoaeoeeeee- Deep-CCusteriiggffo-Taaaaae Taaugaar-aaTaa code-
base for the GCEALS,
6.
library that extends TabPFN function- ality to wide spectrum of machine learning tasks, including unsupervised ones,
7. http::/gghhlll.oommdddrrgreeeyer
code repository for SCARF.
To ensure fair comparison, hyperparameters chosen to maximize the performance of each method with respect to their overall average rank across 5 random seeds. For this reason, all numerical features preprocessed using standard scaler prior to the training phase.
Most parameters of the k-means and GMM methods are left their default values. Only the n-inil option was inereased to 100 for k-means and 50 for GMM in order to improve stability.
As mentioned in Section 3.1 a standard autoencoder with hidden layers [500, 500, 2000] is used for each AE-based method. The resulting network is first pre-trained for 1000 epochs and then fine-tuned for up to 1000 additional epochs, separately for each dataset and model. Following the GCEALS evaluation procedure, multiple latent dimension sizes [5, 10, 15, 20] were tested across all considered methods. The results indicate that the default latent dimension of does not yield the best performance; instead, dimension of 20 generally performs better. Changing other hyperparameters, including the learning rate, optimizer, and clustering loss weight 7, generally did not lead improved results. Therefore, the remaining parameters were left at their default values as proposed by the authors of the respective codebases.
For TabPFN, we use the get_ sie_-mmmeddng__pe per_____________eee____e column method from the TibePuuusuperriseMMdeel class, provided in the TabPFN Extensions repository to extract per-column representations. These embeddings are then averaged across columns to obtain the final representation for each data point. For SCARF, we follow the procedure outlined in the example. notebook from the original repository
3T In both cases, we cluster the learned representations using k-means, consistent with our approach in ZEUS.
F Extended experimental results
This section presents extended tables corresponding to the experiments described in Section
F1 How effective is ZEUS for clustering?
Tables 10
contain the complete results for individual datasets from the experiments discussed in Section 3.2 T as presented in the main text. The reported values represent averages of ARI over 5 random seeds. In addition the rows presenting outcomes for specific datasets, the tables include summary rows labeled Mean, Mean-Rank, Top-3, and Top-1. These represent, respectively: the average ARI, the average rank computed across all datasets, the number of times given model appears in the top 3, and the total number of wins achieved by each method. is worth noting that the Top-1 1 and Top-3 rows indicate clear wins and clear appearances in top-3 positions.
The overall conclusions align with those presented in the primary analysis. One noteworthy point is that ZEUS achieves strong performance in terms of the Top-3 and Top-1 statistics, being outperformed this regard only in T0] where k-means and GMM score better. Another interesting observation
27

---

is that GMM struggles with categorical features, which significantly worsens its average scores in Tables TO and
Table 9: Comparison of clustering quality using ARI metric on OpenML datasets (higher score indicate better performance).
GMM
AE-KM
AE-GMM
DEC
IDEC
G-CEALS
TabPFN
SCARF
ZEUS
14
38.74
45.90
40.94
45.36
49.60
36.78
45.03
45.04
22.02
1.90
50.56
71.38
16
55.62
64.75
61.66
69.97
68.56
55.61
65.84
55.27
14.69
3.74
74.03
54.86
46.88
51.45
51.47
53.54
49.93
51.50
51.63
22
35.48
47.39
31.09
50,80
50.02
35.69
46.05
28.38
11.09
2.63
56.05
35
70.68
70.68
70.22
70.48
68.63
64.58
77.99
56.57
52.24
74.61
85.12
51
28.35
-2.54
37.82
34.13
38.13
34,58
38,93
35.13
38.26
-1,27
43.66
53
45.21
5.17
39.14
40.62
43.50
41.91
33.03
21.32
26.61
24.24
35.76
61
62.01
90.39
59.79
60.21
$7.84
60. 14
$8.30
51.37
77.20
51.19
85. 15
187
93.09
82.87
84.86
85.02
85
80.04
55.67
37.55
40.05
377
56.70
58.47
62.06
59.52
63.03
59.01
62.39
59.79
66.89
14.45
55.30
458
95.08
96.97
64.33
75.70
94.58
63.29
86.77
68.31
61.41
2.87
99.19
58.51
2.77
36.84
48.03
$7.28
52.93
43.17
19.67
2.77
1.79
8.68
694
35.91
43,30
35.26
36.56
26.30
36.02
33.47
30.76
27.68
14.09
33.38
733
50.97
46.29
38.39
55.11
67.47
73:51
68 88
26,26
-8.35
49.41
74.97
745
57.63
55-44
78.35
75.52
46.37
50.94
$9.29
$7.81
1.26
16.75
73.85
756
55,70
4.46
72.81
$3.40
58.60
65.58
53.32
44.14
10.60
64.82
50.31
796
49.62
2.26
76.54
52.19
50.07
55.92
79.59
43,40
0.88
58.25
13.99
820
50.16
36.77
44,29
49.24
51.21
49.30
44.59
24.51
50.26
13.04
28.87
840
39.05
48.37
53,60
46.72
49.14
39.16
19.59
27.95
8.89
32.13
27.00 66.66
-0.21
70.52
1462
1.32
0.31
10.03
10.17
1.00
3.80
8.83
8.48
3.31
-0.03
92.28
1495
96.81
98.40
72.74
82,80
96.81
91.28
83.96
77.51
47.11
5.10
7.35
1499
77.33
62,99
73.68
75.28
69.99
78.48
67.82
29.60
5.82
24.93
82.40
1510
67.07
78.02
63.73
59.77
71.32
69.84
70.55
69.47
54.59
63.57
74.26
1523
41.21
25.48
24.57
28.16
23.05
18.50
19.60
4153
55.78
58.03
59.15
46.23
56.41
58.44
30.44
39.43
27.80
36.21
62.29
54,02
34.79
42.35
37.51
49.74
42.35
27.41
37.35
39.78
32.25
40682
58.32
86.29
57.89
59.22
57.39
86.05
17.95
40.58
86.94
17.86
53.15
40705
43.49
37.92
0.73
30.63
47.90
46.28
31.38
10.71
-4.40
32.90
45.45
42261
62.01
90.39
60.13
58.15
59.78
56.78
55.10
49.09
77.20
52.53
85.15
42585
60.84
30.52
51,26
60.71
91.79
43.40
72,10
38,30
23.04
3.50
95.05
Mean
55.54
48.49
51,43
53,56
$5.93
$4.57
52.28
40.37
31.32
26.95
$7.43
Mean-Rank
4.69
5.65
5.72
5.24
4.69
5.01
5.62
8.18
8.22
8.85
4.13
Top-3
12
15
11
1
21
Top-1
2
12
28

---

Table 10: Evaluation of clustering quality with the ARI metric on Synthetic Gaussian datasets (higher scores reflect better performance).
ID
KM
GMM
AF.KM
AF-GMM
DEC
IDEC
IDC
G-CFAIS
TahPFN
SCARF
7FUS
86.16
82.13
83.44
79.66
$7.06
77.26
60.64
58.57
65.42
4.06
87.65
88.66
87.36
81.13
79.61
85.84
75.62
44.44
54,10
40.26
4.21
72.04
81.30
26.46
71.57
78.04
86.65
78.91
53.53
48.59
20.42
82.96
95.02
36.77
92.03
76.44
92.74
89.81
65.03
62.67
50.57
22.29
95.76
92.97
77.80
84.51
69.70
89.83
25. 85
86.52
69.06
90.01
88.31
70.38
67.70
20.51
10.31
90.72
96.21
98.07
85.99
94.26
94,24
92.39
80.18
73.15
86.50
6.51
89
92
30.52
81.73
79.06
91.35
89
46
58,10
26.62
10.14
93.60
88.29
89.62
81.15
76.53
87.56
78.37
72.72
$0.30
68.54 54.54
13.06
00
93.36
98.54
68.58
85.29
76.64
56.89
65.02
94.55
10
79.26
93.32
72.75
90,90
72.54
73.51
77.64
48.47
89.15
-0.04
87.32
93.60
98.48
90.77
89.11
95.03
92.19
64.24
2.11
97.45
12
86.82
77:45
73.64
71.40
86.04
77.93
60.55
58.45
55.27
19.11
86.04
13
96.77
98.07
94.05
90.15
96.11
85.66
86.14
78.20
80.24
5.29
96.17
14
90.73
43,20
79.65
69.09
92.68
77.93
72,80
67.84
12.93
8.16
94.60
15
93.58
96.23
85.74
89.30
94.43
89 41
72,54
72.88
91.02
3.24
96.68
92.45
85.86
4.85
87.19
17
85.26
87.10
62.25
79.52
84.08
75.00
50.45
42.55
50.06
7.69
86.29
88.77
94.83
84.47
84.74
87.39
83.46
61.36
67.96
74.90
7.69
83.01
19
86.54
89.11
76.18
78.46
86.06
67.99
47.46
70.50
32.39
6,02
73.85
89.90
76.93
81.26
81.40
89.35
82.57
66.43
62.84
55.97
8.32
Mean-Rank
2.65
3.65
3.65
5.60
3.23
5.70
7.95
8.90
8.75
11.00
2.92
Top-3
13
15
14
Top-1
10
0
Table 11: Assessment clustering quality based on the ARI metric on Synthetic Transformed datasets (higher score indicate better performance).
GMM
AE-KM
AE-GMM
DEC
IDEC
G-CEALS
TabPFN
SCARF
ZEUS
60.56
79.33
46.31
68.98
76.93
46.35
56.67
29.42
6.36
0.94
87.42
96.55
70.91
$9.40
89.76
96.31
57.79
65.99
94.42
66. 36.34
72.25
48.34
10.00
1.10
95.88
70.16
50.38
56.23
83.06
60.09
54.54
80,98
44.32
80.63
47.60
44,77
68.39
50.12
36.00
49.37
19.14
1.48
73.66
91.49
96.51
74.48
89.11
86.46
77.84
60.00
66.47
15.12
1.25
81.63
60.98
81.63
61.97
76.29
75.58
61.89
69.43
69.84
18.83
2.11
87.02
76.80
83.98
71.37
77.69
86.56
77.45 49.49
76.75
56.51
18.94
3 2.20
89.57 57
84.40
98.48
82.64
93.03
97.14
81 02
91.02
65.62
9.01
1,66
97,20
82.47
91.91
61.00
86.16
87,58
46.08
70.42
39.27
14.11
86.42
11
90.77
99.32
81.41
91.18
94.78
88.12
80,75
70,29
18.31
1.72
93.19
12
67.64
91.80
52.87
63.56
85.45
60.61
73.66
55.59
28.50
3.21
87.75
85.99
50.80
45.48
35.34
29.04
12.91
30.22
32,89
35.78
6.01
91.60
14
93.52
54.56
15.81
22.79
94.12
38.21
68.54
15.10
13.08
5.54
98.49
16
95.62
65.11
90.13
94,97
97.47
95.32
84,79
98.27
17.27
9.58
99.01
17
60.54
95.24
56.24
91.71
78.72
59.11
82,54
13.29
7.33
-0.10
96.44
18
63.92
17.39
58.47
70.21
54,61
47.93
64.15
10.01
3.79
0.65
79.70
19
62.12
79.06
49.36
65.77
81.07
51.19
63.41
26.84
2.73
1.06
84.13
Mean
75.04
75.86
60.45
71.29
79.94
61.26
66,78
49.17
15.66
2.48
86.33
Mean-Rank
3.50
6.85
4.50
3.20
6.35
6.05
7.80
9.75
11,00
2.20
Top-3
14
15
2
1
0
17
Top-1
8
0
10
F.2
Are ZEUS's assignments well calibrated?
Tables 12 13 and contain detailed extension of Table 3 the main part of the paper. Similar to
the Tables in Appendix F.I these also include additional rows: Mean, Mean-Rank, Top-3, and Top-1, which aggregate the results presented in each table.
ZEUS consistently attains at least the second position across statistics every table presented in this section. For the Synthetic Transformed datasets, it is undeniably the best. However, for the OpenML datasets, is outperformed by IDEC, and for the Synthetic Gaussian datasets, GMM takes the lead in the Top-1 metric, while for the other statistics, k-means performs better.
29

---

Table 12: Soft clustering performance of ZEUS versus competing methods on real- datasets, measured by Brier score (lower is better).
ID
KM
GMM
AE-KM
AE-GMM
DEC
IDEC
G-CEALS
ZEUS
14
0.7660
0.8579
0.8240
0.7144
0.5338
0.6409
0.7993
0.7006
15
0.0858
0.1548
0.2100
0.1791
0.1489
0.0644
0.2003
0.0973
16
0.5012
0.4982
0.4224
0.5372
0.3726
0.5020
0.6099
0.2761
0.6060
0.8529
0.7312
0.6961
0.6099
0.5354
0.6855
0.7094
22
0,8316
0.7533
1.0686
0.7684
0.5071
0.6496
0.7201
0.7173
35
0.0710
0.5016
0,4678
0.5711
0,5881
0.3987
0,3836
0.2552
51
0.3741
0.3741
0.3687
0.3844
0.3297
0.3345
0.3997
0.3333
53
0.4074
0.7052
0.3556
0.6285
0,3361
0.2979
0.5234
0.4000
0.2391
0.2344
0.2520
0.2703
0.2147
0.2088
0.2517
0.1829
61
0.2267
0.0570
0.3600
0.3796
0.3881
0.3445
0.4167
0.0979
187
0,0899
0.0622
0.1349
0.1146
0.3380
0.0785
0.2518
0.0762
377
0,8633
0.8320
0.6200
0.6275
0.5316
0.5047
0.5739
0,8508
458
0.0285
0.0214
0.5222
0.3407
0.3203
0.3656
0.4104
0.0048
481
0,8038
0.8134
0.4536
0.2136
0.3111
0.2146
0.4422
0.6986
694
1.0155
0.8831
0.9858
0.9342
0.7616
0.7525
0.7038
0.7621
721
0.3400
0.3400
0.6620
0.6031
0.4491
0.4359
0.7937
0.3400
733
0.1722
0.2859
0.3082
0.4937
0.2115
0.1129
0.3195
0.1244
745
0,2516
0.2516
0.1359
0.2766
0.3016
0.1793
0.2388
0.1384
756
0,2767
0.3615
0.2113
0.3262
0.3178
0.1329
0.4134
0.2767
796
0.0574
0.7943
0.1340
0.3088
0.2046
0.3002
0.5269
0.6253
820
0.2979
0.3720
0.3557
0.2876
0.2334
0.2594
0.3981
0.4596
840
0,8780
0.2790
0.2868
0.2625
0.2746
0.2445
0.4642
0.4665
854
0.6329
0.8203
0,3468
0.4481
0.4469
0,3372
0.3678
0.1266
1462
0,8484
0.8978
0.7688
0,8310
0.6674
0.7714
0.6747
0.0394
1495
0.0160
0.0080
0.1696
0.0397
0.2667
0.0713
0.1583
0.7200
1499
0.2190
0.2933
0.1867
0.1961
0.3939
0.1376
0.4063
0.1239
1510
0.1441
0.1161
0.2257
0.1954
0.2195
0.1432
0.2060
0.1371
1523
1.0516
0.7284
0.9019
0,9708
0.6099
0.7782
0.6316
1.0002
4153
0.7778
0.7178
0.6311
0,8111
0.5757
0.6377
0.8360
0.4651
40496
0.5064
0.9563
0.8312
0.8432
0.6733
0.6143
0.7376
1.0124
40682
0.2233
0.0747
0.2456
0.2251
0.2694
0.0685
0.3210
0.2884
40705
0,3879
0.3837
0.7095
0.4415
0.2541
0.3347
0.4304
0.3254
42261
0.2267
0.0570
0,3413
0,3760
0,3889
0.2854
0,3800
O. 0.18
42585
0.6279
0.9790
0.5558
0.6111
0.3494
0.7425
0.3795
0.0383
Mean
0,4366
0.4799
0,4643
0,4679
0.3941
0.3671
0.4722
0.3817
Mean-Ram
4.54
4.94
5.12
5.50
3.88
2.85
5.65
3.51
T
12
8
19
22
Top-1
5
6
30

---

Table 13: Soft clustering performance (Brier score) of ZEUS compared to baseline methods Synthetic Gaussian datasets (lower score reflect better quality).
KM
GMM
AE-KM
AE-GMM
DEC
IDEC
G-CEALS
ZEUS
0.1448
0.2200
0.1741
0.3558
0.4251
0.3264
0.3921
0.1242
0.1218
0.1886
0.2337
0.4718
0.4379
0.4109
0.4377
0.4502
0,1985
0.9985
0.2995
0.1922
0.4303
0.2321
0,5400
0.1952
0.0528
0.9078
0.0927
0.3012
0.6196
0.1628
0.4194
0.0413
0.1005
0.0575
0.1793
0.3351
0.4955
0.2312
0.3886
0.0792
0.1020
1.0950
0.1333
0.4892
0.3355
0,2059
0.3995
0.0881
0.0350
0.0156
0.1200
0.0489
0.2971
0.0770
0.3561
0.0209
7
0.0632
0.9781
0.1599
0.2610
0,4693
0.1207
0,6146
0.0543
0.1168
0.0844
0.2079
0.2786
0.4626
0.2970
0.5975
0.1631
0.0677
0.0092
0.3552
0.1925
0.2167
0.2090
0.4586
0.0573
0.1091
0.0292
0.1469
0.0460
0.3300
0.1229
0.3127
0.0655
11
0.0558
0.0117
0.0791
0.1370
0.4110
0.1122
0.4437
0.0234
12
0.1361
0.2296
0,4022
0.3492
0.5917
0.3230
0.3583
0.1545
13
0.0216
0.0115
0.0356
0.0629
0.3223
0.0980
0.1947
0.0270
0.1048
0.8000
0.3009
0.3776
0.3359
0,2380
0.3818
0.0626
0.0734
0.0278
0.2574
0.0746
0.3458
0.2483
0.3120
0.0352
16
0.0537
0.0448
0.0958
0.1747
0.1640
0.2528
0.1454
0.1770
17
0.1309
0,3513
0,4178
0,2891
0.4313
0.2756
0.5425
0.1308
18
0.1035
0.0452
0.1458
0.2319
0.5193
0.2117
0.2961
0.1939
0.1475
0.1132
0.3094
0.2989
0.2444
0,4612
0.3015
0.3942
Mean
0.0970
0,3110
0.2073
0.2484
0.3943
0,2308
0.3946
0.1269
Mean-Rane Rank
2.30
3.25
4.55
4.95
6.65
4.85
6.75
2.70
Top-3
19
13
6
2
1
3
0
16
Top-1
2
11
0
1
0
0
6
Table 14:
Evaluation of soft clustering quality Synthetic Transformed datasets using Brier score (lower score indicate better performance).
KM
GMM
AE-KM
AE-GMM
DEC
IDEC
G-CEALS
ZEUS
0.5116
0.2135
0.6639
0.4457
0.5911
0.5063
0.7116
0.1282
0.0666
0.0484
0.2328
0.1514
0.4755
0.5045
0.2806
0.2718
0.0883
0.0330
0.4996
0.2822
0.4424
0.3441
0.5990
0.0394
0.3254
0.6286
0.5872
0.2118
0.3241
0.4862
0.7537
0.2678
0.6757
0.1547
0.5580
0.6123
0.3920
0.5158
0.3804
0.3190
0.0887
0.0359
0.3201
0.1517
0.6415
0.4114
0.3869
0.2802
0.5771
0.1788
0.4789
0.4049
0.5718
0.3069
0.5448
0.1763
7
0,3634
0.2875
0,4281
0,3326
0.5478
0.3379
0.6068
0.2461
8
0.1048
0.2738
0.1271
0.0817
0.4517
0.0784
0.3278
0.0167
0.3014
0.0120
0.3120
0.0940
0.4474
0.3329
0.2284
0.0481
0.1785
0.0763
0.3674
0.1249
0.3845
0.5057
0.5420
0.1387
11
0.1053
0.0130
0.1684
0.1128
0.6071
0.2580
0.3306
0.1768
12
0.3682
0.0657
0.6462
0.4740
0.3838
0,4585
0.6318
0.1936
13
0.0604
0.2549
0.4206
0.4292
0.3711
0.4073
0.3436
0.0355
0.0445
0.7982
1.1164
0.6259
0,3672
0.4949
0.6715
0.0097
0.7142
0.7088
0.6256
0.7924
0.5801
0.5741
0.4807
0.8854
16
0.0524
0.4745
0.1194
0.0303
0.5832
0.1743
0.1014
0.0809
17
0.2586
0.0230
0.2616
0.0394
0.3480
0.2156
0.6361
0.0177
18
0.2081
0.6698
0.3781
0.2409
0.3774
0.3861
0.7879
0.1056
0.5120
0.1821
0.7577
0.6415
0.3892
0.4845
0.5570
0.1542
Mcan
0.2803
0.2566
0.4535
0.3140
0.4638
0.3892
0.4951
0.1796
Mcan-Ram
3.95
3.00
5.85
3.95
5.75
5.00
6.15
2.35
Top-3
9
14
0
12
4
3
16
Top-1
0
8
0
2
0
0
9
31

---

F.3
How helpful is regularization for ZEUS?
Tables [5 16 and 17 provide extended versions of Table I. which analyzes the impact of different
combinations of regularization functions. Their structure is analogous to the other tables presented in Appendix
The conclusions that can be drawn from these extended tables, along with their statistical summaries, consistent with our original claims. The model with both Ssep and Lep regularizers performs best on real-world data collections and consistently ranks at least second in the considered statistics for synthetic data. On these generated datasets, is frequently outperformed by the variant that includes the Cep component alone. By contrast, the remaining two approaches clearly grapple with proper clustering of the synthetic datasets.
Table 15: Impact of regularization components on OpenML datasets, evaluated using the ARI score (higher better).
ID
Cprob
Lprob + Leep
Cprob Lep
Cxpob Leep + Lep
14
48.40
43.49
46.44
50.56
15
86.64
87.17
82.94
81.28
16
69.87
73.33
81.87
74.03
18
43.06
50.80
49.62
51.63
22
63.27
56.86
65.58
56.05
52.33
74.58
76.90
85.12
32,31
31.54
41.84
43.66
53
5.17
27.38
25.85
35.76
56
58.48
61.34
20.84
66.41
61
48.93
88.57
65.37
85.15
88.38
91.50
88.22
88.15
377
46.31
45.16
$4.56
55.30
458
98.16
98.49
98.49
99.19
481
2.77
25.87
1,53
8.68
42.76
48.86
30.35
33.38
721
43.28
5.39
43.28
43.28
733
34.35
24.94
76.83
74.97
745
59.56
22.80
4,46
73.85
756
76.15
63.49
38.42
50.31
6.56
3.96
14.52
13.99
820
39,95
39.95
34.72
28.87
840
0.95
4.13
31.14
27.66
854
12.37
12.37
24.32
76.14
1462
92.00
94.25
62.85
92.28
1495
-0.29
98.40
7.35
7.35
1499
60.63
44.86
71.34
82.40
1510
70.09
73.67
72.51
74.26
1523
25.68
28.13
17.38
19.60
4153
38.28
51.26
50.24
62.29
40496
34.15
33.63
30.68
32.25
40682
-0,39
23.38
42.79
53.15
40705
40.00
40.51
38.18
45,45
42261
48.93
88.57
65.37
85.15
42585
53.97
95.82
97.39
95.05
Mean
44.80
48.65
57.43
Mean-rank
3.00
2.37
2,68
1.96
Top-3
20
26
25
30
Top-1
2
6
15
32

---

Table 16: Effect of regularization components on Synthetic Gaussian datasets, assessed by the ARI metric (higher score indicate better quality).
ID
Eprob
Eprob + Luc Eprub Lop
Cprob Lxey + Cop
0
72.89
68.81
85.85
87.65
1
89.50
71.68
88.72
72.04
80.18
63.23
92.59
82.96
3
77.05
79.22
91.23
95.76
4
72.32
84.96
92.82
92.85
5
91.88
88.98
82,25
90.72
81.85
81.18
87.49
97.89
7
85.33
79.93
93.75
93.60
8
72.54
76.12
87.92
84.00
9
95.47
95.97
93.59
94.55
10
88.23
89.61
85.07
87.32
97.26
94.91
96.55
97.45
12
77.58
78.07
85.81
86.04
13
74,41
76,08
96.07
96.17
14
84.00
82.11
95.39
94. 60
77.65
78.94
97.10
96.68
16
93.82
93.46
95.82
87.19
17
88.83
86.38
88.05
86.29
18
89.90
90.21
90.02
83.01
19
76.62
77.69
85.63
73.85
Mcan
83.37
81.88
90.59
89.03
Mean-rank
2.80
3.00
2.00
2.20
Top-3
14
Top-1
3
7
Table 17: Regularisation impact on Synthetic Transformed datasets, measured by ARI (higher is better).
Eprob
Cprob + Losep
Cpros Lee
Caros + L.ee + hee
0
67.29
71.62
85.76
87.42
1
91.98
91.46
96.88
89.40
2
93.17
93.84
92.36
95.88
3
68.26
71.68
80.53
80.98
4
80.93
81.64
84,22
73.66
5
81.38
80.06
95.99
81.63
6
84.29
82.06
80.67
87.02
7
83.35
73.07
96.83
89.57
8
97.42
98.61
97.16
98.11
9
97.63
96.43
96.37
97.20
10
91.32
77.37
90.58
86.42
11
91.84
89.07
99.17
93.19
12
74.93
92.20
91.14
87.75
13
0.32
0.73
90.67
91.60
14
98.00
98.83
98.16
98.49
15
43.28
48.54
30.50
28.96
16
93.34
90.38
98.11
99.01
17
97.62
98.21
95.86
96.44
18
82.64
82.22
86.94
79.70
19
77.93
67.71
83.78
84.13
Mean
79.85
79.29
88,58
86.33
Mean-rank
2.80
2,70
2.30
2.20
Top-3
15
14
15
16
Top-1
7
33

---

F.4 What data-generating prior is optimal?
Tables 18
19 and 20 expand upon the findings presented in Table examining the influence of various data generating priors across all datasets within each group. Their format is consistent with other tables in Appendix(F
Table 18: Evaluation of clustering quality with the ARI metric real-world datasets for different combinations of data-generating probabilistic models (higher scores reflect better performance).
ID
Gauss. + Cat.
NN-transf. Cat.
Gauss. NN-transf.
Gauss. NN-transf, + Cat.
14
36.81
49.84
49.84
50.56
15
85.52
85.57
83.46
81.28
16
57.15
73.05
76.01
74.03
18
42.83
46.48
55.29
51.63
22
46.45
62.23
60.63
56.02
35
87.48
76.48
84.25
85.12
51
-0.52
38.26
39.76
43.66
53
37.57
29.79
27.38
35.76
56
48.40
56.35
59.92
66.41
68.44
62.26
85.08
85.15
89.77
94.87
88.22
88.19
377
35.12
58.34
58,95
55,30
458
47.43
98.79
98.47
99.19
481
6.46
6.94
3.58
8.68
27.75
19.90
27.69
33.38
721
43.28
43.28
38.13
43.28
733
73.48
-7.69
75.77
74.97
745
4.46
67.51
28.11
73.85
756
4.46
59,53
4,46
50.31
796
14.80
15.29
13.99
820
19.26
36.77
33.71
28.87
840
-0.83
0.69
2.57
27.66
854
24.32
73,92
12.56
76.14
1462
1.92
92.56
92.56
92.28
1495
-0.99
0.03
55.17
7.35
1499
58.62
74.43
71.07
82,40
1510
72.50
72.95
70.08
74.26
1523
20.38
18.69
22.01
19.60
4153
41.49
46.33
56.13
62.29
40496
33.20
31.69
30.61
32.25
40682
64.49
58.59
77.21
53.15
40705
29.52
39.74
37.15
45.45
42261
66.34
62.26
81.76
85.15
42585
92.06
94.29
64.95
95.05
Mean
40.59
50.90
52.00
57.43
Mean-Rank
3.10
2.56
2.46
1.88
Top-3
17
27
26
31
Top-1
3
8
16
34

---

Table 19: Effect of prior selection on ZEUS's ARI performance on Synthetic Gaussian datasets (higher scores indicate better performance).
ID
Gauss. Cat.
NN-transf. Cat.
Gauss. NN-transf.
Gauss. NN-transf. + Cat.
0
88.78
87.09
82.89
87.65
87.74
79.51
80.38
72.04
91.30
87.30
20.50
82.96
3
95.08
93.27
44,56
95.76
4
94.40
94,46
92.48
92.85
5
91.43
90.08
30.45
90.72
98.05
97.22
98.02
97.89
7
94.56
94.43
26.90
93.60
8
87.65
87.95
88.05
84,00
9
95.92
92.70
94.79
94.55
10
90.99
86.78
90.07
87.32
96.86
96.52
96.74
97.45
12
88.21
86,25
85,55
86.04
13
96.07
93.39
93.81
96,17
14
95.42
92.12
32.24
94.60
15
97.71
95.84
97.14
96.68
16
95.24
94.49
95.50
87.19
17
89.13
84.54
86.66
86.29
18
90.14
87.66
89.37
83.01
19
87.43
76.30
79.00
73.85
Mean
92.61
89.90
75.25
89.03
Mcan-Rank
1.35
3.05
2.80
2.80
Top-3
20
13
12
Top-1
14
2
Table 20: Influence of the chosen data-generating prior ZEUS performance Synthetic Transformed datasets, reported in terms of ARI (larger values denote better clustering quality).
ID
Gauss. + Cat.
NN-transf. Cat.
Gauss. NN-transf. Gauss. NN-transf. + Cat.
61.54
87.33
88,33
87.42
84.60
95.73
94.60
89,40
2
83.73
93.89
97.59
95,88
3
45.52
75.48
52.79
80.98
76.52
73.72
81.24
73.66
5
72.95
83.07
79.33
81.63
6
80.87
83.20
83.19
87.02
7
88.97
95.91
91.18
89.57
8
97.55
98.09
36.45
98.11
9
94.25
96.86
98.29
97.20
10
40.64
81.56
88.46
86.42
11
93.42
98.57
98.98
93.19
12
83.00
90.91
92.21
87.75
13
80.76
90.87
94.12
91.60
14
98.19
98.49
13.51
98.49
15
28.32
28.47
29.06
28.96
16
98.40
99.05
18.65
99.01
17
87.75
98.21
98.21
96.44
18
5.50
83.07
3.79
79.70
19
64.25
88.36
85.72
84.13
Mean
73.34
87.04
71.29
86.33
Mean-Rank
3.65
2.00
2.08
2.27
Top-3
6
20
16
18
T
0
9
35

---

G Additional experiments
In this section, present additional ablation studies and experiments related to ZEUS, including analyses regarding its robustness, architectural choices, clustering strategies and performance in special-case scenarios.
G.1 Robustness to noise and outliers
In order to assess the robustness of our model under noise and outlier conditions, we consider two alternative evaluation setups:
ZEUS + noise, where we add Gaussian noise with a standard deviation of 0.05 to each data point before the ZEUS forward pass,
ZEUS anomalies, where we introduce set of global anomalies corresponding to 5% the dataset size. Each anomaly is generated by uniformly sampling numerical features from the range (-1,1) and randomly selecting categorical values. The resulting dataset, containing both ground truth data points and anomalies, is then passed through our model. Final clustering quality is measured only on the ground truth points, for which the true class assignments are known.
Table 21 displays the outcomes of these experiments, alongside the performance of the vanilla ZEUS model.
Table 21: Impact of Gaussian noise and anomalies on clustering performance.
Metric
Dataset group
ZEUS+noise
ZEUS+anomalies
ZEUS
ARI
Real
55.88
55.83
57.43
ARI
Syn. Gauss.
88.01
88.21
89.03
ARI
Syn. Transf.
84.82
85.17
86.33
Rank
Real
2.24
1.93
1.84
Rank
Syn. Gauss.
2.45
2.10
1.45
Rank
Syn. Transf. 2.45
2.30
1.25
Overall, the average performance in the perturbed setups is not significantly lower than that of the vanilla ZEUS model, highlighting degree of stability of our method in the presence of noise and outliers.
G.2 Impact of token dimension
The choice of token dimension also a substantial influence on the performance and generalizability of ZEUS. This is confirmed by experiment in which we evaluated dimensions of 32, 128, 512, and 768 while keeping the standard setup unchanged. Table 22 reports the clustering performance of these variants, denoted as ZEUS-32, ZEUS-I20 ZEUS-512, and ZEUS-768, respectively.
Table 22: Assessment different token dimensions in the ZEUS architecture.
Metric
Dataset group
ZFUS-32
ZEUS-128
ZEUS-512
7EUS--68
ARI
Real
44.10
47.81
57.43
50.79
ARI
Syn. Gauss.
69.50
83.04
89.03
91.92
ARI
Syn. Transf.
67.57
81.62
86.33
87.21
Rank
Real
2.90
2.96
1.90
2,25
Rank
Syn. Gauss.
4.00
3.00
1.78
1.23
Rank
Syn. Transf.
3.90
2.78
1.70
1.63
As one can infer, increasing the dimensionality leads to consistent improvement in performance for synthetic data, whereas for OpenMI. datasets, there drop between dimensions 512 and 768, which may be caused by overfitting to the prior.
36

---

G.3 Effect of the output clustering method
Since ZEUS a method for learning representations suitable for clustering, in principle, any clustering algorithm could be used to separate the resulting clusters. However, our approach has the additional feature of being inherently designed form spherical clusters; therefore, K -Means (ZEUS- KM) should, in theory, already be sufficient. The primary advantage of K-means lies in its speed, which is a erucial factor for us during inference. Moreover, as demonstrated in the Table 23 its clustering quality (measured by ARI) is comparable to the more computationally intensive Gaussian Mixture Model ZZEUS-GMM) For reference, we also include results from a simplified variant of GMM with fixed identity covariance matrices (ZEUS-SGMM), as employed in Section 3.3
All experiments are
conducted using the same checkpoint and datasets as Section 3.2
Table 23: Evaluation of clustering methods applied to the learned ZEUS representations.
Metric
Dataset group
ZEUS-KM
ZEUS-SGMM
ZEUS-GMM
ARI
Real
57.43
57.33
56.45
ARI
Syn. Gauss.
89.03
88.18
89.46
ARI
Syn. Transf.
86.33
85.57
87.22
Rank
Real
1.94
2.04
2.01
Rank
Syn. Gauss. 1.90
2.33
1.78
Rank
Syn. Transf. 2.13
2.35
1.53
Analysis of the average ART scores indicates that ZEUS's output well-suited the evaluated approaches, there are no significant differences in their results. Deep clustering techniques such as DEC or IDEC could also potentially be employed. Nevertheless, their reliance on intensive, dataset- specific training contradicts ZEUS's objective of delivering efficient, zero-shot representations.
G.4 Integrating TabICI. transformer into ZEUS
To mitigate one of the main limitations of our approach, namely the restricted number of input features and samples, we investigate whether it is feasible to replace the original TabPFN transformer architecture with
recent design, such the used in TabICI. [18] Specifically, adopt the TabICL transformer with its default hyperparameters and integrate it into our unsupervised pipeline (ZEUS + TabICL) in exactly the same manner described in Section 3.1 The resulting clustering quality, alongside comparison with the original ZEUS model, are presented in Table 24
Table 24: Performance of ZEUS vs. its variant with the TabICL transformer backbone.
Metric
Dataset group
ZEUS
ZEUS TabICL
ARI
Real
57.43
52.62
ARI
Syn. Gauss.
89.03
89.81
ARI
Syn. Transf.
86.33
85.97
Rank
Real
1.38
1.62
Rank
Syn. Gauss. 1.60
1.40
Rank
Syn. Transf.
1.75
1.25
Overall, ZEUS+TabICI performs competitively with the original ZEUS across all evaluated data collections and even surpasses it in rank on synthetic data. Nonetheless, its average ARI on real-world datasets is noticeably lower. Future work should investigate whether this is due to random seed variability, our incomplete understanding the TabICL architecture, potential overfitting to the prior, or the need for more expressive prior.
G.5 Special-case scenarios
Finally, we examine the behavior of our model in several extreme scenarios. To this end, we construct four synthetic two-dimensional datasets designed to highlight specific challenges:
Cluster- Imbalanee Three clusters with 1000, 300, and 50 samples, respectively.
37

---

Three equally sized clusters (500 samples each) with covariance matrices scaled by factors of I, 6, and 36.
Sovariance-Scaled::
Moons: Generated using make_moon from scikit-learn with 1000 samples and 0.1 noise.
Connection: Two spherical Gaussians linked by degenerate linear Gaussian.
We report a comparison of ZEUS performance against K K-Means and GMM in Table 25
Table 25:
Comparison of clustering quality on synthetic special-cast case datasets, assessed by the ARI metric (higher scores indicate better quality).
KM
GMM
ZEUS
Clusler-Imbalance
50.12
96.75
96.80
Covariance-scallo
79.68
96.45
94.72
Moons
48.67
49.23
98.80
Connection 72.76
89.30
83.16
In summary, ZEUS outperforms K-means across all cases and performs par with GMM. Notably, on the Moons dataset, ZEUS is the only method that successfully separates the clusters. This suggests that these synthetic examples do represent genuine failure cases for our approach. A more thorough analysis of datasets where ZEUS underperforms remains an important direction for future work.
H Licensing and Third-Party Assets.
Our method builds upon the publicly available TabPFN codebase. which is released under the Apache 2.0 License. All real-world datasets used in the evaluation are sourced from OpenMI.orgg, a platform hosting open datasets for machine learning research; all datasets used are publicly accessible and labeled as open data, although individual datasets may subject to specific licenses (e.g., CC-BY or similar). We ensured that no proprictary restricted datasets were used. Our released code is provided under the Apache 2.0 License and includes full instructions to reproduce the experiments.
Broader Impact. Although ZEUS is designed as general-purpont tool for clustering tabular data, its use in high-stakes domains like healthcare, finance, or criminal justice carries inherent risks. In such contexts, automated grouping of individuals especially without labels fairness constraints can lead to biased or opaque outcomes. We therefore recommend that any deployment of ZEUS in sensitive applications be accompanied by fairness assessment and expert review.
38


Reviews:
[{"review_id": "ETo9ir5J6t", "reviewer": "Reviewer_p4m9", "summary": "The paper suggests ZEUS as a paradigm to pre-train a network for tabular data to allow for general purpose transformations into an embedding space, in which tabular representations are more separated and thus easier to cluster. The chosen pre-training method is extensively explained and theoretically justified and performance benefits are underlined with experimental evaluation.", "questions": "What dimensional output does Zeus have and what role does the dimensionality play?\n\nHow exactly does the nature of zero-shot learning come into play here and how does it differ from more general transfer learning through feature extraction settings? The reference experiments seem to employ the pre-trained ZEUS network to transform features from the raw data domain to a different embedding space, in which the clustering can be performed, which seems to be a common technique under the umbrella term of transfer learning.\n\nAs ZEUS effectively only transforms the features, why weren't other approaches than k-means employed for the clustering to allow a better understanding of how valuable the transformed ZEUS representations are?", "limitations": "The paper lacks some experimental validation with respect to its data basis and model architectures. Further, some details cannot be extracted due to a missing appendix.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper is easy to follow and presents a clear motivation and context for the suggested approach. Furthermore, the experimental evaluation shows overall a clear performance advantage of the ZEUS representations over other classification approaches.\n\nWeaknesses:\n\nIt is not really clear how the pre-training exactly affects the model performance. It seems that ZEUS is the only model that is trained on data other than the downstream task, even though similar pre-training may also be conceivable for some of the baseline models. A comment or even additional experiments on that end may be beneficial.\n\nFurthermore, it would be interesting to see how pre-training on real datasets would affect the ZEUS representations. It also doesn't really become clear how many samples or epochs the model has been trained on. The paper could clearly benefit from additional details and discussion of the data selection.\n\nAt many instances the paper refers to appendices for more details, which are, however, not present in the paper. An evaluation of these crucial aspects is thus not possible in the presented version, which hinders several aspects of understanding and reproducibility.\n\nRegarding the structure, it is not clear why the related work, introducing all relevant baseline models for comparisons is introduced so late and even abbreviations are only introduced at this point.\n\nThe role of the underlying transformer model is not very clear and how its importance measures against that of the data basis. The paper would benefit from additional experiments in this regard.\n\nminor comments.\nthere are some typos that need to be addressed.\nl. 107: --> needs to be performed.\nl. 123:  the introducing --> the introduction of", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and detailed feedback, as well as for recognizing the core idea and potential of ZEUS — particularly its ability to generalize to real-world clustering tasks without downstream supervision. We appreciate the acknowledgement of the zero-shot setting and the novel use of synthetic generative processes as a meaningful contribution.\n\nWe address your main concerns below:\n\n> At many instances the paper refers to appendices for more details, which are, however, not present in the paper. An evaluation of these crucial aspects is thus not possible in the presented version, which hinders several aspects of understanding and reproducibility.\n\nThe appendix to our publication is available in the “Supplementary Material” section on the Openreview website, under the abstract.\n\n> It is not really clear how the pre-training exactly affects the model performance. It seems that ZEUS is the only model that is trained on data other than the downstream task, even though similar pre-training may also be conceivable for some of the baseline models. A comment or even additional experiments on that end may be beneficial.\n\nWe acknowledge the reviewer’s suggestion to analyze the impact of pretraining. However, none of the baseline methods are designed to support pretraining or cross-dataset transfer. We followed the official repositories and authors’ recommendations, which do not include such procedures. Extending these methods beyond their intended scope is outside the remit of this paper.\nThe performance of ZEUS variants, pre-trained under the same settings as in Section 3.1 of the main paper, except for the use of different combinations of data-generating prior (Gaussian, Categorical, and NN-transformed) is presented in the following table. We evaluate four configurations: \n- **Gaussian + Categorical**, \n- **NN-transformed + Categorical**, \n- **Gaussian + NN-transformed**,\n- **Gaussian + NN-transformed + Categorical** (our standard ZEUS model from Section 3.2). \n\nDetails of these data-generating probabilistic models are provided in Appendix B.\n\n| Metric | Dataset group | Gauss. + Cat. | NN-transf. + Cat. | Gauss. + NN-transf. | Gauss. + NN-transf. + Cat. |\n|-|-|-|-|-|-|\n|ARI|Real|40.59|50.90|51.99|57.43|\n|ARI|Syn. Gauss.| 92.61|89.90|75.25|89.03|\n|ARI|Syn. Transf.| 73.34|87.04|71.29|86.33|\n|Rank|Real |3.1029|2.5588|2.4559|1.8824|\n|Rank|Syn. Gauss.|1.3500| 3.0500| 2.8000|2.8000|\n|Rank|Syn. Transf.|3.6500| 2.0000| 2.0750|2.2750|\n\n> Furthermore, it would be interesting to see how pre-training on real datasets would affect the ZEUS representations. It also doesn't really become clear how many samples or epochs the model has been trained on. The paper could clearly benefit from additional details and discussion of the data selection.\n\nZEUS, like in-context learning methods, requires hundreds of thousands or even millions of datasets during pre-training in order to generalize effectively and perform well on entirely new downstream tasks. Unfortunately, it is not feasible to obtain such a large number of real-world datasets with well-defined and consistently labeled clusters. Pre-training on only a handful of real datasets is insufficient, and under such conditions, ZEUS would perform poorly. To address this limitation, we rely on pre-training with synthetically generated datasets.\n\nAs evidence, we conduct an experiment where we split a set of real/OpenML datasets into two groups, each containing 17 datasets. The first group is used for pre-training, while the second one serves as the evaluation set. The results are presented below:\n\n| Metric | Dataset group| ZEUS | ZEUS - OpenML pre-trained |\n|-|-|-|-|\n|ARI|Real (test subset)|58.21|25.28|\n|Rank|Real (test subset)|1.0441|1.9559|\n\nRegarding the pretraining duration, ZEUS was trained for 500 epochs, each comprising 1,000 batches of single dataset sample, whereas details on the synthetic data generation process are discussed in Appendix B.\n\n> The role of the underlying transformer model is not very clear and how its importance measures against that of the data basis. The paper would benefit from additional experiments in this regard.\n\nThe transformer backbone is used in ZEUS because it enables processing the entire dataset jointly — a requirement of our formulation. This design follows similar precedents in NLP and tabular learning (e.g., TabPFN). At the time of writing, transformers are the de facto state-of-the-art architecture for such use cases, and alternative architectures are either not suitable or not competitive.\n\n> What dimensional output does Zeus have and what role does the dimensionality play?\n\nThe output dimensionality of ZEUS is 512. The table below summarizes our model’s clustering performance (ARI) for different output sizes: 32, 128, 512, and 768.\n\n| Metric | Dataset group | ZEUS-32 | ZEUS-128 | ZEUS-512 | ZEUS-768 |\n|-|-|-|-|-|-|\n|ARI|Real |44.10|47.81|57.43|50.79|\n|ARI|Syn. Gauss.|69.50|83.04|89.03|91.92|\n|ARI|Syn. Transf.|67.57|81.62|86.33|87.21|\n|Rank|Real|2.8971|2.9559|1.8971|2.2500|\n|Rank|Syn. Gauss.|4.0000|3.0000|1.7750|1.2250|\n|Rank|Syn. Transf.|3.9000|2.7750|1.7000|1.6250|\n\nFor synthetic data, increasing the dimensionality leads to a consistent improvement in performance, whereas for OpenML datasets, there is a drop between dimensions 512 and 768, which may be caused by overfitting to the prior.\n\n> How exactly does the nature of zero-shot learning come into play here and how does it differ from more general transfer learning through feature extraction settings? The reference experiments seem to employ the pre-trained ZEUS network to transform features from the raw data domain to a different embedding space, in which the clustering can be performed, which seems to be a common technique under the umbrella term of transfer learning.\n\nWe respectfully disagree. While both zero-shot learning and transfer learning involve reusing knowledge from a source domain, they are conceptually distinct. Transfer learning assumes access to the target dataset and typically involves fine-tuning the model on the downstream task — using labeled data (in supervised settings) or task-specific objectives (in unsupervised ones). In contrast, zero-shot learning requires the model to generalize to new tasks without any access to the target task during training or fine-tuning, and offers much faster inference. This is the regime we operate in.\nIn our case, the ZEUS model is trained only on synthetic data generated from known probabilistic models. It never sees the downstream datasets during training and is applied to them as-is. We do not define or optimize any task-specific loss on the downstream clustering tasks. We explored fine-tuning using objectives inspired by DEC, but preliminary results were inconsistent and unstable - further motivating a pure zero-shot approach. In this sense, ZEUS learns a general mapping from data distributions to clustering structure, which can be applied without any adaptation - distinguishing it from conventional transfer learning pipelines.\n\n\n>  As ZEUS effectively only transforms the features, why weren't other approaches than k-means employed for the clustering to allow a better understanding of how valuable the transformed ZEUS representations are?\n\nZEUS is inherently designed to form spherical clusters, so in theory, k-means (ZEUS-KM) should be a sufficient method for separating the learned representation. Its primary advantage lies in its speed, which is a crucial factor for us during inference. Moreover, as demonstrated in the tables below, its clustering quality (measured by ARI) is comparable to the more computationally intensive Gaussian Mixture Model (ZEUS-GMM). For reference, we also include results from a simplified variant of GMM with fixed identity covariance matrices (ZEUS-SGMM), as employed in Section 3.3 of the main paper. All experiments were performed using the same checkpoint and datasets as in Section 3.2. To enhance clarity, the second table presents an extended version of the comparison across 5 arbitrarily selected datasets per group.\n\n|Metric|Dataset group|ZEUS-KM|ZEUS-SGMM|ZEUS-GMM|\n|-|-|-|-|-|\n|ARI|Real|57.43|57.33|56.45|\n|ARI|Syn. Gauss.|89.03|88.18|89.46|\n|ARI|Syn. Transf.|86.33|85.57|87.22|\n|Rank|Real |1.9412|2.0441|2.0147|\n|Rank|Syn. Gauss.|1.9000|2.3250|1.7750|\n|Rank|Syn. Transf.|2.1250|2.3500|1.5250|\n\n|Dataset group|ID|ZEUS-KM|ZEUS-SGMM|ZEUS-GMM|\n|-|-|-|-|-|\n|Real|53|35.76|35.76|35.76|\n|Real|840|27.66|27.66|24.89|\n|Real|1462|92.28|92.28|92.28|\n|Real|1495 |7.35| 7.35|7.35|\n|Real|42261|85.15|83.49|85.15|\n|Syn. Gauss.|1|72.04|71.94|75.61|\n|Syn. Gauss.|6|97.89| 96.05|97.89|\n|Syn. Gauss.|10|87.32|87.32|87.32|\n|Syn. Gauss.|13|96.17|96.17|96.17|\n|Syn. Gauss.|18|83.01|81.69|83.01|\n|Syn. Transf.|0|87.42|87.23|87.51|\n|Syn. Transf.|3|80.98|77.36|81.33|\n|Syn. Transf.|7|89.57|88.69|90.72|\n|Syn. Transf.|14|98.49|98.49|98.49|\n|Syn. Transf.|18|79.70|79.70|80.12|\n\nAnalysis of the average ARI scores indicates that ZEUS's output is well-suited to evaluated approaches, as there are no significant differences in their results. Deep clustering techniques such as DEC or IDEC could also potentialy be employed. Nevertheless, their reliance on intensive, dataset-specific training contradicts ZEUS’s objective of delivering efficient, zero-shot representations.\n\nWe hope that our responses have addressed the reviewer’s concerns. In particular, we have added:\n- a detailed ablation on the impact of different pretraining distributions,\n- a new experiment comparing synthetic vs. real-data pretraining (Table: ZEUS - OpenML),\n- dimensionality sensitivity analysis (ZEUS-32 to ZEUS-768),\n- and a comparison of downstream clustering heads (k-means, SGMM, GMM) using identical checkpoints.\n\nThese additions, which we will include in the final version of the paper, further clarify the design choices and robustness of ZEUS. We believe they strengthen the overall contribution and hope they support a more favorable evaluation."}, {"review_id": "XkAcFskwRy", "reviewer": "Reviewer_2ngo", "summary": "The paper proposes a framework for unsupervised representation learning for tabular data, such that the resulting representations can be used for clustering datasets in a zero-shot manner. To achieve this, the proposed method (ZEUS) leverages a novel method for generating synthetic data for pre-training, and a set of losses that promote the desirable properties in the learned embedding space. The resulting model leads to strong clustering results in the evaluations conducted.", "questions": "My pripary concern is, as stated in the Weaknesses, related to the lack of experiments and comparisons in the paper. I would urge the authors to include more compared methods and ablations in the paper to strengthen the paper's findings (e.g. if more methods were included it might highlight the sifnificance of ZEUS' results). Alternatively, I would urge the authors to justify why works such as the ones I mentioned earlier were not included.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed methodology is novel, both regarding the synthetic sample generation method and the proposed losses.\n2. ZEUS indeed outperforms alternative methods in terms of the quality of the generated clusters.\n3. The paper is well written and its methodology and motivation is clear.\n\nWeaknesses:\n1. The results presented in Tab. 1 & 2 and Appendix Tab. 4-7 show that, on the aggregate, ZEUS does better than the compared methods. However, it is by small margins that undermine the impact of the proposed method.\n2. The evaluation comparisons are lacking. Most methods compared are naive (e.g. K-Means), and more advanced methods (e.g. G-CEALS) may be at a disadvantage given the size of the models involved: my understanding is that the autoencoder used by G-CEALS is much smaller and less powerfull than the transformer architecture used by ZEUS, which means one cannot be sure how much of ZEUS' gains are related to the architecture as opposed to the method. Finally, it is unclear to my why methods such as DEPICT and IDC are not included in the comparisons.\n3. The ablations should be more thorough. Most importantly, I found no study of the impact of using different synthetic data (Gaussian/Categorical/NN transformed) in the training mix, despite that being a key contribution/component of the paper.", "was_revised": true, "final_justification": "The authors have responded to the issues I raised in my review. Μy main concerns remain the architectural differences with competing methods and the (in my opinion) not entirelly convincing gains, but I appreciate the novel approach and the contribution regarding the training data mix and their impact. I therefore raise my recommendation to accept.", "rebuttal": "We thank the reviewer for the thoughtful and detailed feedback. We appreciate the recognition of ZEUS’s core contribution and agree that stronger experimental comparisons and more thorough ablations are essential to supporting its claims. In response, we have expanded our evaluation with additional experiments, which we summarize below and will include in the final version of the paper.\n\n> 1. The results presented in Tab. 1 & 2 and Appendix Tab. 4-7 show that, on the aggregate, ZEUS does better than the compared methods. However, it is by small margins that undermine the impact of the proposed method.\n\nWe respectfully disagree with the claim that the gains reported in Tables 1 and 2 are marginal and insufficient to support the impact of our method. Clustering is inherently an unsupervised task, and without a clearly defined target it is difficult to determine which clustering is \"better.\" Many real-world datasets commonly used in the literature were originally constructed for classification, and their class labels often do not correspond to well-separated clusters. As a result, most methods struggle to recover them. Despite this, ZEUS achieves the highest ARI on real-world data (57.43) and the best average rank (3.60), outperforming all compared baselines.\n\nTo provide a more interpretable evaluation, we also include synthetic datasets with well-defined ground-truth clusters, based on latent variable models and mixture components. In these more structured scenarios, ZEUS shows clear and consistent advantages. We also include Gaussian mixtures as a sanity check — these are deliberately simple and yield similar results across methods, which naturally reduces visible margins.\n\nOverall, ZEUS demonstrates strong and reliable performance across diverse clustering regimes, including both ill-posed real-world data and well-defined synthetic tasks.\n\n> 2. The evaluation comparisons are lacking. Most methods compared are naive (e.g. K-Means), and more advanced methods (e.g. G-CEALS) may be at a disadvantage given the size of the models involved: my understanding is that the autoencoder used by G-CEALS is much smaller and less powerfull than the transformer architecture used by ZEUS, which means one cannot be sure how much of ZEUS' gains are related to the architecture as opposed to the method.\n\nFor all deep clustering baselines, we used the implementations provided by the authors and followed their recommended hyperparameter selection strategies. For instance, the embedding dimension was chosen from the set {5, 10, 15, 20}, as suggested in the GCEALs paper, and we adhered to this choice, as it was deemed beneficial for those baselines. ZEUS contains 25.78 million parameters because of its large-scale pretraining across millions of datasets in a zero-shot regime. Baseline methods are trained on individual datasets but still use autoencoders with over 2.5 million parameters. \n\nAs shown in the table below, increasing the size of the autoencoder from hidden layers [500,500,2000] (GCEALS-2.5M) to [500,500,2000,2000,2000] (GCEALS-10.5M) actually leads to worse performance on both real-world and synthetic Gaussian datasets. This suggests that simply scaling up the model can lead to some form of overfitting. Furthermore, if increasing the autoencoder’s size alone were beneficial, the original authors would have likely reported this. Nonetheless, exploring Transformer-based variants of DEC or GCEALs might be worthwhile, though such extensions go beyond the scope of this study.\n\n| Metric | Dataset group | GCEALS-2.5M | GCEALS-10.5M |\n|--------|---------------|-----------|------------|\n| ARI    | Real          | 40.37    | 26.91     |\n| ARI    | Syn. Gauss.   |62.84    | 56.50     |\n| ARI    | Syn. Transf.  | 49.17    | 50.67     |\n| Rank   | Real          | 1.2059    | 1.7941     |\n| Rank   | Syn. Gauss.   | 1.2000    | 1.8000     |\n| Rank   | Syn. Transf.  | 1.5000    | 1.5000     |\n\nIn the case of GCEALS, the primary issue lies in its instability across different random seeds. The method exhibits high variance, which negatively affects the average performance. For illustration, the standard deviation of ARI scores on selected datasets is as follows:\n\n| Dataset group | ID    | GCEALS-STD |\n|---------------|-------|------------|\n| Real          | 15    | 9.65       |\n| Real          | 458   | 10.45      |\n| Real          | 187   | 30.82      |\n| Real          | 42261 | 23.64     |\n| Syn. Gauss.   | 3     | 10.74      |\n| Syn. Gauss.   | 8     | 16.99      |\n| Syn. Gauss.   | 10    | 44.1       |\n| Syn. Gauss.   | 17    | 7.01       |\n| Syn. Transf.  | 2     | 15.04      |\n| Syn. Transf.  | 7     | 11.65      |\n| Syn. Transf.  | 14    | 8.33       |\n| Syn. Transf.  | 19    | 25.75      |\n\nIn general, finding the optimal configuration for dozens of hyperparameters in an unsupervised setting is inherently challenging due to the absence of labels. Our method offers a clear advantage in this regard, as its zero-shot nature allows it to avoid such issues during inference.\n\n> Finally, it is unclear to me why methods such as DEPICT and IDC are not included in the comparisons.\n\nIndeed, you are right, that IDC should have been included in our evaluation. According to the tables in the GCEALS paper, DEPICT is among the weaker methods and is also 4 to 5 times slower than GCEALS or DEC. Therefore, unfortunately, due to limited time and computational resources, we decided to exclude it from our experiments. However, in addition to our baseline methods, we also include TabPFN-Unsupervised and SCARF, as suggested by Reviewer gk7v. Similar to ZEUS, we cluster their learned representations using k-means. The clustering quality results are presented below:\n\n|Metric| Dataset group |IDC|TabPFN-Uns.|SCARF|GMM|KM|AE-KM|AE-GMM|DEC|IDEC|G-CEALS | ZEUS|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|ARI|Real|52.28|31.32|26.95|48.49|55.54|51.43|53.56|55.93|54.57|40.37|57.43|\n|ARI|Syn. Gauss.|66.43|55.97|8.32|76.93|89.90|81.26|81.40|89.36|82.57|62.84|89.03|\n|ARI|Syn. Transf.|66.78|15.66|2.48|75.88 |75.04 |60.45|71.29|79.94|61.26|49.17|86.33|\n|Rank|Real|5.62|8.22|8.85|5.65|4.69|5.72|5.24|4.69|5.01|8.18|4.13|\n|Rank|Syn. Gauss.|7.95|8.75|11.00|3.65|2.65|5.65|5.60|3.20|5.70|8.90|2.95|\n|Rank|Syn. Transf.|6.05|9.75|11.00|3.50|4.80|6.85|4.50|3.20|6.35|7.80|2.20|\n\nFor IDC, we adhere to the setup and guidelines outlined in the official repository's Jupyter notebooks.\n\n> 3. The ablations should be more thorough. Most importantly, I found no study of the impact of using different synthetic data (Gaussian/Categorical/NN transformed) in the training mix, despite that being a key contribution/component of the paper.\n\nIndeed, such an analysis would significantly strengthen our paper. Therefore, we conduct an experiment using the setup described in Section 3.1 to evaluate the clustering quality under four configurations of the data-generating prior:\n- **Gaussian + Categorical**, \n- **NN-transformed + Categorical**, \n- **Gaussian + NN-transformed**,\n- **Gaussian + NN-transformed + Categorical** (our standard ZEUS model from Section 3.2). \n\n| Metric | Dataset group | Gauss. + Cat. | NN-transf. + Cat. | Gauss. + NN-transf. | Gauss. + NN-transf. + Cat. |\n|--------|---------------|---------------|-------------------|---------------------|----------------------------|\n| ARI    | Real          | 40.59        | 50.90            | 51.99              | 57.43                     |\n| ARI    | Syn. Gauss.   | 92.61        | 89.90            | 75.25              | 89.03                     |\n| ARI    | Syn. Transf.  | 73.34        | 87.04            | 71.29              | 86.33                     |\n| Rank   | Real          | 3.1029        | 2.5588            | 2.4559              | 1.8824                     |\n| Rank   | Syn. Gauss.   | 1.3500        | 3.0500            | 2.8000              | 2.8000                     |\n| Rank   | Syn. Transf.  | 3.6500        | 2.0000            | 2.0750              | 2.2750                     |\n\nAs expected, the “Gaussian + Categorical” model achieves top performance on the Synthetic Gaussian datasets, while the” NN-transformed + Categorical” method excels on Synthetic Transformed data collections. Notably, the latter also performs well on Gaussian sets, as its prior is built upon transformed Gaussian mixtures. In contrast, the “Gaussian + NN-transformed” model yields relatively poor average ARI scores across both synthetic benchmarks. This drop is primarily caused by a significant decline in ARI performance on categorical datasets. Nonetheless, its average rank remains consistently competitive. For the OpenML datasets, the best performance is achieved when pre-training includes all three types of priors, underscoring the complementary importance of each prior in the training process.\n\nThese additions clarify the role of individual components in ZEUS and help isolate the sources of its performance. We believe they provide a more complete and transparent evaluation of the method. We appreciate the reviewer’s feedback, which helped us improve the clarity and depth of the empirical study, and we hope these extended experiments support a more favorable assessment."}, {"review_id": "jUuhOayLkj", "reviewer": "Reviewer_3Vjr", "summary": "The paper presents a transformer-based approach (called ZEUS) where a transformer is trained on datasets with ground truth cluster labels and learns to assign cluster labels to new datasets. Zeus has been compared against k-means, GMMs and deep clustering methods on several families of real and synthetic datasets.", "questions": "Q1. It is not clear how the real datasets have been separated into training and testing.\nQ2. In several cases, ZEUS performance is very low (much lower that k-means and GMMs). This reduces confidence in the method’s validity.\nQ3. It would be interesting to compare performance on datasets with discrete features only. Discrete features cause difficulties in k-means and GMMs.\nQ4. For many real datasets the number of clusters is 2, which is quite small.\nQ5. It should be emphasized that k-means is applied on the transformer output in order to obtain the final cluster labels.\nQ6. Some results on the influence of noise and outliers would add value to the experimental study.", "limitations": "The major limitations of the method are discussed in the paper: dependence on the synthetic datasets used for training and restrictions on the scale of the dataset.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\nS1. ZEUS implements a novel idea on how to train a transformer using annotated clustering datasets in order to assign cluster labels on new datasets.\nS2. The approach can accommodate mixed datasets with both real and categorical features.\n\nWeaknesses\nW1. Since supervised training is used, clustering performance strongly depends on the annotated datasets used for clustering.\nW2. There are limits imposed by the input dimensionality of the transformer (dataset size, number of features)", "was_revised": true, "final_justification": "As emphasized in my discussion with the authors, I believe that the proposed method is interesting, however,  several open issues remain that need to be addressed. I have increased my score putting more emphasis on the novel aspects of the method.", "rebuttal": "Reviewer 3Vjr (3, conf =5)\nWe thank the reviewer for the detailed and insightful comments. We appreciate the recognition of the potential impact of ZEUS, and agree that its claims must be supported by strong comparisons, ablations, and a deeper analysis of edge cases. Below, we respond point-by-point and provide several additional experiments (e.g., discrete-only data, impact of synthetic priors, sensitivity to noise), which will be included in the final version of the paper.\n\n> W1. Since supervised training is used, clustering performance strongly depends on the annotated datasets used for clustering. \n\nWe agree that the choice of training distributions has a strong influence on the resulting clustering behavior, and that this creates opportunities for further improvement. In our current work, we approach the problem from a generative perspective rather than by committing to a fixed clustering loss. Specifically, we adopt the widely used latent variable model (LVM) paradigm, where a discrete latent variable determines cluster membership. To make this framework applicable beyond simple Gaussian mixtures, we apply random neural transformations to the observed space, enabling a rich diversity of data distributions while preserving the underlying cluster structure.\nZEUS is trained to reverse this generative process, allowing it to generalize across datasets without relying on predefined objectives like MSE. That said, we believe that incorporating more structured or domain-specific generative processes, or learning such transformations from real data, is a promising direction for future work.\n\n> W2. There are limits imposed by the input dimensionality of the transformer (dataset size, number of features)\n\nThese limitations are primarily inherent to the architecture of the vanilla TabPFN model. However, recent research, including TabPFN2 [1], TabICL [2], TabFlex [3] has shown that such restrictions can be gradually mitigated. We have conducted preliminary tests indicating that the TabICL architecture can be possibly integrated into our pre-training pipeline. The corresponding results are presented in the table below:\n\n\n| Metric | Dataset group | ZEUS   | ZEUS + TabICL |\n|--------|---------------|--------|---------------|\n| ARI    | Real | 57.43 | 52.62        |\n| ARI    | Syn. Gauss.   | 89.03 | 89.81        |\n| ARI    | Syn. Transf.  | 86.33 | 85.97        |\n| Rank   | Real          | 1.3824 | 1.6176        |\n| Rank   | Syn. Gauss.   | 1.6000 | 1.4000        |\n| Rank   | Syn. Transf.  | 1.7500 | 1.2500        |\n\nZEUS + TabICL performs competitively with the original ZEUS across all evaluated dataset groups and even surpasses it in average rank on synthetic data. However, its ARI on real-world datasets is slightly lower. This may stem from seed sensitivity, architectural factors, or limitations in how the prior interacts with more heterogeneous tabular distributions. Investigating these effects and improving compatibility with TabICL-style architectures is a promising avenue for future work.\n\n> Q1. It is not clear how the real datasets have been separated into training and testing. \n\n\nWe clarify that real datasets are used only for testing. The model is trained exclusively on synthetically generated data. No samples from the real datasets are seen during training, and there is no fine-tuning. This setup ensures a strict zero-shot evaluation.\n\n\n> Q2. In several cases, ZEUS performance is very low (much lower that k-means and GMMs). This reduces confidence in the method’s validity. \n\n\nIt is true that ZEUS underperforms on some individual datasets. However, no clustering method performs well across all settings, which is evident in the performance of other deep models that often fall behind k-means or GMMs on average. What sets ZEUS apart is its ability to generalize in settings where others fail.\nIn particular, on synthetic datasets with transformed latent structures (Appendix Table 10), ZEUS outperforms all baselines by a large margin (~10 pp), showing its strength in realistic, non-Gaussian clustering scenarios with well-defined ground truth.\nFor real-world datasets, we follow standard benchmarks where class labels are used as cluster references. While common in the literature, this is not always meaningful in practice, as classes do not necessarily align with cluster structure. Still, ZEUS ranks first on average and achieves ARI above 0.9 on datasets where many methods fail completely (ARI = 0) - for example, OpenML dataset 1462, as shown in Appendix Table 8. This demonstrates that ZEUS fills an important gap by succeeding where others collapse.\n\n> Q3. It would be interesting to compare performance on datasets with discrete features only. Discrete features cause difficulties in k-means and GMMs. \n\nThe evaluated datasets (GuptaA_2019_PA and QinN_2014_PA) are presence/absence (binary) metagenomic profiles obtained from the curatedMetagenomicData [4] collection. Each feature indicates whether a specific microbial taxon is present in a given sample. The datasets are named after the first author and year of the corresponding publication, following the convention used in the original repository. Their high dimensionality, sparsity, and purely discrete nature make them particularly challenging for clustering methods based on continuous distances, such as k-means or GMMs.\n|                | KM    | ZEUS  |\n|----------------|-------|-------|\n| GuptaA_2019_PA | 12.55 | 31.16 |\n| QinN_2014_PA   | 35.14 | 41.43 |\n\nThe table highlights the potential of ZEUS, which could likely be further enhanced by pre-training specifically on discrete features or by replacing one-hot encoding with alternative embedding methods.\n\n> Q4. For many real datasets the number of clusters is 2, which is quite small. \n\n\nIt is true that many real-world datasets in our benchmark have only 2 classes, as they originate from standard binary classification tasks. However, approximately half of the datasets contain more than two classes, and we observe that ZEUS often performs especially well in these more complex multi-cluster scenarios. For example, on datasets such as 14, 16, 22, 35, 458, 1499, 4153, and 42585, ZEUS outperforms all other baselines, which highlights its strength in capturing more nuanced latent structures. Synthetic datasets, in particular, are predominantly multi-class and further demonstrate this advantage.\n\n> Q5. It should be emphasized that k-means is applied on the transformer output in order to obtain the final cluster labels. \n\nThank you for pointing this out. We agree that this detail should be made explicit, and we will make sure to emphasize it clearly in the final version of the paper.\n\n\n> Q6. Some results on the influence of noise and outliers would add value to the experimental study.\n\n\nThat is a great remark, and we thank the reviewer for highlighting it. We were able to conduct some initial experiment on this matter. In addition to the standard evaluation method (referred to as ZEUS), we introduce two additional variants:\nZEUS + noise: Gaussian noise with a standard deviation of 0.05 is added to each data point before the ZEUS forward pass.\nZEUS + anomalies: For each dataset, we introduce a set of global anomalies corresponding to 5% of the dataset size. These anomalies are constructed by uniformly sampling numerical features from the range (−1,1) and randomly selecting categorical values.\nThe results are as follows:\n\n| Metric | Dataset group | ZEUS   | ZEUS+noise | ZEUS+anomalies |\n|--------|---------------|--------|------------|----------------|\n| ARI    | Real          | 57.43 | 55.88     | 55.83         |\n| ARI    | Syn. Gauss.   | 89.03 | 88.01     | 88.21         |\n| ARI    | Syn. Transf.  | 86.33 | 84.82     | 85.17         |\n| Rank   | Real          | 1.8382 | 2.2353     | 1.9265         |\n| Rank   | Syn. Gauss.   | 1.4500 | 2.4500     | 2.1000         |\n| Rank   | Syn. Transf.  | 1.2500 | 2.4500     | 2.3000         |\n\nAs we can see, the average performance in the perturbed variants is not significantly lower than that of the vanilla ZEUS model, highlighting a degree of stability in the method.\n\nThese additional experiments and clarifications provide a broader view of ZEUS’s strengths and limitations. We appreciate the reviewer’s thoughtful and technically grounded suggestions, which have helped improve the depth and rigor of our evaluation. We hope the extended results and stress tests address the reviewer’s concerns and support a more favorable assessment.\n\n---\n\n[1] Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., & Hutter, F. (2025). Accurate predictions on small data with a tabular foundation model. Nature.\n\n[2] Qu, J., Holzmüller, D., Varoquaux, G., & Le Morvan, M. (2025). TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. arxiv preprint\n\n[3] Zeng, Y., Dinh, T., Kang, W., & Mueller, A. C. (2025). TabFlex: Scaling Tabular Learning to Millions with Linear Attention. arxiv preprint\n\n[4] Pasolli E, Schiffer L, Manghi P, Renson A, Obenchain V, Truong D, Beghini F, Malik F, Ramos M, Dowd J, Huttenhower C, Morgan M, Segata N, Waldron L (2017). “Accessible, curated metagenomic data through ExperimentHub.” Nat. Methods, 14(11), 1023–1024. ISSN 1548-7091, 1548-7105,"}, {"review_id": "RclLmvs6VN", "reviewer": "Reviewer_gk7v", "summary": "The authors propose an unsupervised approach to learn embeddings for zero-shot clustering of the tabular data. The main ingredients of their approach are: 1) pre-train using synthetic data as in TabPFN 2) Transformer architecture for the backbone 3) loss based on GMM. In a series of experiments, the authors demonstrate the applicability of their approach to the", "questions": "## Understanding loss\n1. From your final objective given in (3), I have doubts about the difficulty of the optimization and the importance of good initialization. Do you keep updating $\\hat c_k$ at each iteration? \n2. What role do $\\hat \\pi$'s play? In unsupervised settings, we use EM to fit GMM, so $\\hat \\pi$'s are updated each E-step. In your case, they seem fixed, defined by the training label. \n\n## Architecture\n1. There is little that is said about the architecture in the paper. The authors mention that they re-use TabPFN backbone in Section 3.1. However, for TabPFN the training label is passed together with the features. There's also cross-attention between training and test sets. Could the authors elaborate more on the specific architectural adjustments? \n2. Major drawback of TabPFN is the context length, which is currently limited at around 10,000 samples for TabPFN V2. Recent works such as TabFlex or Mixture of ICP have explored addressing this concern via subsampling or compressing the data. Are these approaches portable to ZEUS?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Context\nDeep learning for tabular (heterogeneous) data has seen large number of works, which can be roughly grouped into architecture improvements (TabNet, NODE), unsupervised learning (SwitchTab, Scarf), retrieval-augmented models (TabR, ModernNCA), and foundational models (TabGPT, TabPFN). The latter, TabPFN, is the latest work on the foundational models for tabular data that has demonstrated an almost magical ability to generalize to previously unseen datasets in a zero-shot fashion via in-context learning. This work builds on the idea of TabPFN and extends it to the tabular clustering problem. \n\n## Strength\n1. **Empirical results.** The method clearly outperforms other approaches based on shallow models (e.g. GMM) and autoencoders (e.g. AE-GMM, G-CEALS). The authors provide an ablation study to elucidate design choices of the loss. In addition, they evaluate model calibration, which is useful in many settings. \n2. **Impact in practice.** The authors formulate the problem as a GMM. Therefore, the predicted embeddings are organized spherically in the latent space, which makes them directly amenable to application with downstream algorithms such as KNN or K-Means with minimal adaptation. \n3. **Novelty.** To my knowledge, this is the first zero-shot embedding algorithm for tabular data. \n\n## Weaknesses\n1. **Incomplete comparison.** **Firstly**, the TabPFN repo has extensions, one of which is the unsupervised learning of embeddings: https://github.com/PriorLabs/tabpfn-extensions/tree/main/src/tabpfn_extensions/embedding. I'm not sure which work has appeared first, but nevertheless, I feel it would be a valuable comparison to understand the difference between TabPFN and ZEUS embeddings. **Secondly**, there are prior works on the unsupervised learning for tabular data such as Scarf or SwitchTab -- these are not zero-shot, but it would be nice to include some reference (or benchmark) to them to understand how they differ from the current work.\n2. **Failure cases not explained.** If we take this work apart, the technical novelty lies in prior dataset generation and loss formulation. Both are formulated / generated using GMM. My complaint is that the section on the results may not accurately depict the real-world performance: 1) simulated / transformed datasets also follow GMM -- all the model needs to learn here is how to learn identity or inverse (of ResNet) transformation 2) real datasets are chosen to have ARI > 0.4 by at least one of the methods. Related to 2), there are multiple  problems: a) the threshold for cutoff appears a bit arbitrary b) all of the methods (AE, GMM) build on top of Gaussian assumption -> therefore selected real datasets are likely to have the Gaussian bias as well c) the problem in (b) could be addressed if the authors include other methods e.g. TabPFN or SwitchTab. In general, it would be great to see a stress test of the model to understand when it fails. Let's say there are no clusters, or if there is a severe class imbalance, or there is a mode connecting, or the geometry of clusters is very difficult.", "was_revised": true, "final_justification": "The authors have effectively answered all my questions and concerns during the rebuttal.", "rebuttal": "We thank the reviewer for their positive evaluation and for the insightful and constructive comments. We especially appreciate the suggestions regarding additional comparisons, architectural clarity, and stress-testing the method. In response, we have conducted new experiments and added clarifications, which we summarize below and will include in the final version of the paper.\n\n> 1. Incomplete comparison. Firstly, the TabPFN repo has extensions, [..] Secondly, there are prior works on the unsupervised learning for tabular data such as Scarf or SwitchTab -- these are not zero-shot, but it would be nice to include some reference (or benchmark) to them to understand how they differ from the current work.\n\nWe appreciate the suggestion to compare ZEUS with unsupervised representation learning approaches in the context of clustering task. Accordingly, we incorporate both TabPFN-Unsupervised and SCARF into our evaluation.\n\nFor TabPFN-Unsupervised, while the linked method relies on label information from the training set to generate representations, the same repository also includes an `unsupervised` folder containing the `TabPFNUnsupervisedModel` class. We use its `get_embeddings_per_column` method to extract per-column representations, which we then average across columns to obtain a final row-level representation. Unfortunately, we were unable to use the `get_embeddings` function due to the following error:\n\nNotImplementedError(\n\"This method is not implemented currently. During the main TabPFN refactor this functionality was removed, [...]\"\n)\n\nIn the case of SCARF, we follow the `example.ipynb` Jupyter notebook provided in the original repository. For both SCARF and TabPFN-Unsupervised, we cluster their learned representations using k-means, analogously to our approach in ZEUS. Additionally, we included IDC in our evaluation, as suggested by **Reviewer 2ngo**.\n\n|Metric| Dataset group |IDC|TabPFN-Uns.|SCARF|GMM|KM|AE-KM|AE-GMM|DEC|IDEC|G-CEALS | ZEUS|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|ARI|Real|52.28|31.32|26.95|48.49|55.54|51.43|53.56|55.93|54.57|40.37|57.43|\n|ARI|Syn. Gauss.|66.43|55.97|8.32|76.93|89.90|81.26|81.40|89.36|82.57|62.84|89.03|\n|ARI|Syn. Transf.|66.78|15.66|2.48|75.88 |75.04 |60.45|71.29|79.94|61.26|49.17|86.33|\n|Rank|Real|5.62|8.22|8.85|5.65|4.69|5.72|5.24|4.69|5.01|8.18|4.13|\n|Rank|Syn. Gauss.|7.95|8.75|11.00|3.65|2.65|5.65|5.60|3.20|5.70|8.90|2.95|\n|Rank|Syn. Transf.|6.05|9.75|11.00|3.50|4.80|6.85|4.50|3.20|6.35|7.80|2.20|\n\nBased on the results in the table, one can infer that methods like SCARF or TabPFNUnsupervised are more suited for feature extraction, which can then be fine-tuned for classification tasks. Without additional regularization components during training, or specialized fine-tuning such as with DEC, we are unable to effectively separate these representations.\n\n> 2. Failure cases not explained. If we take this work apart, the technical novelty lies in prior dataset generation and loss formulation. Both are formulated / generated using GMM. My complaint is that the section on the results may not accurately depict the real-world performance: [...]. In general, it would be great to see a stress test of the model to understand when it fails. Let's say there are no clusters, or if there is a severe class imbalance, or there is a mode connecting, or the geometry of clusters is very difficult.\n\nFirstly, we acknowledge that benchmarking clustering performance in realistic failure scenarios is inherently difficult, particularly for small and medium-sized tabular datasets. To our knowledge, there is no widely accepted benchmark that provides well-defined clustering ground truth beyond class labels, which may not reflect true cluster structure.\n\nSecondly, we are grateful to the reviewer for the insightful suggestion to examine the failure cases of our model. While such analysis requires careful consideration and time, we have taken initial steps in that direction. Specifically, we constructed four synthetic 2D scenarios:\n- **Cluster-Imbalance**: Three clusters with 1000, 300, and 50 samples, respectively.\n- **Covariance-Scaled**: Three equally sized clusters (500 samples each) with covariance matrices scaled by factors of 1, 6, and 36.\n- **Moons**: Generated using make_moons from scikit-learn with 1000 samples and 0.1 noise.\n- **Connection**: Two spherical Gaussians linked by a degenerate linear Gaussian.\n\n| | Cluster-Imbalance | Covariance-scaled | Moons  | Connection   |\n|-|-|-|-|-|\n| KM   | 0.5012| 79.68| 48.67 | 72.76 |\n| GMM  | 0.9675| 96.45| 49.23 | 89.30 |\n| ZEUS | 0.9680| 94.72| 98.80 | 83.16 |\n\nZEUS outperforms K-means across all cases and performs on par with GMM. Notably, on the Moons dataset, ZEUS is the only method that successfully separates the clusters. This suggests that these synthetic examples do not represent genuine failure cases for our approach. A more thorough analysis of datasets where ZEUS underperforms remains an important direction for future work.\n\n> Understanding loss\n> 1. From your final objective given in (3), I have doubts about the difficulty of the optimization and the importance of good initialization. Do you keep updating  $\\hat{c}_k$ at each iteration?\n\nFirst of all, $\\hat{c}_k$​ is estimated as the average representation of the points assigned to cluster $k$. Therefore, it is indeed updated dynamically and differs across datasets.\n\nAs for initialization, it certainly plays a role in optimization process. A good initialization under a given random seed can lead to faster convergence and potentially better performance. Conversely, extremely poor initialization may result in instability or even complete collapse of pre-training. However,  we did not observe this to be a significant issue during our experiments. Nonetheless, investigating the impact of initialization strategies could be a potential direction for future work.\n\n> 2. What role do $\\hat{\\pi}$'s play? In unsupervised settings, we use EM to fit GMM, so $\\hat{\\pi}$'s are updated each E-step. In your case, they seem fixed, defined by the training label.\n\nYes, the values of $\\hat{\\pi}$ are defined based on the training labels and are not updated in any other way. Nonetheless, introducing $\\hat{\\pi}$ helped slightly improve performance, particularly on real-world datasets. This improvement may stem from the fact that incorporating information about cluster balance increases the flexibility of the learning process and provides a beneficial learning bias. The comparison between the version with a prior (ZEUS) and the one without it (ZEUS-without-prior) in the loss function is as follows:\n\n| Metric | Dataset group | ZEUS   | ZEUS-without-prior |\n|-|-|-|-|\n|ARI|Real | 57.43 | 50.12|\n|ARI|Syn. Gauss.|89.03|88.42|\n|ARI| Syn. Transf.|86.33|83.35|\n| Rank|Real| 1.3088 | 1.6912|\n| Rank|Syn. Gauss. | 1.4000 | 1.6000|\n| Rank|Syn. Transf. | 1.2250 | 1.7750|\n\n> Architecture\n> 1. There is little that is said about the architecture in the paper. The authors mention that they re-use TabPFN backbone in Section 3.1. However, for TabPFN the training label is passed together with the features. There's also cross-attention between training and test sets. Could the authors elaborate more on the specific architectural adjustments?\n\nSimilarly to TabPFN, we employ a linear embedding layer to convert each row of the table into a 512-dimensional token. However, since we are addressing an unsupervised problem, no training label embeddings are added to these tokens. Moreover, our setting does not involve a query set, and thus we apply only self-attention within the support set. The transformer architecture matches that of TabPFN: it consists of 12 attention blocks, each with 6 attention heads and GeLU as the activation function. At its output, we obtain a representation for each data point, which is then clustered using k-means. Notably, unlike TabPFN, no additional MLP decoder is applied after the transformer.\n\n> 2. Major drawback of TabPFN is the context length, which is currently limited at around 10,000 samples for TabPFN V2. Recent works such as TabFlex or Mixture of ICP have explored addressing this concern via subsampling or compressing the data. Are these approaches portable to ZEUS?\n\nYes, we believe that architectures of TabPFN’s successors are transferable to ZEUS. To preliminarily test this, we explore the possibility of adapting the TabICL [1] architecture pipeline to our unsupervised clustering setting, following the same methodology as in the case of TabPFN (see response above). The resulting performance is shown below:\n\n| Metric | Dataset group | ZEUS   | ZEUS + TabICL |\n|-|-|-|-|\n|ARI|Real| 57.43 | 52.62|\n|ARI|Syn. Gauss.|89.03 | 89.81|\n|ARI|Syn. Transf.|86.33 | 85.97|\n|Rank|Real|1.3824 | 1.6176|\n|Rank|Syn. Gauss.|1.6000 | 1.4000|\n|Rank |Syn. Transf. |1.7500 | 1.2500|\n\nZEUS + TabICL performs competitively with the original ZEUS across all evaluated data collections and even surpasses it in rank on synthetic data. Nonetheless, its average ARI on real-world datasets is noticeably lower. Future work should investigate whether this is due to random seed variability, our incomplete understanding of the TabICL architecture, potential overfitting to the prior, or the need for a more expressive prior.\n\nWe are grateful for the reviewer’s thoughtful suggestions, which have led to a stronger and more complete version of the work. We hope the added comparisons (TabPFN-Unsupervised, SCARF), architectural clarifications, stress-test scenarios, and ablations help reinforce the contribution and reliability of ZEUS. We appreciate the reviewer’s positive assessment.\n\n---\n\n[1] Qu, J., Holzmüller, D., Varoquaux, G., & Le Morvan, M. (2025). TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. arxiv preprint"}]

Author rebuttal:

Meta-review: This paper presents a method (Zeus) to pre-train a network for tabular data and use it for zero-shot embedding (unsupervised) that generalizes across datasets. The authors provide experimental results to demonstrate the efficacy of the approach against existing clustering techniques.

Overall, the problem is well motivated with practical applications and the paper introduces an interesting & novel approach to address the challenges described. The authors also did a good job at providing details & clarifications around several points raised by reviewers esp. around ablations. Some of the weaknesses of the paper stemmed from the limited complexity around some of the datasets used in evaluations (also highlighted by some reviewers) — for example, small #clusters, grounding the choice of transformations (and correspondingly data used for training) wrt learnability & generalizability of the model, extensions for the architecture choice, etc. The authors address several of the concerns in the rebuttal which def. strengthens the claims, suggest these along with other important feedback to be incorporated into the paper to make it well-rounded and standalone.
Decision: Accept (poster)