Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer
Authors: Yulun Wu
Doron L Bergman
Venue: ICML 2025 poster (2025)
Keywords: zero-shot
meta-learning
adversarial training
tabular deep learning
bayesian inference
in-context learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: We present an Adversarially Pre-trained Transformer (APT) that is able to perform zero-shot meta-learning on tabular prediction tasks without using any real-world dataset to pre-train the model, extending on the recent development of Prior-Data Fitted Networks (PFNs) and TabPFN. Specifically, APT is pre-trained with adversarial synthetic data agents, who continue to shift their underlying data generating distribution and deliberately challenge the model with different synthetic datasets. In addition, we propose a mixture block model architecture that is able to handle classification tasks with arbitrary number of classes, addressing the class size limitation -- a crucial weakness of prior tabular zero-shot learning algorithms. In experiments, we show that our framework matches state-of-the-art performance on small tabular classification tasks without filtering on dataset characteristics such as number of classes and number of missing values, while maintaining an average runtime under one second. On common benchmark dataset suites in both classification and regression, we show that adversarial pre-training was able to enhance TabPFN's performance. In our analysis, we demonstrate that the adversarial synthetic data agents were able to generate a more diverse collection of data compared to the ordinary random generator in TabPFN. In addition, we demonstrate that our mixture block neural design has improved generalizability and greatly accelerated pre-training.

Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer
Yulun Wu Doron L. Bergman"
patible with, and drawn from, the same distribution as the datasets it was previously trained on. Even in transfer learn- ing, where the target of the model is changed, the input is at most expanded, but at least overlaps heavily with the data distribution that the model has previously seen in training. This is in contrast with meta learning (Finn et al., 2017; Nichol & Schulman, 2018; Lemke al., 2015; Vanschoren, 2018; Feurer et al., 2022; Hospedales et al., 2021; Zintgraf et al., 2021), where a model is trained to be adaptive to new datasets such that few gradient updates or fine-tuning are needed, instead of training a new model specialized to every distinct dataset from scratch. In meta learning, rather than modeling a specific dataset, the model is trained learn how to learn. This has multiple advantages. First, meta learning is highly adaptable (Huisman al., 2021; Finn et al., 2017: Frans & Witkowski, 2021) it learns more gen- eralized representations that can be adapted to new tasks and different domains. Second, meta learning makes efficient use of data (Finn et al., 2017; Gevaert, 2021) - supports learning from just few samples. Third, as a consequence of its efficient use of (small) data, the model can reach a point where it is able to make meaningful predictions very quickly ( (Vanschoren, 2018)
Abstract
We present an Adversarially Pre-trained Trans- former (APT) that is able perform zero-shot meta-learning on tabular prediction tasks without pre-training on any real-world dataset, extend- ing on the recent development of Prior-Data Fit- ted Networks (PFNs) and TabPFN. Specifically, APT pre-trained with adversarial synthetic data agents, who continue shift their underlying data generating distribution and deliberately challenge the model with different synthetic datasets. In addition, we propose a mixture block architecture that is able to handle classification tasks with ar- bitrary number of classes, addressing the class size limitation - a crucial weakness prior deep tabular zero-shot learners. In experiments, we show that our framework matches state-of-the-art performance small classification tasks without filtering on dataset characteristics such as number of classes and number of missing values, while maintaining an average runtime under one second. On common benchmark dataset suites in both clas- sification and regression, we show that adversarial pre-training was able to enhance TabPFN's perfor- mance. In our analysis, demonstrate that the adversarial synthetic data agents were able gen- erate more diverse collection of data compared to the ordinary random generator in TabPFN. In addition, we demonstrate that our mixture block neural design has improved generalizability and greatly accelerated pre-training.
In prior work, Verma et al. (2020) discussed the notion of zero-shot meta-learning. They train a generative adversarial network conditioned on class attributes, that can generate novel (previously unseen) class samples. This relies on the inputs present in the training data (class attributes) to be indicative of the new unseen classes. While they do not gradient updates on the unseen data for prediction, they rely on the input data coming at the very least from a very similar distribution to that of the training data. The scope of prob- lems this work aims to address is pristine zero-shot meta learning: given an unseen dataset from an unseen task after the model is pre-trained and deployed, can we do prediction this dataset without training the model on it? Specifi- cally, with zero gradient update on the model, and with no reliance on the context similarity between this dataset and the datasets that the model was pre-trained on. Note that this concept of zero-shot is slightly different from that in large vision and language models (Mann et al., 2020; Perez et al., 2021; Tsimpoukelli al., 2021; Cahyawijaya et al., 2024; Ahmed & Devanbu, 2022) the unseen datasets can
1. Introduction
In standard deep learning workflows, models are either trained per dataset, employed on data in form com-
University of California, Berkeley CCapitl One. Correspon- dence to: Yulun Wu <yulun_wu@bualeeeee www.ereeece...d..........
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Adversarially Pre-trained Transformer
entail heterogeneous fields or class labels that were never observed during pre-training, and zero-shot in this context refers to the amount of model optimization conducted be- ing zero given the unseen dataset rather than the amount of empirical examples seen being zero. The advantage of successfully establishing such a model is the exceptional generalizability and runtime.
bility of zero-shot meta-learning under the scope of tabular data prediction problems. Our contributions are listed as follows:
We propose an adversarial synthetic data pre-training approach on PFNs to establish a zero-shot meta-lcarner that is able to handle tabular prediction tasks with im- proved performance.
A few recent breakthroughs (Müller al., 2021; Hollmann al., 2022) have demonstrated that achieving this aspiration is possible: Müller et al. (2021) introduced Prior-Data Fitted Networks (PFNs). They pursue zero-shot meta-learning by using transformers pre-trained on synthetic data generated from a collection of prior distributions, to perform approx- imate Bayesian inference using in-context learning (Luo al., 2018; Mann et al., 2020). PFNs do not fit model on downstream training data, instead feeding training data into the context in forward pass and making predictions condi- tioned on the context. Hollmann et al. (2022) introduced a PFN specifically aimed at tabular datasets - TabPFN. A more detailed background review on PFNs and specifically TabPFN can be found in Appendix A. Tabular data- data or- ganized in rows and columns, and characterized by an unlim- ited heterogeneity of data fields, remains an area of machine learning where deep neural networks (DNNs) still struggle (Borisov et al., 2022; Shwartz-Ziv & Armon, 2022; McEl- fresh et al., 2024; Ye et al., 2024b) to push the boundaries of the otate-of-the-art gradient boosted decision trees (GBDTs) (Prokhorenkova et al., 2018; Chen & Guestrin, 2016; Ke et al., 2017), despite numerous approaches (Borisov al., 2022; Somepalli et al., 2021; Grinsztajn et al., 2022; Gor- ishniy et al., 2021; Rubachev et al., 2022; Levin et al., 2022: Kadra et al., 2021a; la; Arik & Pfister, 2021; Popov et al., 2019). Yet, tabular data is one of the most common data types in real-world machine learning (ML) applications (Chui et al., 2018; Borisov et al., 2022: Shwartz-Ziv & Armon, 2022). Although TabPFN has demonstrated exceptional zero-shot meta-learning capability on certain small tabular prediction tasks, we show that the distribution of synthetic data used in its pre-training is actually quite limited. Besides, the class size constraints of TabPFN pose significant limitation on its generalizability- - this might not be an important concern for the traditional mee-model-forr -or---ooo---ooonnee one-domain pipeline, but a crucial weakness for a zero-shot meta-learner (ZSMI.) since an unprecedented number of class labels could be present in inference time. Note that zero-shot meta-learning is largely similar to foundation modeling but slightly differ- ent in its scale and objective - it does not necessarily involve billions of parameters to learn the distribution of data and acquire token representations in a broad domain such as language or health records, but to model the general predic- tion logic and learn how to acquire data representations in unseen domains during inference time.
We eliminated the class size limitation for TabPFN on classification tasks by proposing the mixture block neural design, which yields a zero-shot meta-learner with better generalizability.
In experiments, we show that our framework achieves state-of-the-an performance on small tabular classifi- cation tasks without filtering on class size, feature size, number of categorical features number of missing values, and improved upon TabPFN in both classifica- tion and regression. We show that the adversarial data agents are able to enrich the synthetic data generating distribution, and the mixture block able to generalize to unseen class size and accelerate pre- pre-training.
2. Proposed Method
Our Adversarially Pre-trained Transformer (APT) model is pre-trained once offline using a mix of random synthetic data generators and adversarial synthetic data agents. In this phase, the goal of the model not to learn the specific pattern or probability distribution of any given dataset, but to learn the general prediction logic and means to represent various data, i.e. learning to learn. Once pre-trained and deployed, the model makes predictions on the testing set of any real-world dataset of interest in one forward pass, with- out performing any back- propagation or gradient updates of its weights. A demonstration of the workflow is shown in Figure 1. In Section 2.1, we describe the adversarial data agents in detail, whose goal is to continuously produce diverse and more challenging datasets for the meta-learning model during pre-training; in Section 2.2, we elaborate on the architecture of our transformer model, which has no restrictions on the class size of any real-world datasets prac- titioners provide.
2.1. Adversarial Data Agents
In the pre- rre-traiiiig phase, we compose a batch of m datasets (x(e), (8) Jisksm in each iteration using m different data generators [gi...., .mm 9m3 that each independently generate n number of data points, where x(k = [=] [=."))] = (k)
[ Lij lignjsd, and y = [v] [ are the predictor ma- trix and response vector (denoted as X and y when no index is specified) with feature size dk- We adopted the multi-layer perceptron (MLP) construction introduced in
Similar to Hollmann et al. (2022), we investigate the capa-
2

---

Adversarially Pre-trained Transformer
Pre-training
Inference
Data Agents
Real-world Dataset
( (xcc
(x
Xr00 yr
Xttaia trale
yerr
data
feedback
data
Test-masked Transformer
Test-masked Transformer
y
y
Figure 1. The model workflow of Adversarially Pre-trained Transformer (APT). Pre-training is done once, offline, with datasets generated by mix random synthetic data generators and adversarial synthetic data agents. The train-test split is randomly sampled for each batch of datasets. After the model is pre-trained and deployed, predictions done per real-world dataset, online, with one forward pass and no parameter update. The transformer is test-masked, meaning that each token only attends to training data tokens. cleanliness of the figure, only the attentions to and from the first training data token and the first testing data token plotted.
Hollmann al. (2022) for each generator instance, where predictors x; (k) and response yi are values of randomly selected neurons in sparsified noisy MLPs with some addi- tional pre-pre pre-processing More details regarding this approach can be found in Appendix A.1.
Relation Classic Adversarial Training In relation GANs (Goodfellow et al., 2014), the data agents here the generators, and the meta-learner is the discriminator. Contrary to classic adversarial training, there is no real ver- sus fake samples for the discriminator to distinguish in this context. The generator (data agent) and the discriminator (meta-learner) have one coherent competing objective: the meta-learner seeks to minimize the prediction loss on data generated by the data agents, while the data agent seeks to generate data that maximize the prediction loss by the meta-learner As result, the desired gradients for updating the discriminator is but flip of sign to its gradients calcu- lated through back propagation on the generator's 's objective. Hence, both the meta-learner and the data agents can be updated in one single iteration after loss calculation in this scenario. This results in more efficient adversarial training, and further reduce its potential of mode collapse with data agent reset described in the last paragraph of this sec- tion. Note that contrary to classic GANs, the discriminator is the final product in this context rather than the generator.
Different from Hollmann et al. (2022), instead of generating datasets solely from randomly initialized sparse MLPs, subset of the m generators in our framework are adversar- ial agents that learn from the model's performance on the generated data, and perform gradient ascent on the model's prediction loss. In other words, these adversarial agents chal- lenge the model by constantly shifting the synthetic data generating distributions to deliberately produce datasets that are more difficult for the model to handle. The loss for an adversarial agent 9, with respect to prediction model can be written as
((gg)) = Ex log go (D ( (X Via))
(1)
where (XI:, F1a) and (X 1).m are the train- ing and testing set split from generated dataset (X,y) y at position l. In the following sections, we refer to the for- mer (generators based on randomly initialized MLPs) as ordinary data generator, and the latter (generators based on adversarially updated MLPs) as adversarial data agents.
Discretization of Variables A key challenge in estab- lishing adversarial data agents is the gradient flow under discretization: how do we generate synthetic data with cat- egorical features while being able to perform end-to-end loss paopaaropagttion? Inspired by the Gummbel-Softmax
3

---

Adversarially Pre-trained Transformer
70
00
30
00
25
.05
20
10
15
10
06
10
embed
train
test-masked transformer blocks
mixture block
Scaled Dot-product
yored
project
Feedforward
train
Rest
(a) Model architecture for classification tasks
(b) Mixture block
Figure 2. Model architecture and the mixture block. a) X (Xtrrain Xtiett and grain embedded on Rdmodde using feature embedding block and linear projection respectively. Then, embeddings for X-trin and grrain are added as h train embeddings for X denoted as h t Emheddings (h" train h!ad) are then passed the transformer blacks with attention towards test embedding test masked, same as Hollmann et al. (2022). Finally, the outputs from transformer blocks are transformed to class probabilities through the mixture block for classification tasks, or directly transformed point predictions through standard dense final layer for regression tasks. b) For each data point in the testing set, we use its output after transformer blocks to query training data's 's outputs K. With two different dense feedforwards, two sets logits are predicted: one set of logits are used to calculate the scaled softmax probabilities these probabilities indicate how likely that the testing point is in the same class as the corresponding training points; the other set of logits are used to sample soft-discrete binary gates via Concrete distribution to sparsify these probabilitics. Finally, the gated probabilities from the same class are added together to yield the final predictions.
trick (Jang et al., 2016) and the Conerete distribution (Mad- dison et al., 2016), we propose continuous relaxation of discretization that naturally extends the ranking dis- cretization approach introduced in Hollmann et al. (2022), controlled by a user-specified temperature hyperparameter T. For the j-th feature column z., of a predictor matrix X and the corresponding N, - randomly sampled Gaussian quantiles 0.0² < <
stration of this conversion can be found on the right side of Figure 6 in the Appendix. Same Hollmann al. (2022), the extended ranking discretization approach decides the value of a categorical variable using only the continuous scalar Zijj- i.e. the value of one neuron in the sparsified noisy MLP, as opposed to the Gumbel-Softmat or Concrete distribution approach which would require selecting N, neu- rons as logits of the N, classes. In our early experiments, found that sampling multiple neurons to decide the value of one categorical feature achieved significantly worse per- formance than ranking discretization. Furthermore, since we do not desire to learn the explicit form of these distribu- tions, explicitly generating class logits is not a necessity, and hence we prefer a more efficient differentiable discretization technique that does not involve reparameterization tricks, softmax operations excessive samplings.
x/x:-1) at the initializa- tion of the corresponding data agent, the soft-discretization that converts the i-th value of the j-th feature Fi,y to a soft- categorical value with cardinality N2 is given by
(1 (1 2,,
(11 (1 (((022200),)) 1.
Jij Q
1+ à (1 20 1.) -0 (1
Data Agent Reset In terms of the diversity of generated data, there is a balance between adversarially updating the neurons in the MLPs and re-initializing the MLPs all to- gether. Although in the short run, re-initializing the MLPs and the corresponding random factors (number of features, number of classes, etc.) instantancously yield new datasets with a high chance of possessing much different fields and distributions from the previous, such generation is con- strained by the domain of distribution defined by the preset
where æ is a permutation function on integer domain 21,2,... ... N2 1), :00h p( + (mx.)) .) 200 for 1 < L < Nj are the unnormalized quantiles with bound- aries 200) min(x..). (x) and @) max(x.;). and He 2 )/ Σ, [(u 2 (() is the position of value in the ordered sequence (Q) ( hSigN,, A visual demon-
4

---

Adversarially Pre-trained Transformer
range of hyperparameters in the long run (we show some evidence on this in Section 3.2). On the other hand, al- though adversarial data agents are performance-driven and could explore distiiitiiiittionnnn regions better than ran- dom initialization, it also has the potential to converge to the Nash equilibrium and reach a stalemate with the meta- learner - for example, converging to state where generated predictors x and response y have no correlation. Hence, combine the two approaches and reset the adversarial data agents every Ne epochs to avoid such convergence. To speak from the GANs angle, we are letting the discriminator, i.c the meta-learner, to periodically gain an advantage and slightly beat the generator. Different from classic GANs, the discriminator the desired model here while the generator the supporting entity, hence exploration is more important than optimization for the generator in this context.
Contrary to tree-based methods, the splits are point-specific, i.e. there is a different split decided for each testing data point, and the decision within the split is weighted rather than via majority voting. Note that this approach does not change the order of computation complexity in terms of data size and data dimensions - it simply removes the final dense layer and adds two more multi-head attentions and feedfor- wards to the transformer architecture in non-sequential manner.
Large Data Size and Feature Size Compared to the class size limitation, the feature size limitation of PFNs is rel- atively less tricky in theory, and there are already a few straightforward solutions concurrent with this work (Holl- mann et al., 2025; Qu et al., 2025) that extend TabPFN's capabilities in handling datasets with larger number of fea- tures, as well as larger number of samples, Besides, the data capacity of PFNs could be adequately expanded by incorporating some of the recent advancements in general transformer and state-space model research (Wu et al., 2022: Bulatov et al., 2023). Therefore, we do not put emphasis on addressing these problems in this work, and make two sim- ple adaptations to APT based on patch embedding and batch aggregation in the event that prediction on large datasets is required. See Appendix B for details. Note that concurrent solutions such as Hollmann et al. (2025); Qu et al. (2025) do not pose conflict with our proposed architecture (mix- ture block only modifies the last layer of the model), thus can be naturally incorporated into our framework as the practitioners desire.
2.2. Mixture Block Architecture
Contrary to modern DNNs, traditional ML algorithms such as K-nearest neighbors and tree-based methods more flexible in terms of their ability to handle varying cardinality of classification labels, in the sense that they do not entail fixed-size dense layer parameters that cannot generalize to a different classification task with different label cardinality. This is not much of an issue for the traditional one-model- for-one-dataset MI. pipeline, but is of significant importance for zero-shot meta-learnes, yet unaddressed in prior works. Inspired by how rree-based methods solve classification tasks in a manner that is compliant to the empirical values and cardinality of training labels, we propose a scatter-sum mixture block as the output prediction head for classifica- tion tasks that significantly departs from the ordinary dense final layer approach. A visual demonstration can be found on the right of Figure 2. For each data point in the testing set, we use its embedding after the transformer blocks to query the embeddings of training data, and yield two sets of logits via two separate feedforwards: one set of logits is used to calculate softmax probability weights of keys and the other set is used to sample soft-discrete gates via Con- crete distribution (Maddison et al., 2016) to sparsify these weights. In essence, these gates govern the splits of training data in relation to the testing query, such that the final pre- diction only pays attention to subset of relevant training data representations. In our preliminary experiments, we discovered that sparsifying attention through these gates are crucial to performance, and the mixture block works poorly without this component. The output class probabilities are then acquired by a scatter summation of non-gated values using their original labels as index. Relating tree-based methods, the gates here are used to determine the subset of training data that are in the same split of leaf nodes as a given testing data point, and the weights are used to de- termine the relative importance of each label in that split.
3. Experiment
We evaluated our model and competing algorithms on com- mon ML benchmarking dataset suites for tabular classifi- cation and tabular regression problems. In Section 3.1, we show that APT achieves state-of-the-art performance on small tabular classification tasks with runtime comparable to that of TabPFN. In Section 3.2, we present qualitative analysis on the impact and characteristics of the adversarial data agents. In Section 3.3, we demonstrate the generaliz- ability of the mixture block and its effect on pre-training. In Section 3.4, we provide ablation study and show that adver- sarial pre- nre-training was able to enhance the performance of TabPFN on both classification and regression tasks.
Datasets For classification, we used the curated open- source OpenML-CC18 dataset suite (Bischl et al., 2021) containing 68 popular tabular benchmark datasets (4 vi- sion datasets mnist 784, CIFAR 10, Devnaguri- MaSHion-MNIST are not treated as tabular and removed from the total 72 datasets), and our main results are presented on all small datasets (number of samples no larger than 2, 2,000) in OpenMI--CC18 similar to Hollmann et al. (2022),
and
5

---

Adversarially Pre-trained Transformer
Table Performance of algorithms on small datasets with larger than 2,000 data points in the OpenMI--CC18 suite, given hour of time budget. Note that there are two styles of standard deviation (std.) calculation for AUC: 1) first take the mean of AUC across datasets, then calculate the std. across splits (std. of mean), as used by TabPEN (Hollmann al., 2022); 2) first calculate the std. across splits on each dataset, then take the mean across datasets (mean of std.), as used by TabZilla (McElfresh al., 2024). Our result table we present both std.'s 's here for clarity. The std. mean shows variation on suite level, which
largely adopted the style of TabZilla, more likely to result in statistical significance compared to mean of std., which shows average variation dataset level. The mean of AUC taken across splits are used as the scoring metric to calculate "Rank" and "Wins" of cach algorithm across datasets. many algorithms are tied for first, win assigned to each first-place algorithm. Same TabZilla (McElfresh al., 2024), the table is ordered by the mean of rank. The full results on each dataset for top algorithms are shown in Table 5 of Appendix D.
Time (sec.)
Rank
ROC-AUCT
Wins
(Tune + Train Predict)
mean
med.
min
max
mean
std of mean
mean of sld.
num.
mean
med.
APT
3.86
3
11
0.921
0.003
0.019
13
0.90
0.40
CatBoost
4.03
4
1
9
0.918
0,002
0,020
6
3542.42
3555.74
TabPFN
4.57
4
11
0.913
0.003
0.020
0.86
0.37
SVM
4.89
4
12
0.904
0.003
0.023
10
1175.58
481.50
XGBoost
5.37
5
1
10
0.914
0,006
0,020
4
3607.78
3598.91
LightGBM
5.60
6
0.917
0.003
0.019
3542.94
3582.07
LASSO-Logistic
6.69
8
12
0.908
0.001
0.023
1519.41
1227.52
Ridge-Logistic
6.91
8
1
11
0.907
0.001
0,022
1479.93
845.59
RandomForest
7.17
7
12
0.908
0.003
0.021
1736.71
1476.37
ResNet
7.69
9
12
0.825
0.004
0.040
3582.15
3597.41
KNN
9.57
11
1
12
0,884
0,006
0.024
127.82
77.31
SAINT
9.97
12
12
0.759
0.017
0.077
3597.41
3594.41
except that 1) there is no additional filtering, i.c. all datasets regardless of number of classes, number of features, num- ber of categorical features, and number of missing values are kept in our evaluation pool, composing a more general collection of datasets. This brings the number of datasets in the evaluation pool from 18 to 35; 2) The train-test split is set to 80-20 instead of the unconventional 50-50. For regression benchmarking, we used the curated open-source OpenML-CTR23 dataset suite (Fischer et al., 2023).
maximal class size of datasets in the OpenML-CCIS suite, For APT, all common hyperparameters shared with TabPFN are directly inherited from TabPFN. See Appendix C for more details. A total of 12.5% of the data generators are adversarial data agents during the pre-training of APT, with learning rate 10 weight decay 10- 05-5, soft-discretization temperature 10-2, and 2, 000 gradient steps between resets.
3.1. APT Achieves State-of-the-art Performance on Small Tabular Classification Tasks
Algorithms We compared APT to the top 3 GBDT algo- rithms (CatBoost (Prokhorenkova et al., 2018), XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017)) and the top 3 DNN methods (TabPFN (Hollmann et al., 2022), Tabular ResNet (Gorishniy et al., 2021), SAINT (Somepalli et al., )021)) in the main experiments of TabZilla (McEl- fresh et al., 2024), as well as 5 standard machine learning algorithms (KNN (Cover& & Hart, 1967), Ridge (Tikhonov, 1963), LASSO (Tibshirani, 1996), SVM (Cortes, 1995), Random Forest (Ho, 1995)).
We evaluated APT and benchmark models small datasets in OpenML-CC18 using area under the receiver operating characteristic curve (ROC-AUC) with the one-vs-one (OVO) multi-class evaluation configuration, similar to Hollmann et al. (2022). Previously. Hollmann et al. (2022) has shown that TabPFN matches the performance of state-of-the-arr GBDT algorithms and outperforms them on small datasets that have less than 100 features, less than 10 classes, no categorical features, and no missing values in their main results. In this work, we do not impose any of these re- strictions to further examine APT's and TabPFN's zero-shot meta-learning capability. The results are presented in Table 1. For datasets with number of features larger than 100, we subsample 100 features similar to (McElfresh et al., 2024). In these experiments, APT achieved state-of-the-art per- formances with a runtime similar to that of TabPFN. The average runtime of APT increased by 4.6% compared to
Hyperparameters The hyperparameter search space of benchmark models is directly inherited from Hollmann et al.
(2022), and directly inherited from McElffreh et al. (2024) if the benchmark model is not in Hollmann et al. (2022). TabPEN pre-trained with hyperparameters directly in- herited from their released checkpoint, only changing the maximum number of classes from 10 26, which is the
6

---

Adversarially Pre-trained Transformer
0.00
0.00
0.00
-0.02
-0.02
-0.02
0.04
-0.04
-0.04
-0.06
-0.06
-0.06
-0.02
0.04
-0.02
0.04
(a) Distribution of data in datasets generated (b) Distribution of data generated by another (c) Distribution of data generated by set of independent set ordinary generators adversarial data agents
by set of ordinary data generators
Figure 3. Contour plot of two-dimensional data generated by ordinary data generators and adversarial data agents. Each subplot contains total of 100,000 data points from 2,000 datasets. Note that subplot (a) and subpiot (b) are two independent sets of ordinary generators with mutual, each dataset is generated by independently initialized random sparse neural network. Each dataset in subplot (c) generated by an adversarial data agent after each consecutive loss back-propagation
TabPFN and remained within second on GPU (NVIDIA H100), showing that neural modifications from the mix- ture block have not made APT significantly heavier. Note that there is no cherry-picking being performed on model checkpoints for APT the APT model that we released and used for evaluations is the last model after the final itera- tion of pre-training. Realistically, PFN-based models are pre-trained on synthetic data, and picking checkpoints for evaluations ad hoc is not ideal unless using whole different collection of real-world datasets for validation. But even in that case, it would still raise the concern of data leakage.
From Figure 4, it can be observed that APT has fairly dealt with TabPFN's weakness in handling datasets with missing values, and has closened the gap between performance on datasets with and without categorical features compared to TabPFN, although GBDTs such CatBoost still shows the greatest capability in handling datasets with categorical fea- tures. We further break down the performance contributions from each proposed component in Section 3.4.
3.2. Qualitative Analysis of the Adversarial Data Agents
Even though arbitrary MLPs have the potential to serve as universal function approximators given certain regularity constraints (Hornik et al., 1989), the pre-set hyperparame- ters (e.g. sampling distribution of neurons, sampling distri- bution of the number of layers, choices activations, etc.) as well the lack of gradient updates restrict the family of data distributions that randomly initialized sparse neural networks put forward in practice. As shown in Figure 3, the distribution of two-dimensional data generated by two whole different sets of random sparse neural networks align fairly precisely with merely 2,000 independent initializa- tions. On the contrary, even without resetting neural archi- tecture and neural parameters, the adversarial data agents still managed generate a more diverse collection of data and diffuse the concentrated peaks presented in the density distribution of data generated by ordinary data generators. To be exact, for collection of 2, 000 datasets generated by ordinary data generators, we evaluated a KL-divergence of 0. 1344 0.141 between it and a collection of 2, 000 datasets generated by another set of ordinary data generators, and a KL-divergence of 0.813 + 0.072 between it and collection of 2, 000 datasets generated by adversarial data agents.
In these experiments, the deep learning algorithms under the standard supervised learning pipeline, ResNet and SAINT, yielded subpar performances. Note that the computing bud- get in Hollmann et al. (2022) and ours is set to hour per dataset per split contrary to the 10 hours in McElfresh et al. (2024). The deep learning algorithms under the zero-shot meta-learning pipeline, APT and TabPFN, yielded ideal per- formances, but has been previously shown that TabPFN sees significant drop performance on datasets with cate- gorical features or missing values (Hollmann et al., 2022). In Figure 4, we further break down the results on datasets with and without these characteristics.
Has Categorical Features
Has Missing Values
APT
False
CatBoost
Tre
TabPFN
SVM
XGBoost
LightGBM
Ranks
Ranks
Figure 4. A breakdown of performance by dataset characteristics. The mean ranks are plotted orange on datasets with the respec- tive characteristic, and as blue on datasets without the respective characteristic.
As motivation of imposing data agent reset, we were wary that the data agents after many adversarial updates could yield synthetic datasets whose features have little to no
7

---

Adversarially Pre-trained Transformer
signal on the response variable. With our hyperparame- ter settings, we have not observed such behavior and to our surprise, the synthetic datasets generated by advensarial agents exhibit slightly stronger signal with Pearson corre- lation of 0.311 + 0.026 between predictors and responses on datasets with two-dimensional features as oppose to the 0.268 0.013 of ordinary data generators. We postulate that this is partially in consequence of the high reset frequency and high generator learning rate.
learn remarkably faster than models without it. For APT, the model reaches AUC of 0.70 in merely 40 gradient steps, AUC of 0.80 in 591 gradient steps and 0.90 in 11,780 gradient steps.
3.4. Ablation Study
Classification Although we discovered that the mixture block gives the model nice performance acceleration in the previous section, the original purpose of designing such ar- chitecture was not performance-driven. and we still expect that the final performance improvement was largely con- tributed by the adversarial pre-training. We present ablation study in Table 3 verify this expectation.
3.3. Generalizability of the Mixture Block
After a ZSML is deployed, one should not be required to re-do its pre-training given certain characteristics of the datasets in evaluation pool that the model cannot handle, and this is why the mixture block architecture is important. For TabPFN, we have to look at the evaluation dataset pool first, calculate the largest class size, before using it as a hyperparameter for pre-training. This not procedure that fits well into the zero-shot learning concept. Our proposed mixture block architecture does not have such class size limitation, and show the performance of APT on datasets with more than 10 classes in OpenML-CC18 8 in Table 2.
Table 3. Ablation study on tabular classification. Note that APT is TabPFN with adversarial pre- wve-training and mixture block.
Small
All
mcan AUC
rank
mcan AUC
rank
APT APT wro Misture
0.921 = 0.003
2.11 0.16
0918 + 0.006
2
0,917 0.005
2,09 : 0.06
0917 0.005
2.1
TabPFN w/ Mixture TabPFN
0.914 = 0.004
2.55 = 0.22
0.914
2.6
0.913 0.003
2.49 0.16
0.015
2.4
Unsurprisingly, models with and without the mixture block did not dominate each other mean AUC and rank col- lectively. Note that the mixture block was proposed for generalizing on datasets with unseen number of classes, and we expect it to have little to no impact on datasets with seen number of classes performance-win
Table 2. The ROC-AUC on datasets with more than 10 classes. APT pre-trained on datasets with maximum of 10 classes able to match APT without mixture block pre-trained on datasets with maximum of 26 classes on 3 the datasets.
letter
isolet
vowel
texture
APT w/o Mixture APT
.975 .02
.970 + .003
1
1
.95 + .02
.999 + 0111
1
1
Regression Although ZSMLs are gradually catching up with GBDTs on classification problems and likely reached a performance mark close saturation on small classification problems, tabular regression remains an area where ZSMLs have not yet shown exceptional performance. We addition- ally report a study on the 35 datasets in OpenML-CTR23 regression suite in Table 4, and show the progress APT has made on regression tasks over TabPFN.
Interestingly, the mixture block's generalizability signif- icantly accelerated pre-training in our experiments. The ROC-AUC evaluated after each iteration of pre-training with and without the mixture block is presented in Figure 5. Note that ensembling over permutations (Hollmann et al., 2022) is not performed in this experiment as it would dramat- ically increase runtime given that evaluation performed following every gradient step.
Table 4. Ablation study tabular regression. Small datasets the 12 datasets in OpenML-CTR22 with data size no larger than 2,000. Note that APT is TabPFN with adversarial pre-training this setting, since the mixture block was only used for classification tasks.
0.9
0.8
ROC-AUC AUC
0.7
Small
All
PFN (no Mixture)
0.6
APT w/o Mixture
MSE
mean MSE J
wins
0.5
APT (has Mixture)
PFN w/ Mixture
TahPFN
0.412 + 0.077
3.8 1.2
0.340 + 0.025
6.4 + 1.4
APT 0.344 + 0.068
8.2
1.2
0.306 +: 0.023
28.6 1.4
10000
20000
30000
Iterations
From Table 4, it can be observed that incorporating adver- sarial pre-training has boosted the performance of TabPFN, yielding a larger number of wins with a significant margin. Note that we used the exact same synthetic data sampling distributions and hyperparameters that were used in TabPFN
Figure ROC-AUC on the 35 small datasets OpenML-CC18 evaluated after each of the first 30,000 gradient steps.
From Figure 5, we can see that models with mixture block
8

---

Adversarially Pre-trained Transformer
for the purpose of ablation, in order clearly demonstrate the contribution of adversarial training. TabPFN was trained only on classification problems, and therefore it is possible that these hyperparameters are over-optimized for classifica- tion tasks, and under-optimized for regression tasks.
ingful, but not guaranteed for broad tabular data problems.
4.3. Adversarial Training
Upon generative adversarial networks (GANs) (Goodfellow et al., 2015; Madry et al., 2018; Kurakin al., 2017), recent work such as Shafahi et al. (2019) improved on the efficiency by combining the back-propagation steps of the generator and discriminator. However, this method has been shown to suffer from catastrophic overfitting (Andriushchenko & Flammarion, 2020; Kim et al., 2021) without further modi- fications. Other works focusing on improving the efficiency of GAN training include Wong et al. (2020) and Zhang et al. (2019) where they restrict most of the forward and back propagation within the first layer of the network dur- ing adversarial updates. Zhang et al. (2021) in particular noted that weight updates frequently go back and forth in opposite directions in one training epoch, suggesting those updates might be redundant. Many other variations have been introduced to mitigate vanishing gradient and addi- tional challenges of GAN training (Jabbar et al., 2021): failing finding Nash-equilibrium (Ratliff et al., 2016), and internal covariate shift (loffe, 2015).
4. Related Work
4.1. Tabular Learning
GBDTss such XGBoost and others (Chen & Guestrin, 2016; Prokhorenkova et al., 2018; Ke et al., 2017) are com- monly used for tabular data problems, in the traditional one- codel-for-oneedattee approach. At this point, numerous deep learning approaches have been developed for tabular data, mostly taking the one-modell-forooee-dae (Borisov et al., 2022; Somepalli et al., 2021; Gorishniy et al., 2021; Rubachev et al., 2022; Kadra et al., 2021a; Arik & Pfister, 2021; Popov et al., 2019; Arik & Pfister, 2021; Kotel- nikov et al., 2023; Gorishniy et al., 2024; 2022; Chen et al., 2024; Kadra et al., b: Huang et al., 2020), but some also venturing into transfer learning, many but not all leveraging large language models to find relevant information for the tabular data problem at hand (Levin et al., 2022; Yan et al., 2024; Borisov et al., 2023; Ye et al., 2024a; Spinaci al., 2024; Hegselmann et al., 2023; Kim et al., 2024; Zhu et al., 2023).
approach
5. Conclusion
In this work, we gave the first effort in exploring the adver- sarial pre-training of deep zero-shot meta-learners, specifi- cally PFNs. We proposed APT, zero-shot meta-leamner that improves the performance of TabPFN tabular prediction tasks and matches state-of-ther thrt GBDTss on small tabular classification tasks. In addition, we proposed a mixture block neural design to eliminate the class size restriction of PFNs, addressing a crucial issue in their generalizability to broad classification problems. As for limitations, APT does not outperform GBDT's on large tabular datasets, and shares the quadratic computational complexity of TabPFN. Hence, extensions of this work could explore means of acquiring data representations in a more inexpensive manner. For example, considerable research in recent years has signifi- cantly accelerated the transformer and increased its context length (Wu et al., 2022; Bulatov et al., 2023). It is a worth- while effort for future research apply these advancements to APT as well as other PFNs. Besides, future research could extend the mixture block to standard (non-zero-shot) classification settings in light of its ability to generalize and greatly accelerate convergence, which could improve the performance of traditional DNNs on small classification datasets. Mixture block or other alternatives to the dense final layer could also be explored in both standard and zero- shot regression settings, which could have an impact on the inductive bias of DNNs and their underperformance in com- parison to GBDTs (Grinsztajn et al., 2022) under certain tabular data nature.
Tabular Meta-Learning Auto-Skleam introduced in Feurer al. (2015) and improved upon in Feurer et al. (2022) use Bayesian optimization to determine the best algo- rithm and feature pre- pre-prceessing steps for modeling a given dataset. Meta learning is used for initializing the Bayesian optimization. In contrast to Auto-Sklearn and methods of transfer learning for tabular data, TabPFN (Müller et al., 2021) is trained solely on synthetic data to learn the gen- eral prediction logic of tabular classification and acquire meaningful data representations in inference time. Helli et al. (2024) introduced a variant TabPFN that was trained on a drifting synthetic data distribution, but the drift is inde- pendent of the performance of the model being optimized.
4.2. Zero-shot Learning
Recent work such as Xian et al. (2018; 2017); Chang et al. (2008); Larochelle et al. (2008); Palatucci et al. (2009) have shown impressive capability of zero-shot learning in the space of language and vision problems. Recent approaches to zero-shot or few-shot learning for tabular data problems mostly encode tabular data language, and then leverage large language models (LLMs) for their zero- or few- ca- pabilities (see Hegselmann et al. (2023); Nam et al. (2023); Gardner et al. (2024)). These approaches rely on relevant information about the tabular data existing in LLMs- - this most obviously the case when column names are mean-
9

---

Adversarially Pre-trained Transformer
Acknowledgements
Chen, J., Yan, J., Chen, Q. Chen, D. Z.,, Wu, J., and Sun, J. Can deep learning model be sure bet for tabular prediction? In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '2, pp. 288-296, New York, NY, USA, 2024, Asso- ciation for Computing Machinery. ISBN 9798400704901. doi: 10. 145/3637528.36718899 URL https: :ttp:::dsi..... 36718999
We thank Tyler Farnan, Gang Mei, and C. Bayan Bruss for the insightful discussions.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
org 10
Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd inter- national conference on knowledge discovery and data mining, pp. 785-7944 2016.
References
Ahmed, and Devanbu, P. Few-shot training ilms for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Auto- mated Software Engineering, pp. 1-5, 2022.
Chui, M.. Manyika, J., Miremadi. M., Henke, N., Chung, R., Nel, P., and Malhotra, S. Notes from the ai frontier: Insights from hundreds of use cases. McKinsey Global Institute, 2:267, 2018.
Andriushchenko, M. and Flammarion, N. Understanding and improving fast adversarial training. Advances in Neu- ral Information Processing Systems, 33:16048-16059, 2020.
Cortes, C. Support- networks. Machine Learning, 1995.
Cover, T. and Hart, P. Nearest neighbor pattern classification. IEEE transactions information theory, 13(1):21-27, 1967.
Arik, S. Ó. and Pfister, T. Tabnet: Attentive interpretable tab- ular learning. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 6679-6687, 2021.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010. 2020. URL
Bischl, B., Casalicchio, G., Feurer, M., Gijsbers, P., Hutter, F., Lang, M., Mantovani, R. G., van Rijn, J. N., and Van- schoren, J. Openml benchmarking suites. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmurks, 2021.
Borisov, V., Leemann, T., Seßler, K., Haug. J., Pawelczyk, M., and Kasneci, Deep neural networks and tabular data: survey. IEEE transactions on neural networks and learning systems, 2022.
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems 28 (2015), pp. 2962-2970, 2015.
Borisov, V., Sessler, K., Leemann, T., Pawelezyk, M., and Kasneci, G. Language models are realistic tabular data generators. In The Eleventh International Conference Learning Representations, 2023. URL https:// openreview.ner
Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., and Hutter, F. Auto-sklearn 2.0: Hands-free automl via meta-learning. Journal of Machine Learning Research, 23:1-61, 61, 2022.
Bulatov, A., Kuratov, Y., and Burtsev, M. S. Scal- ing transformer to Im tokens and beyond with RMT. CoRR, abs/2304. 1062, 2023. doi: 10.48550/ARXIV. 2304.11062, URL arXiv. 2304 11062.
Finn, C., Abbeel, P., and Levine, Model-agnostic meta- learning for fast adaptation of deep networks. In Interna- tional conference machine learning, pp. PMLR, 2017.
48550/
Cahyawijaya, S., Lovenia, H., and Fung, P. Llms are few- shot in-context low-resource language learners. arXiv preprint arXiv:2403. 165/2, 2024.
Fischer, S. F., Feurer, M., and Bischl, B. Openml-ctr23-a cu- rated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop). 2023.
Chang, M.-W., Ratinov, L.-A., Roth, D., and Srikumar, V. Importance semantic representation: Dataless classifi- cation. In Aaai, volume 2, pp. 830-835, 2008
Frans, K. and Witkowski, O. Population-based evolu- tion optimizes a meta-learning objective. arXiv preprint arXiv:2103.06435, 2021.
10

---

Adversarially Pre-trained Transformer
Gardner, J., Perdomo, J. C., and Schmidt, L.. Large scale transfer learning for tabular data via language model- ing, 2024. URL 12031.
Hollmann, N., Müller, S., Eggensperger, K., and Hut- ter, F. TTbpff:: A transformer that solves small tabu- lar classification problems in second. arXiv preprint WrXiv:220700148 2022.
Gevaert, O. Meta-lcarning reduces the amount of data needed to build ai models in oncology. British Journal of Cancer, 125(3):309-33 2021.
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. Accurate predictions on small data with tabular foundation model. Nature, 637(8045) 2025.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warle- D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neu- rul networks, 2(5):359-366, 1989.
Goodfellow, I., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http:/ /tte//axxxxe arxiv. org/ abs/1412.655 6572..
Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning neural networks: survey. IEEE transactions on pattern analysis and machine intelligence, 44(9)5544--516, 2021.
Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. Revisiting deep learning models for tabular data. Ad- vances in Neural Information Processing Systems, 34: 18932-18943, 2021.
Huang. X., Khetan, A., Cvitkovic, M., and Karnin, Tab- transformer: Tabular data modeling using contextual beddings, 2020. URL https: /ttp:///aaxvvvoo/ org.orggg/ao...... abs/ 2012.06678.
Gorishniy, Y., Rubachev, I,, and Babenko, A. On embed- dings for numerical features in tabular deep learning. In NeurlPS, 2022.
Huisman, M., Van Rijn, J. N., and Plaat, A. A survey of deep meta-leaming. Artificial Intelligence Review, 54(6): 4483-4541, 2021.
Gorishniy, Y., Rubachev, I,, Kartashev, N., Shlenskii, D., Kotelnikov, A., and Babenko, A. Tabr: Tabular deep learning meets nearest neighbors. In The Twelfth In- ternational Conference on Learning Representations, http://pennvieeeeeeeeeeeeeeeeeee nwt.//rrntrtt
loffe, S. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint / 2015.
2024. URL id=rhgIgTSSxW.
Jabbar, A., Li, X., and Omar, B. A survey on generative adversarial networks: Variants, applications, and training. ACM Computing Surveys (CSUR), 54(8):1-49, 2021.
Grinsztajn, I.., Oyallon, E., and Varoquaux, G. Why do tree- based models still outperform deep learning on typical tabular data? Advances in neural information processing systems, 35:507-520 2022.
Jang, E., Gu, S., and Poole, B. Categorical repa- rameterization with gumbel-softmax. arXiv preprint 1611v:161..011 144, 2016.
Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X. and Sontag. D. Tabllm: Few-shot classification of tabular data with large language models. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 5549-5581. PMLR, 25-27 Apr 2023. URL hwwps:tttss... htt.ttttss////oeeeee http://oocceedeeeoooo wgw.occeediig....o....o. ngs mlr, .press/ v206 2006/hegselmann22
Kadra, A., Lindauer, M., Hutter, F., and Grabocka, J. Well- tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 2021a.
Kadra, A., Lindauer, M., Hutter, F., and Grabocka, Well- tuned simple nets excel on tabular datasets. In Beygelz- imer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Sys- tems, 2021b. URL https:/ / http///eeeeeeeeess.e.eee forum?id=d3k3381m0
html.
Helli, K., Schnurr, D., Hollmann, N., Müller, S., and Hutter, F. Drift-resilient tabPFN: In-context learning distribution shifts tabular data. In AutoML Con- ference 2024 (Workshop Track), 2024. URL https: // //openreve/ eww n http://ooommmm/////eeoeomo d=VomqcoHpTT.
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: highly efficient gradient boosting decision tree. Advances in neural infor- mation processing systems, 30, 2017.
Ho, T. K. Random decision forests. Proceedings of 3rd international conference on document analysis and recognition, volume 1, pp. 278-282. IEBE, 1995.

---

Adversarially Pre-trained Transformer
Kim, H., Lee, W., and Lee, J. Understanding catastrophic overfitting in single-step adversarial training. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35. pp.
Müller, S., Hollmann, N., Arango, S. P., Grabocka, J., and
Hutter, F. Transformers can do bayesian inference. arXiv preprint arXiv:2112.10510, 2021.
2021.
Nagler, T. Statistical foundations of prior-data fitted net- works. In International Conference on Machine Learning, pp. 25660-2561777 PMLR, 2023.
Kim, M. J., Grinsztajn, L.,, and Varoquaux, G. CARTE: Pretraining and transfer for tabular learning. In Forty- first International Conference on Machine Learning, 2024. URL https:// http///pperreeeeeeeeeeee/e net. net/forum? forum? id 1d=9kA:OnKLDp.
Nam, J., Tack, J., Lee, K., Lee, H., and Shin, J. STUNT: Few-shot tabular learning with self- generated tasks from unlabeled tables. In The Eleventh International Confer- ence Learning Representations, 2023. URL https: //openreview. net/
Kotelnikov, A., Baranchuk, D., Rubachev, I., and Babenko, A. TabDDPM: Modelling tabular data with diffusion models, 2023. URL ptttp:://opereveeee n forum? d=
x1s jehDvvy.
Nichol, A. and Schulman, J. Reptile: scalable metaleam- ing algorithm. arXiv preprint arXiv: 2(3):4, 2018.
Kurakin, A., Goodfellow, I. J., and Bengio, S. Adversarial machine learning scale. In International Conference on Leurning Representations, 2017. URL https:// mett/foouue d=BJm4444Kg)
Palatucei, M., Pomerleau, D., Hinton, G. E., and Mitchell, T. M. Zero-shot learning with semantic output codes. Advances in neural information processing systems, 22, 2009.
openreview, seeerreiew.net//orroee
Larochelle, H., Erhan, D., and Bengio, Y. Zero-data learning of new tasks. In AAAI, volume I, pp. 3, 2008.
Perez, E., Kiela, D., and Cho, K. True few-shot learning with language models. Advances in neural information processing systems, 34:11054-11070, 2021.
Lemke, C., Budka, M., and Gabrys, B. Metalearning: survey of trends and technologies. Artificial intelligence review, 44:117-130, 2015.
Popov, S., Morozov, S., and Babenko, A. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.0633 2019.
Levin, R., Cherepanova, V., Schwarzschild, A., Bansal, A., Bruss, C. B., Goldstein, T., Wilson, A. G., and Goldblum, M. Transfer learning with deep tabular models. arXiv preprint arXiv:2206. 15306, 2022.
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409, 2021.
Luo, R., Tian, F., Qin, T., Chen, E., and Liu, T.-Y. Neural ar- chitecture optimization. Advances in neural information processing systems, 31, 2018.
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. Catboost: unbiased boosting with categori- cal features. Advances neural information processing systems, 31, 2018.
Maddison, C. J., Mnih. A., and Teh, Y. W. The concrete distribution: continuous relaxation of discrete random variables. arXiv preprint arXiv: 1.00712 2016.
Qu, J., Holzmüller, D., Varoquaux, G., and Morvan, M. L. Tabicl: A tabular foundation model for in-context learn- ing on large data. arXiv preprint arXiv:2502.05564 2025.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018 URL https: openreview.o wwt. 1
Ratliff, L. J., Burden, S. A., and Sastry, S. S. On the characterization of local nash equilibria in continuous games. IEEE transactions on automatic control, (8): 2301-2307, 2016.
Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., Agarwal, S., et al. Language models are few-shot learners. arXiv preprint 11X:: 1, 2020.
Rubachev, I., Alekberov, A., Gorishniy, Y., and Babenko, A. Revisiting pretraining objectives for tabular deep learning. arXiv preprint arXiv:2207.0322 2022.
McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., and White, C. When do neural nets outperform boosted trees on tabular data? Advances in Neural Information Processing Systems, 36, 2024.
Shafahi, A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson,
J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! Advances in neural information processing systems, 32, 2019.
12

---

Adversarially Pre-trained Transformer
Shwartz-Ziv, R. and Armon, A. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90 90, 2022. Somepalli, G.. Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. Saint: Improved neural networks for tabular data via attention and contrastive pre- training, arXiv preprini arXiv:2
Yan, J., Zheng, B., Xu, H., Zhu, Y., Chen, D., Sun, J., Wu, J., and Chen, J. Making pre-trained language models great on tabular prediction. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openrevile ew net/ / forum? www.znzznnuu......
2021.
Ye, C., Lu, G., Wang, H., Li, L.,, Wu, S., Chen, G., and Zhao, J. Towards cross-table masked pretraining for web data mining, In The Web Conference 2024, 2024a. URL https: http::///ppeeeeeww...//// fet.iiwwwmee//ommm id=9jj7cMOXCo.
Spinaci, M., Polewezyk, M., Hoffart, J., Kohler, M. C., The- lin, S., and Klein, T. PORTAL: Scalable tabular founda- tion models via content-specific tokenization. In NeurlPS 2024 Third Table Representation Learning Workshop, 2024. URL https: http://pppeeeeeeeeeeeeeeeee net/ forum? id Td=22v0kKbLO.
Ye, H.-J., Liu, S.-Y.. Cai, H.-R., Zhou, Q.-L., and Zhan, D.-C. A closer look at deep learning on tabular data. CoRR, 48550/ arxiv. 2407.00956
2024b. URL https: /doi.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09888 2021.
org/10 /t///..48555
Zhang, D., Zhang, T., Lu, Y., Zhu, Z., and Dong, B. You only propagate once: Accelerating adversarial training via maximal principle. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Gamett, R. (ods.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/ сc ca/paper wwwps:/ewteeeeeeeeeeeeeeeaeeeeet 81 hitpppeesiieeepppppe///////////1 nnw.eeeisd.nnnnnnnn.......... neurips. pdf.
Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288) 1996.
Tikhonov, A. N. Solution of incorrectly formulated prob- lems and the regularization method. Sov Dok, 4:1035- 1038, 1963.
Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S., Vinyals, O., and Hill, Multimodal few-shot learn- ing with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
Zhang, H., Shi, Y., Dong, B., Han, Y., Li, Y., and Kuang, X. Free adversarial training with layerwise heuristic learning. In International Conference Image and Graphics, pp. 120-131. 131. Springer, 2021.
Vanschoren, Meta-learning A survey. arXiv preprint arXiv:1810.03548, 2018.
Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., and Shoaran, M. Xtab: Cross-table pretraining for tabular transformers. arXiv preprint arXiv:2305.06090 2023.
Verma, V. K., Brahma, D., and Rai, P. Meta-learning for generalized zero-shot learning. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 6062-6066 6069, 2020.
Zintgraf, L., Schulze, S., Lu, C., Feng, L., Igl. M., Shiarlis, K., Gal, Y., Hofmann, K., and Whiteson, S. Varibad: Variational bayes-adaptive deep rl via meta-learning. Journal of Machine Learning Research, 22(289):1- 39, 2021. URL http::/tp//// http://.orgppprr//////pop/ 21-0657, 11--657.htm..
Wong, E., Rice, L., and Kolter, Z. Fast better than free:
Revisiting adversarial training. In International Confer- on Learning Representations, 2020. URL https: / //openreview net. ftt///orum///// w 4=BJx04 OFFVH.
Wu, Y., Rabe, M. N., Hutchins, D., and Szegedy, C. Memorizing transformers. In International Conference on Learning Representationss 2022. URL https:// openreview.nor http::/ee///tooeeeeeeee dot.rruummddd-rrrrxxoeoooe..
Xian, Y., Schiele, B., and Akata, Z. Zero-shot learning the good, the bad and the ugly. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Xian, Y., Lampert, C. H.. Schiele, B., and Akata, Z. Zero- shot learning- a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern anal- ysis and machine intelligence, 41 41(9):2251-2263 2018.
13

---

Adversarially Pre-trained Transformer
A. Background
In this section, we give a brief introduction to PFNs, and specifically the synthetic data generating mechanism TabPFN. For more complete description, see Müller al. (2021); Hollmann et al. (2022); Nagler (2023). Given training dataset Dtrain = (Xtrain y g the goal is to approximate the conditional outcome distribution y p(-|2x D given test point ae In the Bayesian framework for supervised learning, the prior of the dataset is hypothesis of the data generating mechanism & drawn from hypothesis space Q under which p( pl-larlest Dirsin) is posterior predictive distribution (PPD) and can be factorized as follows by the Bayes' rule:
/
p(-|a> test Dtrain)
p(-lax .b)p(o) de
(3)
JOc
test .)) p(p)p( Dtrain (6)
p(-|2x
(4)
dEo
p D(Drai)))
pl-1artest ,d) (Dtrain (o)) déb
(5)
0
PFNs conduct synthetic prior fitting by defining family of data generating mechanisms & from which independent samples a; ~ p( (r:)) = pt) [p(ar,(b)) [(b] and Yi ~ p(us) op) [p(ys [m,, 6)] are drawn to compose feature matrix (Xusiin Xtest)
and response vector ( traann yrest) ) of synthetic dataset D) = Dtrain U Diesst and use transformer model g- Ditain
to approximate p(-|Xtsst test Dirain) by minimizing their expected divergence over the synthetic data distribution
Ep(Diciin,xeo [KL (p(y test test Dirain), go(y best XX test D Dtrriin)
(6)
Since
V Bpteaann Derain Xxmee
KL (ply (est X Derain), ), 2o ( Dtrain)) ]]
(7)
BpBproniiin X
H (p(y* eest X Xtest Diraiu), ) (o test X Derain))
(8)
Vp Dirain )
[- log go best |xtoss Dtrain)]
(9)
is equivalent to minimizing the expected negative log- logllikelihood loss
[(go)= = EpDD) [- log go (ytsstxxtest |
(10)
TabPFN in particular, conducts synthetic prior fitting by defining a family of sparsified-andoo-MMLPPP data generating mechanisms L which we call ordinary data generators in the context of this paper. The following section gives a detailed description of the workflow of these generators.
A.1. Ordinary Data Generator
To sample data generating mechanism á ~ Φ TabPEN first initializes a random MLP by sampling a collection of hyperparameters from a pre-defined hyperparameter space, including number of layers, hidden size, activation function, dropout probability, noise scales, etc. Specifically,, dropout probability used to sparsify neural connections between neurons, and noise scales dictate the amount of random noise injected into neurons each layer. After the sparsified noisy random MLP is initialized, TabPFN randomly selects subset of neurons in this MLP to be predictors æ;, and randomly select one neuron to be response ys- With n different random inputs to the MLP, dataset with n. instances of (a, y) is thus generated.
Discretization Since generated data are selected neurons from MLPs, their values are naturally continuous. To mimic real-world datasets that possess categorical features and to generate discrete class labels for classification tasks, TabPFN uses ranking discretization approach that converts subset of continuous values to discrete by designating certain quantile ranges of the continuous value to certain categories. A visual demonstration of this conversion can be found on the left side of Figure 6.
The generated synthetic data (as well as real-world datasets during inference time) are normalized across samples within each dataset, with the range of the values clipped to four standard deviations. Although the meta-learner might see datasets with unseen fields and -isttribbbtt predictor-respons. relations during inference time, this ensures that at least the range of values will not be disttribittitnno as well.
Normalization
14

---

Adversarially Pre-trained Transformer
Pent
cat
5
3
1
Pmin
Vmax
Pmin
Pmax
(a) Ranking Discretization
(b) Ranking Soft-discretization
Figure 6. Discretization continuous variables. a-axis the value generated by the data generator, and y-axis is its value after discretization. The soft-diserctization approach produces near-categorical features that are differentiable and thus do not disrupt gradient flow. Intuitively, adversarial agents will try to produce new value that escapes range the current category the meta-leamer becomes very good at identifying signal from the current category. However, the new category it escapes to is arbitrary and cannot be targeted by gradient updates, giving additional exploration potentials to adversarial agents.
A.2. Limitations
Although there no theoretical limitation the number of data PFNs can handle, the transformer architecture does entail significant computation complexity and memory usage for large datasets. Besides, given the nature of dense input embedding layer and dense final prediction layer, there is theoretical limitation on the number of features and the number of classes that PFNs can handle. The former is less of an issue since feature selections simply random sampling of features can be performed, and PFNs would still yield ideal performance as shown in McElfresh et al. 2024. The latter is a rather big problem for classification tasks because there hardly any direct and effective work-around.
B. Handling of Large Datasets
Since expanding TabPFN's capabilities in handling large datasets is not the focus of this work, we only provide two simple adaptations such that APT can practically handle datasets of this nature. We recommend that practitioners try out the concurrent and future developments in more involved model innovations for large datasets, but in case they do not wish to do so, the following approaches can serve as a baseline.
B.1. Uncertainty-based Batch Aggregation
For datasets with large number of samples, to avoid calculating attention spanning all training data points which results in quadratic order of operations and memory usage with respect to data size, we estimate the PPD with batches drawn from the training set:
S (0 test trest b) p(b Drraing,
ply pest |teett Dtrain) x
(11)
which is equivalent drawing uniform samples from training set Dtrain and scale the resulting predictions with weights p(b Dtrain). We cap the batch size at 3000 in alignment with McElfresh et al. (2024). For classification datasets with number of samples larger than 3000, we split the training set into batches and weigh the resulting predictions in proportion to the batch size (prediction from the last batch may have less weight than the others). For regression tasks, we parameterize model go (y!est (esest b) as Gaussian distribution (z b), an ra b)) similar to Hollmann et al. (2022), and directly produce the point estimation using the inverse variance estimator in inference time:
E 10,
test test trrain
xx
x
Eg Byly
test test b) p(b) (o(b)ttrain Dtrain x
(12)
Σ
pa (wo test bb)..
(13)
(m bn.)
o (mrees
test by)
15

---

Adversarially Pre-trained Transformer
The intuition is that, prediction on each batch is weighted by its uncertainty more weights put the predictions that the model is more certain of, and vice versa.
B.2. Patch-based Feature Embedding
We drew inspiration from Dosovitskiy al. (2020) and developed patch-based embedding approach that adapts datasets with arbitrary number of features. In Hollmann et al. (2022), embeddings of acquired by padding clipping the number of features de to certain maximum feature size d', such that can be fed to dense feedforward Instead, we split features into patches, setting d' as the patch size, and only pad the last patch to d' dimensions if d # O (mod d²). After feeding each patch to dense feedforward En, we pass them to an attention block with optional relative positional encoding (Su et al., 2021; Press et al., 2021), and average pool across the resulting embeddings of patches. Essentially, this is half-way approach between using a dense feedforward to embed all features, and using an attention block tokenize each individual feature. In this way, the embedding block can handle features more flexible manner while controlling computational complexity and memory usage.
IR + RR (moddee)
C. Hyperparameter Settings
All common hyperparameters of APT are directly inherited from TabPFN and not tuned, including learning rate 10-4, number of blocks 12, hidden dimensions 512, hidden feedforward dimensions 1024, number of heads 4, effective batch size (batch size per step number of gradient accumulation steps) 64, total number of training datasets (number of epochs x steps per epoch number of datasets per step) 400, 000, well as all data generator hyperparameters, For more details on the data generator hyperparameters, the code repository in our supplementary material.
D. More Results
We list the performance of top algorithms on small classification datasets in Table 5. Standard deviations are calculated across 5 different splits.
16

---

Adversarially Pre-trained Transformer
Table 5. The ROC-AUC of top algorithms on the 35 small datasets in OpenMI--CC18.
LightGBM
XGBoost
SVM
TabPFN
CatBoost
APT
mfeat-fourier
.981 + .004
.982 + .004
.982
.004
.985 + .002
.944 + .002
.983 .003
breast-w
.933 + .006
.993 .006
.995 .007
.997 +
.966 .005
.977 .003
mfeat-karhunen
.999 + .001
.99 + .001
+ 0
.999
.999 +
1 + 0
mical-morphological
959 + .004
.611 + .002
.65 = .006
.977 + .003
964 + .003
966 = .006
mfeat-zernike
970 .004
.73 + .004
.992 .003
.922 + .001
.94 + .003
.977 .003
.751 + .036
.758 + .036
.690 .020
.736 + .031
.758 + -037
.739 .026
credit-approval
.931 + .030
.900 + .022
.912 .024
.928 + .029
931 .03
.900 = .022
credit-g
809 = .018
.824 + .028
.866 .020
.835 = .018
816 .025
.866 = .024
diabetes
821 + .027
812 + .037
.811 .050
.817 .026
.27 + .025
.824 .016
tic-tac-toe
1 1
1
1 0
.933 + .003
1+0
.977 = .002
vehicle
936 .009
.455 + .008
.65 .011
.955 = .005
.41 + .008
.911 .008
eucalyptus
900 + .022
894 + .024
.844 + .009
908 .013
.055 .019
912 .017
aunlcatdata_authoose
1 1
1
1 0
1
1+0
1 + O
po4
.53 + .008
.944 + .012
.077 .058
.977 + .013
.61 .011
.64 .016
pc3
8844 .031
831 + .048
.706 .055
.848 + .044
.29 .042
.65 .032
kc2
887 + .060
.622 .102
.81 + .052
875
885 + 088
.896 + .087
bioxd-raasftusionservve---ee
740 + .085
.722 + .068
.755 .075
750
.72 1077
.751 .086
enac-9
.91 + .005
.994 + .005
.988 = .001
.812
.911 0005
.911 .014
ilpd
.767 + .067
.751 + .038
.628 = .085
-992
.787 + .059
.808 = .035
wdlbe
.93 + .006
.89 + .007
.988 + .003
+
993 + .003
.977 + .004
dresses-sales
.685 + .028
.618 .045
.669 .027
.552 .056
.637 .051
.617 .049
MiceProtein
1
1
1 0
1
1
1 0
steel-plates-fuut
975 + .003
.79 + .003
964 = .006
970 + .005
978 + .003
969 = .006
944 .043
.936 .052
.911 .070
.900 1 .053
.49 0044
.600 .058
balance-scale
970 .027
998 + .003
994 .006
.97 .004
949
.988 .003
mfeat-factors
.999 .001
.999 + .001
.999 = .001
.999
.999 = .001
vowel
999
.999 .001
.999 .001
1+0
1+0
analcatdata dmft
595 + .032
597 + .029
.601
.033
.577 + .044
582 + .027
.53 .040
pel
9011 + .065
917 + ,063
.82 + .127
.977 + .059
916 + .058
.922 = .041
bankmooe-aauhenticaaion
1
1
1#0
1+0
1+0
1
qsar-biodeg
934 + .015
.95 + .012
932 .017
+
935 .017
944 .013
semeion
.988 + .001
.999 + .001
.999 4
+
.99 101
.80 + .004
cylinder-bands
898 + .041
.833 + .036
.913 .03
911
904
.913 .031
car
0
1
1 1
.999 + 4001
1
.97 .005
mfeat-pixel
999 0
1+0
1 0
.99
1+0
.999 +
17


Reviews:
[{"review_id": "1QMTSY6DZ3", "reviewer": "Reviewer_Ezzc", "summary": "This paper introduces the Adversarially Pre-trained Transformer (APT), which is a novel zero-shot meta-learning method for tabular data prediction tasks. By employing adversarial synthetic data agents and a mixture block architecture, APT addresses key limitations in prior tabular learning methods, particularly in handling datasets with multiple classes and complex feature distributions. APT achieves state-of-the-art performance on small tabular classification tasks, generating more diverse synthetic data while maintaining computational efficiency comparable to existing methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their kind and detailed examination of our work.\n\nIn particular, the reviewer raises many profound questions about PFNs in general -- many of which our paper does not resolve on its own. Resolving the full gamut of these questions is a task for the entire research community working on PFNs, and most likely cannot all be achieved in a single paper. Our much more modest aspirations in this work are to develop certain aspects and capabilities of PFNS further than the state-of-the-art.\n\nTo address the specific questions and weaknesses the reviewer raised:\n\n ---\n\n**Question 1**: We have not seen evidence of overfitting in this work. We can pose the question of how we would know if our APT model is overfitting to synthetic data. One possible answer could be that the model would perform poorly at zero-shot predictions on *real* data. Given the competitiveness of our APT model with GBDTs and other models we benchmarked on, the empirical answer to this questions seems to be that PFNs thus far do not seem to be overfitting on synthetic data to a degree that it prevents them from excelling at prediction on real data.\n\n ---\n\n**Question 2**: We appreciate the question. Our belief is specifically that the adversarial data generating process makes the model more robust across domains when compared to traditional PFNs which are trained on narrowly defined data. The adversarial process is both novel and sophisticated. The training makes no use of any real data, and it is therefore not biased by any real data domain.\n\n---\n\n**Question 3**: We think this is a great question and it is very possible. We think the reason why prior research has not been focusing on the last-layer generalizability of deep models is because such mechanism is not in dire need in traditional ML problems, where the cardinality of label is fixed (e.g. fixed number of classes in tabular tasks, fixed vocabulary size in language tasks, etc.). In this case, we are free to let the model learn label-specific parameters. However, the mixture block could be very useful in other ML problems: for instance, it could help a vision model such as an ImageNet generalize to a set of different labels. These labels can have different meanings and higher cardinality than the original labels. This is thanks to the fact that the design of the mixture block ensures that it does not have parameters tied to any specific classes.\n\n---\n\n**Question 4**, and **Weakness 5**: The limitations are imposed on PFNs by the transformer architecture’s quadratic computation scaling. However, considerable research in recent years has significantly accelerated the transformer and increased its context length, in some cases up to 1M tokens. It is a worthwhile effort for future research to apply these accelerations to PFNs but out of scope of this paper. The model architecture we employ can be modified to include these advancements. \n\nWe intend to add the previous paragraph to the conclusion section in the final version of the paper, in light of the reviewer’s suggestion. This should provide a clear roadmap for future work to improve on APT’s computational capabilities.\n\n---\n\n**Question 5**: We don't believe that this is surprising: the original TabPFN paper shows that “tuned SVM” (Figure 12, pink triangular symbol) performed comparably with Catboost, and exceeded performance of decision forests. Because we are dealing with small datasets, kernel SVMs are tractable, which would not be the case for large datasets due to their computational intensity.\n\n---\n\n**Weakness 1**: We have benchmarked our methods and the competing methods on a variety of data tasks widely accepted in the tabular data research community. We have not seen evidence that suggests these datasets are not diverse, and the fact that an increasing number of papers are using the same benchmarks (TabPFN, TabZilla, etc.) suggests the community views these data tasks as a worthwhile set of challenges for tabular deep learning.\n\n---\n\n**Weakness 2**: We were able to directly incorporate our innovations to TabPFN without re-tuning its hyperparameters to synthetic data distribution and transformer model. We did not experience such sensitivity to hyperparameter issue, and one can argue that the adversarial approach of directed exploration of synthetic data space actually mitigates the sensitivity issue.\n\n---\n\n**Weakness 3**: This paper is intended to be empirical. For more details on the theoretical foundation of PFNs, please see [T. Nagler ICML'23], and we would like to respectfully remind the reviewer that this is not the focus of this work.\n\n---\n\n**Weakness 4**: The entirety of the code we used is attached in the supplemental material, from which readers can easily reproduce our results, and we followed best practices in allowing reproducibility.\n\n---\n\nThank the reviewer for their time and effort. We hope that in light of our responses, the reviewer will consider raising their score on our paper to an accept.", "claims_and_evidence": "The scatter-sum mixture block represents a conceptually groundbreaking architectural approach that transcends conventional multi-layer perceptron final layer designs, offering tantalizing possibilities for cross-domain generalizability. This design is particularly compelling due to its potential to reimagine output prediction mechanisms across diverse machine learning domains.\nThe meticulous dataset preprocessing approach demonstrates exceptional rigor through the deliberate exclusion of four vision datasets (MNIST 784, CIFAR-10, Devnagari-Script, and Fashion-MNIST) from the original collection, ensuring a laser-focused analysis of genuinely tabular data representations.", "methods_and_evaluation_criteria": "The Adversarial Data Agents methodology presents profound methodological challenges that demand exhaustive critical examination. Fundamental epistemological questions emerge that fundamentally challenge the approach's conceptual and practical foundations below.\n- How are benign samples conceptualized and potentially utilized within the synthetic data generation process?\n- Are original datasets meaningfully incorporated, or are they systematically marginalized?\n- What substantive computational and statistical principles underpin the concept of \"random initialization\"?\n- What precise algorithmic mechanisms constitute the categorical sampling strategy?\n\nMoreover, the approach raises critical questions about numerical feature generation:\n- What sophisticated computational strategies ensure the synthetic features maintain meaningful statistical properties and representational fidelity?\n- How did you prevent the generation of mathematically correct but contextually meaningless synthetic data points?\n- The methodology bears a fascinating conceptual resemblance to Bayesian nonparametric methods like Gaussian processes, yet simultaneously surfaces profound computational concerns. Does this approach necessitate maintaining the entire training set during testing, potentially incurring prohibitive memory and computational resource expenditures?", "theoretical_claims": "The research confronts inherent challenges in adversarial training, including potentially catastrophic methodological limitations of persistent risks of model collapse, intrinsic training instability, and potential representational degradation mechanisms.\n\nThe approach inherently risks replicating classic Generative Adversarial Network (GAN) pathologies, raising significant questions about the long-term stability and generalizability of the proposed synthetic data generation strategy.", "experimental_designs_or_analyses": "The experimental design reveals several noteworthy anomalies and potential methodological limitations. In detail, the performance of Support Vector Machines (SVM) appears statistically unexpected and demands rigorous investigative scrutiny.\nThe research conspicuously lacks a comprehensive exploration of large-scale tabular datasets and extensive classification scenarios.", "relation_to_broader_scientific_literature": "The research builds upon and extends Prior-Data Fitted Networks (PFNs), zero-shot meta-learning approaches, adversarial training techniques, and transformer architectures for tabular data.", "essential_references_not_discussed": "It seems that there are no essential references not discussed.", "other_strengths_and_weaknesses": "Strengths:\n- The research presents remarkable methodological innovation by creatively combining adversarial synthetic data generation with transformer-based learning architectures. The scatter-sum mixture block represents a groundbreaking approach that transcends conventional multi-layer perceptron design, offering promising cross-domain generalizability while addressing class size limitations in zero-shot meta-learning approaches.\n\nWeaknesses:\n- Significant generalizability limitations emerge from the current implementation. Critical questions persist regarding the method's applicability across different data types, particularly its performance on datasets with varied distributional characteristics and complex feature interactions.\n- The research is fundamentally constrained by its reliance on a specific computational approach, potentially restricting broader methodological applicability. The notable sensitivity to hyperparameters suggests potential challenges in achieving consistent performance without extensive manual tuning.\n- The approach's theoretical foundations remain incompletely developed, predominantly relying on empirical validation rather than rigorous mathematical substantiation. This empirical emphasis, while demonstrating practical effectiveness, leaves critical theoretical mechanisms unexplored and potentially undermines the method's deeper scientific understanding.\n- Reproducibility concerns are substantial. The complex methodological design, combined with the potential absence of comprehensive implementation details, may pose significant challenges for researchers attempting to replicate or build upon this work.\n- The computational complexity and memory scaling limitations inherited from transformer architectures represent a profound methodological constraint, potentially limiting the approach's utility for large-scale or resource-intensive applications.", "other_comments_or_suggestions": "Please refer to the other sections.", "questions_for_authors": "- How can the potential overfitting to synthetic data distributions be comprehensively mitigated?\n- What sophisticated mechanisms ensure the adversarial data generation approach's robustness across diverse data domains?\n- Can the mixture block's innovative architectural approach be effectively translated to other machine learning domains?\n- What strategies might comprehensively address the computational limitations for larger, more complex datasets?\n- How do the researchers explain the statistically unexpected SVM performance characteristics?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6suquChzID", "reviewer": "Reviewer_WkGE", "summary": "The paper introduces an Adversarially Pre-trained Transformer (APT) for zero-shot meta-learning on tabular prediction tasks. APT is pre-trained using adversarial synthetic data agents that continuously generate challenging datasets, enabling the model to generalize to unseen tasks without requiring real-world data for training. The authors also propose a mixture block architecture to handle classification tasks with arbitrary class sizes, addressing a key limitation of prior methods like TabPFN. Experiments show that APT achieves state-of-the-art performance on small tabular classification tasks, with improved generalization and faster pre-training compared to existing methods.\n\n## Update After Rebuttal\nThanks for the authors' response! I will keep the rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their succinct and extremely clear examination of our work.\n\nIn light of your feedback, we have added the following paragraphs:\n\n---\n\nWeakness 1:\n\n**Response:** *The limitations are imposed on PFNs by the transformer architecture’s quadratic computation scaling. However, considerable research in recent years has significantly accelerated the transformer and increased its context length, in some cases up to 1 million tokens [1][2]. It is a worthwhile effort for future research to apply these accelerations to PFNs beyond the scope of this paper. The architecture we employ can be modified to include these advancements.*\n\nWe intend to add this paragraph to the conclusion section in the final version of the paper, together with specific references to the notable advancements in transformer capabilities. This should provide a clear roadmap for future work to improve on APT performance.\n\n*Reference*:\n\n[1] Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. ICLR 2022\n\n[2] Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Scaling transformer to 1m tokens and beyond with RMT. CoRR, 2023.\n\n---\n\nWeakness 2:\n\n**Response:** *Note that rather than re-designing the base synthetic data sampling distributions or re-performing extensive hyperparameter search, we use the exact same synthetic data sampling distributions and hyperparameters that were used in TabPFN for the purpose of ablation, in order to clearly demonstrate the contribution of adversarial training. TabPFN was trained only on classification problems, and therefore it is possible that these hyperparameters are over optimized for classification tasks, and under optimized for regression tasks.*\n\nWe will add this paragraph to end of the experiments section in the final version of the paper, to point out and attempt to explain the relative performance in regression.\n\n---\n\nWe hope that in light of including these additional brief discussions in the paper, the reviewer would consider raising their score of our paper to a strong acceptance. We appreciate the reviewer for their time and effort.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There's no proof in the paper", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Adding synthetic adversarial training data could help the generalization of tabular model training, which can provide beneficial insights.", "essential_references_not_discussed": "I am not in the field of tabular data paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The use of adversarial synthetic data agents enhances the diversity and difficulty of the training data, leading to better generalization and robustness in zero-shot learning scenarios.\n\n2. The proposed mixture block eliminates the class size limitation of previous methods, making the model more flexible and applicable to a wider range of classification tasks.\n\n3. Extensive experiments demonstrate that APT achieves competitive or superior performance on small tabular classification tasks, demonstrating its efficiency and effectiveness.\n\nWeakness:\n1. The proposed approach performs well on small tabular datasets, but it might be difficult to be applied for large-scale dataset because of the quadratic runtime and memory complexity. A more detailed discussion on potential optimizations for scalability would strengthen the paper.\n2. Regarding the improvements in regression tasks, while better than TabPFN, APT still falls short of tree-based models in regression. More discussion about this could be helpful.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H0hCX249ka", "reviewer": "Reviewer_pxwT", "summary": "This work introduces APT, a zero-shot meta-learning model for tabular prediction, pre-trained with adversarial synthetic data agents. It improves TabPFN, removes class size limitations via a mixture block architecture, and matches SOTA GBDTs on small tabular tasks. While enhancing performance in classification and regression, APT retains quadratic scaling and struggles on large datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their clear and valuable feedback on our work, and we address your questions and comments as follows:\n\n---\n\n“Other Comments Or Suggestions” & Question 1:\n\n**Response**: Thank the reviewer for the suggestion. We added citation to TabPFN v2 [Nature'25] per your comment, but we want to respectfully explain to the reviewer why comparison to TabPFN [ICLR'23] is more appropriate than comparison to TabPFN v2 under the scope of this work:\n\n- Per ICML guidelines (https://icml.cc/Conferences/2025/ReviewerInstructions), the main conference submission has the same four-month concurrent cutoff adopted from ICLR, and reviewers should not expect authors to discuss these works that have been recently made public. TabPFN v2 was made public within one month of the submission deadline.\n\n- We do not see the Nature version of TabPFN to be a better comparison to APT than the ICLR version of TabPFN. The primary reason is reproducibility. We take ablation *very* seriously in this work, and TabPFN v2 does not release their pre-training code, while the primary contribution of this work is the improvement to an existing pre-training scheme. As stated in our work, all setups and hyperparameters of our synthetic data distributions (as well as the transformer model) are set to be exactly the same as TabPFN, in order to fairly examine the impact of adversarial data agents to pre-training.\n\n---\n\nQuestion 2:\n\n**Response**: Yes, the proposed method already uses the SCM-based data generating mechanism. For the purpose of ablation, we did not make any further changes to TabPFN’s underlying generating mechanism as well as hyperparameter settings, other than necessary changes that enable stable adversarial training (L304-305 left column; Appendix B; Appendix C).\n\nNote that in this work, we described the SCM-MLP simply as “sparsified noisy MLPs” to be more explicit and straightforward, since all structural equations in the so-called SCM are modeled unifiedly as linear equations plus simple activation, and the exogenous noise in the structural equations are unifiedly modeled as gaussian with the same mean and pre-sampled variance. But in light of the reviewer’s comment, we understand that it’s important to draw connections to concepts and terminologies used in prior work, so we have added this information to L137-142 left column which now reads: *“predictors $\\mathbf{x}^{(k)}_i$ and response $y^{(k)}_i$ are values of randomly selected neurons in sparsified noisy MLPs (i.e. the SCM data generating mechanism) with some additional pre-processing. More details regarding this approach can be found in Appendix B.1.”*.\n\n---\n\nWe hope that in light of our responses and further clarifications, the reviewer would consider raising their score of our paper to a strong acceptance. Thank the reviewer for their time and effort.", "claims_and_evidence": "Yes, the claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, it follows the previous works of tabpfn and tabular benchmark.", "theoretical_claims": "This paper does not have theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The ability of tabpfn is highly dependant on the quality of the synthetic data, this paper is trying to improve the quality of the synthetic data by leveraging the adversarial learning. It also gives a solution to remove the class size limitation of the tabpfn.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The idea of adversarially generating synthetic data is interesting.", "other_comments_or_suggestions": "More related works of tabpfn [ICLR'23, Nature'25] should be discussed, both of them are the stem of the tabpfn. Furthermore, the tabpfn [ICLR'23] has been extended to the journal version [Nature'25] and release the tabpfn v2, more discussion about the extension and the comparison with the tabpfn v2 is needed.", "questions_for_authors": "1. Recently, the extended version of tabpfn [Nature'25] is published and release the tabpfn v2, the experiments of this paper can be compared with the tabpfn v2.\n2. The tabpfn [ICLR'23] use the structure causal model to generate the synthetic data, can the proposed method be applied to the structure causal model? (ref: TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y4uPd2K6qH", "reviewer": "Reviewer_TeqX", "summary": "This paper introduces APT, extending on PFNs and TabPFN, which leverages adversarial synthetic data agents for pretraining and incorporates a mixture block architecture to handle classification tasks with an arbitrary number of classes, addressing the class size limitation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewers for their valuable feedback and comments. We see that the reviewer’s main concern lies in the confusion about zero-shot meta-learning, so first and foremost, we want to address this concern and try our best to provide the reviewer a clear picture.\n\n---\n\nWeakness 1 & 2 (Question 1):\n\n**Response**: As the reviewer has noticed, APT only improves on TabPFN’s pre-training procedure and inference-time generalizability, and that’s the goal of this research. While we believe these innovations are interesting and significant, they do not change the fundamental Bayesian mechanism of TabPFN as well as how and why it can perform meta-learning. TabPFN is a zero-shot meta-learner. We will reiterate the precise definition in the next paragraph, but for a detailed explanation on how and why PFN-based models can perform meta-learning, how and why PFN-based models can adapt to zero-shot scenarios, and why zero-shot tabular prediction is important, please kindly see [1] and [2]. Per the reviewer’s feedback, we have added a sentence at L75-76 left column in the introduction section that reads *“PFN-based methods, such as TabPFN, are zero-shot meta-learners. For more details on how PFN’s underlying Bayesian mechanism can perform meta-learning with zero optimization on an unseen dataset, see [1].”* to clarify that PFN-based models' capabilitiy to perform zero-shot meta-learning is *not* where our unqiue contribution lies.\n\nSome recent TabPFN-based works tend to call this class of method “tabular foundation models” [3][4], while the term tabular “zero-shot meta-learner” is slightly more specific. We have provided the definition of zero-shot meta-learner in the introduction section, and we will further highlight it in the proposed method section per reviewer’s suggestion:\n\n- A model that performs no optimization (i.e. zero gradient updates for a deep learning model) on its parameters while predicting on unseen datasets.\n\nThis is mentioned in [5] and it is a simple extension on the definition of meta-learner:\n\n- A model that performs few optimizations (i.e. few gradient updates for a deep learning model) on its parameters while predicting on unseen datasets.\n\nWe chose to not categorize PFN-based methods as foundation models because the reason why PFN-based models work is very different from the reason why large vision and language models work – they do not pretrain on any real-world data and their goal is to learn to adaptively acquire data representation of unseen data during inference time. Nonetheless, both concepts have very general and straightforward definitions, and are both correct categorizations of PFNs. We hope that this choice of categorization would not be a major deciding factor, as how PFN-based models perform meta-learning under the zero-optimization scenario is well documented in [1].\n\n---\n\nWeakness 3 (Question 2):\n\n**Response**: Yes, this baseline selection is appropriate. It is no different from the baseline selection of of other prior works that benchmark PFNs such as [1], TabPFN, and TabZilla [6], as Reviewer pxwT also pointed out. We want to respectfully note that the reviewer might have a slight misunderstanding regarding the evaluation setups of PFN-based works? (If this is not the case, please kindly ignore this comment) The traditional methods such as CatBoost in the evaluations are not trained under zero-shot scenarios, they are trained on the full training sets and given much more computational time for tuning – this actually gives zero-shot methods more disadvantages and further showcases their zero-shot capabilities. The research on tabular zero-shot meta-learning is still very young, and there are no alternatives to select as baselines other than PFNs, to the best of our knowledge.\n\n---\n\nWeakness 4 (Question 3):\n\n**Response**: The advantage of APT compared to TabPFN is its inference-time generalizability (it can predict on datasets with unseen classes of unseen cardinality) and pre-training data diversity; the advantage of APT compared to CatBoost is its exceptional runtime thanks to the zero-shot capability. We want to note that the sheer performance on small tabular classification tasks is nearly saturated [6], and none of the recent zero-shot meta-learners or other deep methods is able to significantly beat GBDTs such as CatBoost on small tabular prediction tasks with no limitation on feature size, class size, or number of missing values [1][6]. We believe the data diversity improvement in pre-training as well as the proposal of the mixture block that fundamentally addressed TabPFN’s class size limitation are important contributions to PFNs aside from the improved performances, and we believe we have showcased their power through extensive ablations.\n\n---\n\n*References*:\n\n[1] [S Müller, et al. ICLR 2022]\n\n[2] [T Nagler. ICML 2023]\n\n[3] [N Hollmann, et al. Nature 2025]\n\n[4] [SB Hoo, et al., Arxiv 2501.02945]\n\n[5] [VK Verma, et al. AAAI 2020]\n\n[6] [D McElfresh, et al. NeurIPS 2024]", "claims_and_evidence": "No, the authors claim that they employ meta-learning methods for tabular prediction in the zero-shot scenarios. However, this claim is not adequately reflected in the method section, leaving the meta-learning process and its adaptation to zero-shot scenarios unclear.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The paper does not introduce any significant theoretical contributions on meta learning in the zero-shot scenarios.", "experimental_designs_or_analyses": "Yes, the authors conduct the experiments that adversarial pre-training enhances the performance of TabPFN, showing that adversarial synthetic data agents generate more diverse datasets compared to standard random generators in TabPFN. However, this paper does not clearly explain how APT effectively performs meta-learning or adapts to zero-shot scenarios.  \nAdditionally, while mixture block architecture design has been shown to improve generalization and significantly accelerate pretraining, Table 1 indicates that most selected baselines are not designed for the zero-shot scenarios. This raises concerns about the effectiveness of APT in truly zero-shot settings.", "relation_to_broader_scientific_literature": "This work extends on TabPFN by enhancing the diversity of its pretraining dataset through adversarial generation and enabling arbitrary classification through a mixture block model architecture. TabPFN, leveraging context learning and PFN, has demonstrated strong performance in small-scale tabular prediction.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. Data Agent Reset helps improve the diversity of generated data.\n\nWeaknesses:\n1. APT extended on PFN and TabPFN, which is proposed to perform zero-shot meta-learning on tabular prediction tasks without using any real-world dataset to pre-train the model, but the authors do not clearly explain its meta-learning mechanism and how it enables zero-shot adaptation in the task of tabular prediction.\n2. The research motivation of this paper is not clear, authors do not clarify the significance of zero-shot table prediction or its practical implications. Additionally, authors do not also provide a clear task definition for zero-shot meta-learning in tabular prediction. \n3. The experimental baselines primarily consist of some methods with requiring labeled samples for effective training. This raises concerns about whether directly comparing these methods with APT in a zero-shot scenario meaningfully reflects APT’s zero-shot capabilities.\n4. As shown in the Table1, APT does not exhibit a significant improvement in ROC-AUC when compared to CatBoost and TabPFN.", "other_comments_or_suggestions": "We suggest that the author should provide a detailed definition of zero-shot meta-learning for tabular prediction and clarify the significance of zero-shot scenarios in tabular prediction tasks, specifically explaining why zero-shot table prediction is necessary.", "questions_for_authors": "1. How does APT perform meta-learning, and how does it enable zero-shot adaptation?  \n2. Are the selected baseline methods appropriate for evaluating APT’s zero-shot capabilities, given that they require labeled samples for training?  \n3. What are the advantages of APT compared to CatBoost and TabPFN?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work builds upon prior-fitted networks, in particular TaPFN, by introducing two new components: A mixture layer that allows scaling to arbitrary numbers of classes in classification tasks, and adversarial data generation for pretraining.
Both of these directly advance the state of zero-shot learners for tabular classification, an important emerging research area.
This is a clear advancement of the original TabPFN architecture, though as the reviewers remark, the system still inherit some of the limitations of TabPFN, in particular with respect to the number of features natively handled, and with respect to the dataset size.
While some of the shortcomings of TabPFN have been addressed in TabPFN v2 (Nature 2025), this work should be considered concurrent with TabPFN v2, and improvement made here are orthogonal to those made in TabPFN v2, and are likely to transfer to new architectures.
Decision: Accept (poster)