ICML 2025

Zero-shot Meta-learning for Tabular Prediction Tasks
with Adversarially Pre-trained Transformer
Yulun Wu 1 Doron L. Bergman 2
Abstract
We present an Adversarially Pre-trained Trans-
former (APT) that is able to perform zero-shot
meta-learning on tabular prediction tasks without
pre-training on any real-world dataset, extend-
ing on the recent development of Prior-Data Fit-
ted Networks (PFNs) and TabPFN. Specifically,
APT is pre-trained with adversarial synthetic data
agents, who continue to shift their underlying data
generating distribution and deliberately challenge
the model with different synthetic datasets. In
addition, we propose a mixture block architecture
that is able to handle classification tasks with ar-
bitrary number of classes, addressing the class
size limitation – a crucial weakness of prior deep
tabular zero-shot learners. In experiments, we
show that our framework matches state-of-the-art
performance on small classification tasks without
filtering on dataset characteristics such as number
of classes and number of missing values, while
maintaining an average runtime under one second.
On common benchmark dataset suites in both clas-
sification and regression, we show that adversarial
pre-training was able to enhance TabPFN’s perfor-
mance. In our analysis, we demonstrate that the
adversarial synthetic data agents were able to gen-
erate a more diverse collection of data compared
to the ordinary random generator in TabPFN. In
addition, we demonstrate that our mixture block
neural design has improved generalizability and
greatly accelerated pre-training.
1. Introduction
In standard deep learning workflows, models are either
trained per dataset, or employed on data in a form com-
1University of California, Berkeley 2Capital One. Correspon-
dence to: Yulun Wu <yulun wu@berkeley.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
patible with, and drawn from, th...