Generative Table Pre-training Empowers Models for Tabular Prediction
Authors: Tianping Zhang
Shaowen Wang
Shuicheng YAN
Li Jian
Qian Liu
Venue: EMNLP 2023 Main (2023)
Keywords: tabular prediction
generative table pre-training
TL;DR: Empower machine learning models for tabular prediction by synthesizing high-quality data with generative table pre-training

Abstract: Recently, the topic of table pre-training has attracted considerable research interest. However, how to employ table pre-training to boost the performance of tabular prediction remains an open challenge. In this paper, we propose TapTap, the first attempt that leverages table pre-training to empower models for tabular prediction. After pre-training on a large corpus of real-world tabular data, TapTap can generate high-quality synthetic tables to support various applications on tabular data, including privacy protection, low resource regime, missing value imputation, and imbalanced classification. Extensive experiments on $12$ datasets demonstrate that TapTap outperforms a total of $16$ baselines in different scenarios. Meanwhile, it can be easily combined with various backbone models, including LightGBM, Multilayer Perceptron (MLP) and Transformer. Moreover, with the aid of table pre-training, models trained using synthetic data generated by TapTap can even compete with models using the original dataset on half of the experimental datasets, marking a milestone in the development of synthetic tabular data generation. The code and datasets are available at https://github.com/ZhangTP1996/TapTap.

Generative Table Pre-training Empowers Models for Tabular Prediction
Tianping Zhang', Shaowen Wang', Shuicheng Yan', Jian Li Qian Liu
"Tsinghua University, FFudan University, Skywork AI, 'Sea Lab
(ztphs980, shuicheng, yan)ggganl com, wangsw www.uddanndu...m
iiqiannsea.comm com, 1ijian83@mail. tsinghua. edw..ccceddcccmm.
Abstract
Table
Question Answering
Education
Occupstion
Income
Quection How people with a Eachelor's degios com luss than SOK?
Recently, the topic of table pre-training has attracted considerable research interest. How- ever, how to employ table pre-training to boost the performance of tabular prediction remains open challenge. In this paper, we propose TAPTAP, the first attempt that leverages table pre-training to empower models for tabular pre- diction. After pre-training on large corpus real-world tabular data, TAPT^P can gen- erate high-quality synthetic tables to support various applications on tabular data, including privacy protection, low resource regime, miss- ing value imputation, and imbalanced classifi- cation. Extensive experiments on 12 datasets demonstrate that TAPTAP outperforms total of 16 baselines in different scenarios. Mean- while, it can be easily combined with vari- backhone models, including LightGBM, Multilayer Perceptron (MLP) and Transformer. Moreover, with the aid of table pre-training, models trained using synthetic data generated by TAPTAP can even compete with models using the original dataset on half of the experi- mental datasets, marking milestone in the de- velopment of synthetic tabular data generation. The code and datasets are available at https: //gi thub. wtt.u..ooo/....eoooemo.......ee. Zhang TP1 9ap.
16
HE grad
Machine- op-inspct
50K
Some- Colliege
Table Fact Verification
26
Craft ropai
[Itermedions TAPIK
Hypocheaks
h2s
29
Bachelans
Encc- managerial
>> 50K
Tabular Prediction
(Cura)
Handlers- cleaners
Fsample:
college
Agc
Education
Ocoupption
2-5KK
Figure 1: An illustration of different sable-related tasks with representative table pre-training models, including TAPAS (Herzig et al., 2020), TaBERT (Yin et al., 2020), Intermediate (Eisenschlos et al., 2020), TAPEX (Liu al., 2022) and our TAPTAP.
of pre-trained LMs has inspired much research in pre-training over structured tables, one of the most common types of data used in real-world applica- tions (Benjelloun et al., 2020). Different from text, tables usually contain rich and meaningful struc- tural information, and thus LMs on text corpus are not well suited for tabular data. To this end, there has been a growing amount of recent work on table pre-training (Herzig et al., 2020; Yin et al., 2020; Wang et al., 2021b; Liu et al., 2022).
However, the vast majority of cxisting table pre-training works aim to enhance joint reasoning text and table (e.g., table question answering, tableQA), while neglecting tabular prediction, important task in real-world applications. The goal of tabular prediction is to predict a specified tar- get (c.g., the income) based on set of features (e.g.. the age and the occupation). As illustrated in Figure 1, most pre-trained LMs on tables such as TAPAS (Herzig et al., 2020) typically apply MLM variants on crawled tables and text segments boost their joint reasoning capability in tableQA.
1 Introduction
Recently, pre-trained language models (LMs) have attracted lot of research interest in different do- mains, especially in the area of natural language processing. After pre-training on a large-scalc unstructured text corpus with a self-supervised training objective, e.g., masked language modeling (MLM) proposed by BERT (Devlin et al., 2019), LMs can significantly benefit downstream tasks. Furthermore, recent progress generative LMs (Radford al., 2019; Raffel et al., 2020: Lewis al., 2020) suggests that it is possible unify dif- ferent tasks via one LM. The remarkable success
Nevertheless, of yet, there is little evidence that these table pre-training methods can enhance the performance of tabular prediction tasks. This is probably because tabular prediction tasks are
*Corresponding Authors.

---

quite challenging, In contrast the exceptional performance of deep learning in many domains, recent studies (Shwartz-Ziv and Armon, 2022: Gor- ishniy et al., 2021) question the necessity of deep learning models for tabular prediction, as their per- formance is usually outperformed by traditional machine learning models. To summarize, it is still open challenge employ table pre-training to boost models for the tabular prediction task.
sive benchmark covering four practical set- tings in tabular prediction. Experimental re- sults on the benchmark demonstrate that TAP- TAP can be easily combined with different SOTA backbone models and outperforms a total of 16 baselines across 12 datascts.
2 Related Work
Table Pre-training Previous works on table pre- training can be categorized by the applicable downstream tasks and can be divided into four lines (Dong et al., 2022): table question answering which outputs the answer for questions over tables (Yin et al., 2020; Herzig et al., 2020; Yu et al., 2021; Liu et al., 2022: Andrejezuk et al.. 2022), table fact verification which verifies whether hypothesis holds based on the given table (Eisenschlos et al., 2020), table to text which generates textual descrip- tions from the given table (Gong et al., 2020; Xing and Wan, 2021) and table structure understanding which aims at identifying structural types in the given table (Tang et al., 2021: Wang et al., 2021b: Deng al., 2021). Our work is different from theirs because we focus on the application of table pre- training on tabular prediction. There were some previous studies that performed tabular pre-training for tabular prediction (Wang and Sun, 2022; Arik and Pfister, 2021; Yoon et al., 2020; Bahri et al., 2022). The major difference between TAPTAP and the previous studies is that, TAPTAP performs cross-table pre-training on language models using large number of tables to leverage the knowledge embedded in language models, and previous works usually perform single-table pre-training (Arik and Pfister, 2021) (or few tables with lots of overlapped columns (Wang and Sun, 2022)) on models specifi- cally designed for tabular data.
In this paper, we present TAPTAP (Table Pre- training for Tabular Prediction), which is the first attempt that leverages pre-training of language models on tables to significantly benefit tabular pre- diction tasks. To benefit different backbone models, we apply table pre-traieet pre-training from a data perspec- tive, i.e., we utilize TAPTAP to synthesize high- quality examples that can be used to train back- bone models. Based on the widely used generative language model GPT (Radford et al., 2019), after ongoing pre-training on a large-scale corpus of real- world tabular data, TAPTAP is expected to capture generic tabular data distribution. Then, TAPTAP can be quickly adapted to downstream tables via fine-tuning and can generate high-quality synthetic tables to support various applications on tabular data, including privacy protection, low resource regime, missing value imputation, and imbalanced classification. Mcanwhile, such a design decouples the backbone model from the pre-trained model architecture, allowing TAPTAP to benefit different backbone models. Extensive experiments on 12 public datasets demonstrate that generative table pre-training can empower models on tabular pre- diction in various ways, and TAPTAP outperforms a total of 16 baselines in different scenarios and supports three state-of-the-art (SOTA) backbone models. The contributions of this paper can be summarized as follows:
Table Generation TAPTAP supports backbone models by generating synthetic tables, and thus is close the line of table generation. Exist- ing methods for the generation of synthetic tab- ular data mostly leverage generative adversarial networks (Choi et al., 2017: Park et al., 2018; Mot- tini et al., 2018; Xu et al., 2019; Koivu et al., 2020) or variational autoencoders (Xu et al., 2019; Ma et al., 2020; Darabi and Elor, 2021). However, it is hard for these methods to leverage the textual mantics in tables. More recently, GReaT (Borisov et al., 2022) has successfully applied LMs in gen- crating synthetic tabular data. There are some sig- nificant differences between GReaT and TA TAPTAP:
To our knowledge, we are the first to success- fully apply table pre-training of language mod- els to tabular prediction. With carefully de- signed generation strategies, method com- bines the advantages of backbone models for tabular prediction and pre-trained LMs.
To accomplish the pre-training, we collect and filter out 450 public tabular datasets from Kag- gle, UCI, and OpenML platforms, and finally construct a large-scale pre-training corpus.
To systematically evaluate the proposed table pre-training method, we build a comprehen-

---

Pre treiniiinng Corpus
Data Prompts
Privacy Protection
Education Occupation i4
House Age
Avg, Rooms
House Val
Low Resource Regime Missing Value Imputation Imbalaneed Classifiation
Age is
Education Occupation is
41
6.98
4.53
Age 28, Education is Some college. Occupation
21
6.24
3.59
Incomee 50K, Age
Education Occupation
Textual Encodingg
TAPTAP Model
Age
Education
Occupation
Backbone Model
Fine-tuning Table
Age 18, <ts>
HS grag
Exec- manceeanao
Nti
GBDT
Some-colage Exec man-graneeiiial
Education
Octpalin
Auto Augo-regressive Langunge Model
MLP
18
H5 grad
Machine- 30 inspct
50K
cole-colloge
Craft repair
28 Some collage
5OK
Transformer
Ago
Data Sampling
Some-colinge
Itndiers-cleaners
6OK
Labeling
The illustration of our method. The TAPTAP model is firstly pre-trained on the pre-training corpus, and then fine-tuned on the downstream table. During both pre-training and tibe-tuning. tables are serialized into sequences via textual encoding, and TAPTAP is trained to predict them token by token. During inference, TAPTAP is prompted to sample values for " in data prompts, and the filled values build up a synthetic table. Finally, once the backbone model has yielded labels for the synthetic table, it can be used to strengthen the backbone model.
Figure 2:
1) We successfully apply table pre-training, which brings significant improvements over GReaT. 2) We design delicate pre-training and data synthe- sis procedure, including number encoding and data labeling, which greatly benefit TAPTAP. 3) GReaT only exploits existing LMs for the privacy protec- tion setting, along with basic backbone models like linear regression and decision trees. However, our proposed table pre-training can notably enhance the performance of three SOTA backbone models for tabular prediction across range of scenarios.
and LightGBM (Ke et al., 2017) still outperformed those deep-learning counterparts. Therefore, TAP- TAP aims at synthesizing high-quality examples, which is able to empower both machine learning and deep learning models.
3
Methodology
3.1 Preliminary of Tabular Prediction
A tabular data usually contains two parts, the fea- tures and the label. Given the features as the input, the goal of tabular prediction is to predict the la- bel. Taking the example from Figure 1, the task is to predict the income (label) of person based on her / his age, education and occupation (features). Below we formalize tabular prediction using the binary-classifice
Tabular Prediction Due to the tremendous suc- cess of deep learning and transfer learning in var- ious domains, there has been lot of research interest extend this success tabular predic- tion (Song et al., 2019; Wang et al., 2021a; a; Arik and Pfister, 2021). As for deep learning, we refer readers to Gorishniy et al. (2021) for a comprehen- sive comparison of different deep learning models for tabular data. Our work is technically orthogonal to these efforts, as it can be integrated with differ- ent backbone models (e.g., Transformer). As for transfer learning, there has been some pioneering attempts (Levin al., 2022; Wang and Sun, 2022). More recently, researchers even explore the ability of LMs on zero/ few-shot classification of tabular data (Hegselmann et al., 2022). However, there is often some gap between their experimental setup and real-world applications. For example, Levin et al. (2022) only investigates transfer learning on tables with lots of overlapping columns. Despite all these efforts in advancing deep learning on tabular data, recent studies (Shwartz-Ziv and Armon, 2022; Gorishniy et al., 2021) found that machine learning models like XGBoost (Chen and Guestrin, 2016)
task, and the formulation can be easily extended to multi-class classification or re- gression problems. Formally, a tabular data with n samples (i.c., rows) and m features (i.e., columns) can be represented by D D=[(x;, 1/3,,,,, y1) hi=t..... where X; (Ii,), "Tijj. Zi,m) E I and y E {0, 1). The j-th feature has feature name fj (e.g., "age"). A model F takes the features x; input predict the label yı- Our goal is train a model such that the test error is as small as possible.
Existing works on improving F either design better model architectures (Gorishniy et al., 2021) or improve the quality of training data (Zhang et al., 2022). We follow the second path to improve the model performance by generating synthetic data. There are four typical scenarios where high-quality synthetic samples are helpful: (1) Privacy pro- tection (Gascón et al., 2017). In many application domains, each party only has part of the dataset and several parties can collaboratively train a model on

---

joint dataset. But tabular data usually contains sensitive personal information or confidential busi- ness secrets that cannot be directly shared with other parties. In this case, TAPTAP can be used to generate synthetic data D3 to replace the real data D, while achieving similar model performance. (2) Low resouree regime. Data collection can be very expensive in some applications and hence handling the small data regime is an important challenge. For example, over 44% classification datasets on the UCI platform (Asuncion and Newman, 2007) have less than 1000 samples. this case, we can leverage TAPTAP to perform data augmentation in order to boost the backbone model. (3) Missing value imputation. Missing values are ubiquitous in tabular data (Stekhoven and Bühlmann, 2012). In this case, TAPTAP is able impute the missing values to improve the performance of the model. (4) Imbalanced classification. It is common to have a long-tail label distribution tabular data (Cao et al., 2019). In this case, TAPTAP can used to balance the class distribution by conditional sam- pling (from the minority classes).
sex
Mole
RAsk
score
word freq
TIRE
year Nid
Code
day
dete
ii
Incone
type
group
Yes
tota.
number
Iegoi CONOTTION dOUNT
Toeste
Gender Sale
Figure 3: The word cloud for the pre-training corpus.
datasets used in pre-training and the downstream benchmark datasets are non-overlapping, so there is no data leakage issue.
3.3.1 Textual Encoding
Table Serialization Since TAPTAP starts with the GPT model, work (Borisov et al., 2022: Liu et al., 2022) serialize each sample into sequence of tokens to reduce the difficulty table pre-training. As suggested by Hegselmann et al. (2022), we take the text template serialization strategy and serialize samples using the "[Feature] is [Value]" tem- plate. Taking the example in Figure 2, the first sample in the fine-tuning table is converted into sentence "Age 18, Education is IIS-grad, Occu- pation suachine-opinsspe Income S 50K". Formally, given a table be the j-th feature value in x, and fy be the j-th feature name. The textual encoding is to transform the i-th sample x; into splice of sentences sepa- rated by commas t; = (fz,1, "," 4,,, an tim), where tij (f,, "is", zz.)).
follow the previous
3.2 Overview
As shown in Figure 2, TAPTAP consists of four steps. (1) Pre-training: train an auto-regressive LM on the table pre-training corpus compiled by lots of public tabular datasets. (2) Fine-tuning: train the LM on the downstream table; (3) Data Sampling: prompt the fine-tuned LM sample synthetic tables with only tabular features. (4) Data Labeling: assign pseudo labels to the synthetic tables via downstream backbone models. Below we describe these steps in details.
[(x,,))), let zis
Number Encoding Numerical features (e.g.- age) are important and widely used in tabular data over 70% of features in our pre-training corpus are numerical features, but how to properly encode these features has always been neglected in previ- ous work of tabular prediction. Meanwhile, recent studies LMs show that they not good at dealing with numbers (Pi et al., 2022) and suggest the character-level representation is better suited to capture the number semantics than its counter- parts (Wallace et al., 2019). Therefore, we use the character-level representation for all numerical fea- tures, which means that the phrase "Age is 18" in Figure 2 would be converted into "Age is 1 8".
3.3 Pre-training
Corpus Construction To build the pre-training corpus, we leverage publicly available tabular datasets from Kaggle', UCI (Asuncion and New- man, 2007), and OpenML (Vanschoren al., 2013) platforms. We believe the table pre-training should be performed on tabular data with rich semantic information, therefore we eliminate datasets with meaningless column names (e.g., V1). After the fil- tering, we finally collect 450 tabular datasets with a total of nearly 2 million samples. To illustrate it better, we show in Figure 3 a word cloud com- posed of feature names and feature values. Note that careful to guarantee that the tabular
Permutation Function The features in the tab- ular data are not ordered, but they are encoded an ordered sentence, which introduces spurious positional relationships in textual encoding. In or- der to reconstruct the order independence among
1https: h/www kaggle.com/

---

Table 1: Properties of benchmark datasets.
Classification
Regression
Dataset
1.0
AD
HE
CR
S1
BE
DI
CA
GE
ME
AG
DU
samples (k)
49
9.9
150
14
102
3.9
1.9
# numerical features
6
16
8
7
categorical features classes
8
22
0
39
0
3
2
7
features, we follow previous work (Borisov et al., 2022) to apply a permutation function P to ran- domly shuffle the order of features when encod- ing a table. Therefore, the encoded sentence be- comes t; (t k2. km where k2 ,, km]] 2(1,,2.... m... Such per- mutation enables conditional sampling when do- ing inference on downstream tables (Borisov et al., 2022), i.e., TAPTAP can generate a synthetic sam- ple conditioned on any set known features. We take step further to demonstrate that the condi- tional sampling helps TAPTAP perform well in the missing value imputation scenario.
3.5 Data Sampling
Given the sequence (w01, (ox-1 as the prompt, TAPTAP is able to output the categorical distribu- tion of the next token w/. E V after fine-tuning, where V denotes the vocabulary. In general, wa is sampled from the conditioned probability distribu- tion p wwwg/wi..- (k-1).
Since we also employ permutation during fine- tuning, the fine-tuned TAPTAP is able to gener- ate synthetic samples given any prompt. We em- ploy three kinds of prompting strategies for dif- ferent application scenarios (Borisov et al., 2022). (1) Feature name as prompt. This strategy is used in the privacy protection and low resource regime, where only feature names in the tabular data are selected as the prompt. The synthetic sam- ples are generated by TAPTAP according to the prompt "[Feature] is ". (2) One feature-value pair as prompt. This strategy is used in the im- balanced classification scenario, where the feature names and the minority label(s) are both provided as the prompt. With the label treated as feature, TAPTAP generates synthetic samples based on the prompt "[Feature] is [Value], ". (3) Multiple feature-value pairs as prompt. This strategy is used in the missing feature scenarios, where the feature names and available feature values are pro- vided as the prompt. TAPTAP generates synthetic samples according to the prompt "[Feature1] is " The
3.3.2 Pre-training Procedure
As mentioned before, the pre-training follows an auto-regressive manner, i.e., TAPTAP is trained to predict the encoded sentence token by token. As- suming we have q tabular datasets for pre-training, the whole pre-training corpus T can be obtained by combining each tabular data after textual en- coding as (t." U .... U (.))). Then, each sen- tence 'T can be encoded into sequence of tokens using (wi,.... WN) = tokenize(t). In general, TAPTAP factorizes the probability of generating in an auto-regressive manner as wk-1).
p(t) = II p(wa) (i, ,...),) During pre- training. TAPTAP optimized towards maximiz- ing the probability I(z)((t;) p(t,) the entire pre- training corpus. The pre-training can start with any auto-regressive LM such as GPT (Radford al., 2019), so that TAPTAP can benefit from the com- mon knowledge already learned by these LMs.
[Valuel], [Feature2] is [Value2], order of the given features in the prompt is random. Data prompt examples can be found in Figure 2.
3.6 Data Labeling
An accurate label is arguably one of the most cru- cial ingredients in synthetic samples. Noisy labels can severely degrade the generalization capability of backbone models (Gorishniy al., 2021). In contrast the previous work relying on LMs generate labels (Borisov et al., 2022), we propose to assign pseudo labels using the SOTA backbone models. We argue that LMs are not yet the best
3.4 Fine-tuning
Fine-tuning TAPTAP on the downstream table fol- lows similar procedure in pre-training. The only difference is that the encoded sentences for fine-tuning are generated by applying textual en- coding to the downstream table.

---

Table 2:
The experimental results in privacy protection. Here we present the difference in metrics between the model trained the synthetic data and the trained the original data, the lower the better. A gap close to zero suggests that the synthetic data is of comparable quality to the original data. Below the backbone model is Li Results of MLP and Transformer can be found in Table and 14.
Difl w.t.f Origin 1
LO
AD
HE
CR
SI
BE
D1
CA
GE
ME
AG
DU
Avg.
Rank
CTGAN
12.1
3.7
8.6
87.2
4
24.2
5.6
38.4
14.3
101.1
27.5
104.1
5.88 + 0.91
CopulaGAN
14.2
3.4
8.7
0.2
4
27.8
5.5
$7.3
14,5
83.6
26.6
105.7
5.79 + 1.47
TVAE
5.7
0.8
1.4
7.6
7.9
2.6
16.7
5.1
109.4
33.7
34.2
5.00 +: 1.41
GReaT-distill
1.4
2
2.8
19.8
1.8
4.8
22.8
8.4
7.6
87.7
4.50
1.09
GReaT
2.5
0.9
3.7
2.6
14.5
1.9
1.7
13.1
2.4
0.7
4.5
25.7
3.75 + 1.29
TAPTAP-distil
0
0.7
0.4
1.1
0.4
1.6
3.7
0.7
0.2
0.6
16.7
1.71 0.45
TAPTAP
0.5
0.3
0.6
0.4
1.6
2.5
1.5
0
4.6
12.8
1.38
0.64
Table 3: The experimental results in low resource regime. "+ Ori" means training with the original data. "+ Ori + Synthetic Data" means training with the original data plus the synthetic data. Below the backbone model is Transformer with piece-wise linear encoding. The full results on all datasets can be found in Table 15 and 16.
Metric
LO
HE
BE
SI
CA
GE
ME
AG
DU
Avg. Rank
Transformer Ori
76.8
72.5
92.7
98.5
82,9
98.2
86.6
52.6
96.5
Transformer + Ori + Synthetic Data by Models
CTGAN
74.7
71.5
92.7
97.8
81.5
96.3
72.1
51.6
71.7
5.83 + 1.27
CopulaGAN
74.7
71.8
92.5
97.8
81.7
95.9
72.8
52.0
86.8
5.39
1.32
TVAE
76.2
72.8
92.5
97.4
82.0
97.2
85.7
47.3
80.0
4.56
2.01
GReaT-distill
76.1
72.0
92.6
98.3
77.9
96.6
86.2
52.4
79.0
4.89
1.05
GReaT
74.5
72.1
92.7
98.4
80.5
98.1
86.4
53.3
80.3
4.11
1.45
TAPTAA-distil
76.2
72.5
92.8
98.5
83.7
98.2
86.9
53.8
98.2
1.78
1 0.67
TAPTAP
77.5
72.5
92.9
98.5
83.7
98.2
86.7
53.5
97.9
1.44 + 0.53
choice for label generation, since most commonly used tabular prediction models (e.g.. LightGBM) are carefully designed for tabular data and generally more accurate at predicting the labels (Hegselmann et al., 2022; Shwartz-Ziv and Armon, 2022).
share the same splits. We provide some important statisties of each dataset in Table and more details in Appendix A. Following previous works (Grin- sztajn et al., 2022; Borisov et al., 2022), we use accuracy and R2 score as the evaluation metrics for the classification and regression tasks. For the im- balanced classification scenario, we employ AUC as the evaluation metric. All the experimental re- sults averaged over 10 different random seeds.
Formally, given a downstream table D = [(x; v:)), we first fine-tune TAPTAP on it to gen- erate synthetic tabular features {x(}- Next, back- bone model Fi is trained to fit the original table D. Then, the synthetic labels v/f can be derived using the well-trained model via y/ Fxx)). Finally, the synthetic labels and the synthetic tabular features make up the final synthetic table D. = [xx,, 1/)) The following model analysis in the Section 4.3 reveals that our design of data labeling (i.e., not using LMs for label generation) is crucial for the superior performance of our approach.
Backbone Models To comprehensively evalu- ate TAPTAP, we experiment with various SOTA backbone models for tabular prediction, includ- ing LightGBM (Ke et al., 2017), MLP, and Trans- former (Gorishniy et al., 2021). Modern GBDT models (such as LightGBM and XGBoost) have been the most popular models for tabular predic- tion in the past few years (Shwartz-Ziv and Armon, 2022). We choose LightGBM in our experiments. Recently, MLP and Transformer with piece-wise linear encoding (Gorishniy et al., 2022) are pro- posed to be competitive against LightGBM.
4 Experiments
4.1 Experimental Setup
Datasets and Evaluation Metries We collect 12 diverse real-world datasets from various do- mains (Asuncion and Newman, 2007; Vanschoren et al., 2013). Each dataset is split into a train set (75%) and a test set (25%), and all experiments
Language Models. TAPTAP uses the original GPT2 (Radford et al., 2019) with 355M parameters, while distill uses the distilled version of

---

GPT2
(Sanh al., 2019) with 82M parameters.
TAPTAP- (Pull)
Wco. character
4.2
Main Results
data labeling
71.2
We measure the quality of the synthesized samples by their performance in the application scenarios.
pre-training
70022
70.0 75.0
50.0
Figure 4: Experimental results in the ablation study. The y-axis is the average metric values across all datasets in the privacy protection setting with LightGBM.
Privacy Protection Following the previous work (Borisov et al., 2022), we include baselines CT-
GAN (Xu et al., 2019), TVAE (Xu et al., 2019), CopulaGAN (Patki et al., 2016), GReaT-distill and GReaT (Borisov et al., 2022). All methods are used to generate the same amount of synthetic data the original dataset. The backbone models trained on the synthetic data, and then evaluated on the original test set. The experimental results are presented in Table 2. One can observe that TAPTAP and TAPTAP-distill outperform most of the baseline methods. Noticing that GRea'T also utilizes GPT2, the fact that TAPTAP surpasses it by large margin suggests the superiority of table pre- training. More importantly with table pre-training, the quality of the synthetic data generated by TAP- TAP can even match that of the original data. On half of the privacy protection datasets, LightGBM models trained with our synthetic data achieve al- most the same performance as with the original data. This is highly impressive, especially when considering that none of the synthetic samples ap- pear in the original dataset.
ous work (Jarrett et al., 2022), two missing mech- anisms are used to yield missing values: missing completely at random (MCAR) and missing at ran- dom (MAR). The miss ratio is set be 0.3. We present the results in Table 4. As observed, TAP- TAP always outperforms most baseline methods using one LM and receives the highest average ranking, indicating its superiority.
Imbalanced Classification By generating syn- thetic samples for the minority class, TAPTAP addresses the imbalanced classification problem. Therefore, we compare our methods against pop- ular oversampling methods (Camino et al., 2020), including Random, SMOTE (Chawla et al., 2002), ADASYN (He et al., 2008), Borderline (Han et al.. 2005), SMOTE+ENN (Alejo et al., 2010) and SMOTE+Tomek (Zeng et al., 2016). Following the standard approach (Buda et al., 2018), we down-sample the minority class of each binary- classification dataset so that the imbalanced ratio is 50 (#majority / 1iminority-5000 Experimental results on five binary-classffication datasets are pre- sented in Table 5, where TAPTAP still has the high- est average ranking among all baselines.
Low Resource Regime We perform data aug- mentation to mitigate the low resource dilemma. The baseline methods are identical to those in pri- vacy protection. During fine-tuning, following the experience of multi-task learning in T5 (Raffel al., 2020), first use the synthetic data to fine- tune backbone model. Then, we use the original data to continually fine-tune the model. Experimen- tal results on datasets with less than 30k samples are presented in Table 3, which show that TAPTAP is able to perform comparably or better than all baseline methods on most datasets. Furthermore, TAPTAP contribute significant gains to of the 9 datasets, which is highly non-trivial.
Overall Summarization First, TAPTAP gener- ally improves the performance of different back- bone models in tabular prediction and outperforms the majority of baseline methods various tabu- lar prediction scenarios. Second, the advantage of TAPTAP over P--Tsil suggests that table pre- pre-training can also benefit from scaling up LMs. Third, TAPTAP is the first to successfully gener- ate synthetic data for comparable backbone model performance to original data.
Missing Value Imputation We compare with top methods as baselines in recent benchmarking study (Jarrett al., 2022), including GAIN (Yoon al., 2018), Hyperlmpute (Jarrett al., 2022), MICE (Van Buuren and Eroothuis-Oudssoonrt 2011), MissForest (Stekhoven and Bühlmann, 2012), MIWAE (Mattii and Frellsen, 2019), and Sinkhorn (Muzellec et al., 2020). Following previ-
4.3 Ablation Study
To investigate the effectiveness of each component in TAPTAP, we conduct an ablation study. We name TAPTAP without different components as follows: (1) w.o. pre-training refers to TAPTAP without table pre-training. (2) w.o. data labeling

---

Table 4: The experimental results in missing value imputation. "+ M--rri means training with the original data processed by the MCAR mechanism. "+ M-Ori Synthetic Data" means training with the M-Ori data where the missing values are imputed by different models. Below the backbone model is MLP. Results using LightGBM and Transformer as backbone models can found in Table and 18. Results with the MAR mechanism can be found in Appendix B.6. X denotes the method cannot successfully the dataset due too many missing values.
Metric 1
LO
AD
HE
CR
BE
DI
CA
GE
ME
AG
DU
Avg. Rank
MLP M-Ori
73.2
85.4
71.1
93.6
90.3
57.4
63.1
93.0
68.4
41.4
70.6
MLP
M-Ori + Synthetic Data by Models
MIWAE
71.3
X
68.7
X
95.7
90.0
X
59.0
90.6
65.0
42.6
75.7
7.54
+ 0.78
Sinkhorn
73.2
83.9
69.3
93.5
95.8
88.6
56.6
62.5
93,4
73.3
49.9
72.8
5.75
1 1.14
GAIN
86,2
86.2
72.9
57.6
97.6
90,9
53.8
54.9
92.9
69.4
44.2
82.0
5.00
+ 2.49
MICE
73.1
84.5
70.0
93.6
96.0
88.3
57.2
63.0
93.8
72.3
53.1
91.1
4.75
1 1.82
MissForest
72.9
79.8
69.9
92.7
96.7
91.6
57.2
74.0
94.2
79.5
46,4
88.5
4.75
+ 1.54
Hyperfmpute
73.4
86.7
70.5
83.0
96.8
92.8
X
77.7
96.2
78.4
58.1
90.6
3.54
+ 1.78
TAPTAP-distil
74.9
86.9
72.2
93.7
97.5
93.4
57.2
78.5
94.5
72.6
53.6
69.2
2.83 + 1.90
TAPTAP
73.6
87.0
73.0
93.7
96.9
93.1
57.8
82,7
97.3
81.2
53.2
85.5
1.83 + 1.11
Table 5: Experimental results in imbalanced classi- fication. "I-Ori" is the imbalanced data. Below the backbone model is LightGBM. denotes the method cannot run successfully on the dataset due to too few samples in the minority class. The metric is AUC.
85
Aug. Metric (%)
81.1
75
70.2
70.9
70
65
(Full)
Metric
LO
AD
HE
CR
S1
Avg. Rank
Amount of Pretraining Cospus (Millions)
LightGBM 1-Oii 71.2
90.2
82.3
84.0
99.4
Figure 5: The influence of pre-training scale the downstream performance. The value of each method the average metric values across all datasets in the privacy protection setting with LightGBM.
LightGBM 1.Ori + Synthetic Data by Models
SMOTE+ENN
SMOTE-Tomek
89.3
80.3
84.1
99.5
5.80
1.44
ADASYN
89.5
79.6
84.0
99.5
5.30
1 0.97
Random
$1.7
89.4
82.3
82.9
99.7
5.00
+ 2.00
SMOTE
89.5
80.3
84.1
99.5
5.00
+ 1.37
Table 6: The comparison between TAPTAP and TAP- TAP with additional web tables for pre-training.
Borderline
71.2
89.6
79.3
83,5
4.20 +:
TXPTAP-distill
91.3
84.8
99.7
1.80
TAPTAP
85.5
91.3
85.0
99.7
1.60 + 0.89
AD
HE
CR
S1
DI
CA
TAPTAP
87.5
72.2
93.8
97.6
57.8
81,5
Web Tables
87.7
72.3
93.8
98.2
57.6
82.1
refers to TAPTAP using LMs to generate labels. (3) character refers to TAPTAP without using the
4.4 Analysis
The Scale of Pre-training Corpus Figure il- lustrates the influence of the pre-training scale on the downstream performance. We present the re- sults with 0.02, 0.1, 0.5 and 2 million samples. As one can observe, scaling up the pre-training corpus brings positive effects. However, the number of high-quality real-world tabular datasets is limited. Therefore, it may be helpful to take advantage of the millions of tables available the Web.
character-level representations for numerical fea- tures. (4) w.o. feature name. The column names of each dataset are replaced by dummy names (e.g., "V1") to remove semantic information.
The experimental results are visualized in Figure 4. We present the average metric values (i.e., Acc. or R2) of each method across 12 datasets in the pri- vacy protection setting, since it is the most straight- forward setting to indicate the quality of synthetic data. We see that pre-training and data label- ing are particularly important for TAPTAP. The semantic information in column names and the character-level representation to enhance number encoding also provide considerable improvement.
Pre-training using Web Tables To explore the above direction, we present a preliminary study on using tables from Web for pre-training. We parse over 130k Web tables with a total of 8 million samples from the WikiTables corpus (Bhagavatula et al., 2015). We use the Web tables together with

---

the tabular datasets for pre-training. The results of the privacy protection setting are presented in Table 6, We can see that even with large number of Web tables, it is still hard to further boost the backbone models. We attribute it to the quality issue. The collected tabular datasets have already been examined by the platforms, and usually have higher quality than noisy Web tables. How to au- tomatically identify high-quality tables from the huge number of Web tables for pre-training a promising future direction.
millions of samples. For example, there may be tables that have the potential to wrongly associate recruitment result to gender. Also, since our model is pre-trained based on GPT, readers may be con- cerned that the synthetic tables generated by our model contain offensive content. On this point, we argue that one might not be worried too much since for categorical features, our model can be easily tuned to only generate the values that appear in the downstream table, which is relatively controllable.
5 Conclusion & Future Work
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 2623-26331 ACM.
In this paper, we propose TAPTAP, a table pre- training method to empower models for tabular prediction. It can be combined with various back- bone models and boost them via synthesizing high- quality tabular data. A large-scale empirical study demonstrates that TAPTAP can benefit different SOTA backbone models on four tabular prediction scenarios. In the future, we plan to extend TAPTAP to process tables with a large number of features.
Roberto Alejo, José Martínez Sotoca, Rosa Maria Val- dovinos, and P. Toribio. 2010. Edited nearest neigh- bor rule for improving neural networks classifications. In Advances in Neural Networks ISNN 2010, 7th International Symposium on Neural Networks, ISNN 2070, Shanghai, China, June 6-9, 2010, Proceedings, Part I, volume 6063 of Lecture Notes in Computer Science, pages 303 310. Springer.
6 Acknowledgement
We thank all the anonymous reviewers for their constructive feedback and insightful comments. Tianping Zhang, Shaowen Wang, and Jian Li are supported in part by the National Natural Science Foundation of China Grant 62161 146004.
Ewa Andrejczuk, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, and Yasemin Altun. 2022. Table-to-text generation and pre-training with tabt5. CoRR, abs/2210.00166.
Limitations
Sercan Ö. Arik and Tomas Pfister. 2021. Tabnet: Atten- tive interpretable tabular learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thine Conference Innovative Ap- plications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Ar- tificial Intelligence, EAAI 2021, Virtual Event, Febru- ary 2-9. 2021, pages 6679-6687. AAAI Press.
The major limitation of TAPTAP is the scalability. While we enjoy the advantages of LMs, we also introduce the drawbacks of LMs. In practice, TAP- TAP usually requires more running time and GPU memory than other methods. Detailed comparison can be found in Appendix B.3. In addition, TAP- TAP can only process tabular data with less than 100 features due to the input length limitation that GPT can process (i.e., 1024 tokens).
Arthur Asuncion and David Newman. 2007. Uci ma- chine learning repository.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. 2022. Scarf: Self-supervised contrastive learning using random feature corruption. In The Tenth In- ternational Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net.
Ethics Statement
In this paper, we collected and filtered out 450 publicly available tabular datasets to construct the pre-training corpus for TAPTAP. As these datasets have been reviewed by well-known machine learn- ing platforms such as Kaggle, they should have no private information about individuals. How- ever, we cannot confirm whether these datasets contain potential biases since the corpus contains
Omar Benjelloun, Shiyu Chen, and Natasha F. Noy. 2020. Google dataset search by the numbers. In The Semantie Web ISWC 2020 19th International Se- mantic Web Conference, Athens, Greece, November 2-6, 2020, Proceedings, Part II, volume 12507 of Lecture Notes Computer Science, pages 667-682 Springer.

---

Chandra Sekhar Bhagavatula, Thanapon Noraset, and Doug Downey. 2015. Tabel: Entiy linking in web tables. In The Semantic Web ISWC 2015 14th International Semantic Web Conference, Bethlehem, PA, USA, October 1 11-15, 2015, Proceedings, Part I, volume 9366 of Lecture Notes in Computer Science, pages 425-441. Springer.
for text-to-SQL. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 1337-1350, Online. As- sociation for Computational Linguistics.
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. 2022. Language models are realistic tabular data generators. CoRR, abs/2210.06280.
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. 2018. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-25
Haoyu Dong, Zhoujun Cheng. Xinyi He, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dong- mei Zhang. 2022. Table pre-training: A survey on model architectures. pre-training objectives, and downstream tasks. In Proceedings of the Thirty-His International Joint Conference Artificial Intelli- gence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5426-5435. ijcai.org.
Ramiro Daniel Camino, Radu State, and Christian A. Hammerschmidt. 2020. Oversampling tabular data with deep generative models: Is it worth the effort? In "I Can't 't Believe It's Not Better!" at NeurlPS Work- shops, Virtual, December 12, 2020, volume 137 of Proceedings of Machine Learning Research, pages 148 157. PMLR.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. 2019. Learning imbalanced datasets with aabel-distribution aware margin loss. In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1565-1576
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with interme- diate pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 281-296, Online. Association for Computational Lin- guistics.
Adrià Gascón, Phillipp Schoppmann, Borja Balle, Mari- ana Raykova, Jack Doerner, Samee Zahur, and David Evans. 2017. Frivacy-preservin distributed linear regression on high-dimensional data. Proc. Priv. En- hancing Technol. 2017(4):345-
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: synthetic minority over- sampling technique. J. Artif. Intell. Res., 16:321-357.
Tianqi Chen and Carlos Guestrin. 2016. Xghoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Duta Mining, San Fran- cisco, CA, USA, August 13-17, 2016, pages 785-794. ACM.
Heng Gong, Yawei Sun, Xiaocheng Feng. Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. 2020. TableGPT: Few-shot table-to-text generation with table structure reconstruction and content matching. In Proceedings of the 28th International Conference on Computational Linguisties, pages 1978-1983 Barcelona, Spain (Online). International Committee Computational Linguistics.
Edward Choi. Siddharth Biswal, Bradley A. Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. 2017. Generating multi-label discrete patient records using generative adversarial networks. In Proceedings of the Machine Learning for Health Care Conference, MLHC 2017, Boston, Massachusetts, USA, 18-19 August 2017, volume 68 of Proceedings of Machine Learning Research, pages 286-305. PMLR.
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. 2022. On embeddings for numerical features in tabu- lar deep learning. CoRR, abs/2203.0555
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting deep learn- ing models for tabular data, Advances Neural Information Processing Systems 34: Annual Confer- ence Neural Information Processing Systems 2021, NewrlPS 2021, December 6-14, 2021, virtual, pages 18932-18943.
Will Cukierski Credit Fusion. 2011. Give me some credit.
Sajad Darabi and Yotam Elor. 2021. Synthesising multi- modal minority samples for tabular data. CoRR, abs/2105.08204.
Léo Grinsztajn, Edouard Oyallon, and Gaël Varo- quaux. 2022. Why do tree-based models still out- perform deep learning on tabular data? CoRR. abs/2207.0881
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. 2021. Structure-grounde pretraining

---

Hui Han, Wenyuan Wang, and Binghuan Mao. 2005, Borderline smote:
Roman Levin, Valeria Cherepanova. Avi Schwarzschild, Arpit Bansal, C. Bayan Bruss. Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. 2022. Transfer learning with deep tabular models. CoRR, abs/2206. 15306.
sontee: A new over-sampling method in
imbalanced data sets learning. In Advances in Intelli- gent Computing, International Conference on Intel- ligent Computing, ICIC 2005, Hefei, China, August 23-26, 2005, Proceedings, Part I, volume 3644 of Lecture Notes in Computer Science, pages 878-887. Springer.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising
Haibo He, Yang Bai, Edwardo A. Garcia, and Shutao Li. 2008. ADASYN: adaptive synthetic sampling approach for imbalanced learning. In Proceedings of the International Joint Conference on Neural Net- works, IJCNN 2008, part the IEEE World Congress on Computational Intelligence, WCCI 2008, Hong Kong, China, June 1-6, 2008, pages 1322-1328. IEEE.
pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880, Online. Association for Computa- tional Linguistics.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin. Weizhu Chen, and Jian-Guang Lou. 2022. TAPEX: table pre-training via learning neural SQL executor. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David A. Sontag. 2022. Tablim: Few-shot classification of tabular data with large language models. CoRR, abs/2210.10723.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computa- tional Linguistics.
Chao Ma, Sebastian Tschiatschek, Richard E. Turner, José Miguel Hernändez-Lobatee and Cheng Zhang. 2020. VAEM: a deep generative model for hetero- geneous mixed type data. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Daniel Jarrett, Bogdan Cebere, Tennison Liu, Alicia Curth, and Mihaela van der Schaar. 2022. Hyperim- pute: Generalized iterative imputation with automatic model selection. In International Conference on Ma- chine Learning, ICMI. 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 9916-9937. PMLR.
Pierre-Ales Alexandre Mattei and Jes Frellsen. 2019. MI- WAE: deep generative modelling and imputation of incomplete data sets. In Proceedings of the 36th In- ternational Conference on Machine Learning, ICML, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Re- search, pages 4413-4423. PMLR
Alejandro Mottini, Alix Lheritier, and Rodrigo Acuna- Agost. 2018 Airline passenger name record gener- ation using generative adversarial networks. CoRR, abs/ 00/1187..000 506.06657.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma. Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boost- ing decision tree. In Advances in Neural Information Processing Systems 30: Anmual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 146-3155.
Boris Muzellec, Julie Josse, Claire Boyer, and Marco Cuturi. 2020. Missing data imputation using optimal transport. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13- July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 7130-71 140. PMLR.
Ron Kohavi. 1996. Scaling up the accuracy of naive- bayes classifiers: A decision-tree hybrid. In Pro- ceedings of the Second International Conference on Knowledge Discovery and Data Mining ( (KDD-96), Portland, Oregon, USA, pages 202-207. AAAI Press.
Muhammad Ferjad Nacem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. 2020. Reliable fidelity and diversity metrics for generative models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume l 119 of Proceedings of Machine Learning Research, pages 7176-7185. PMLR.
Aki Koivu, Mikko Sairanen, Antii Airola, and Tapio Pahikkala. 2020. Synthetic minority oversampling of vital statistics data with generative adversar- ial networks. J. Am. Medical Informatics Assoc., 17(11):16771167.
Murat Koklu and llker Ali Özkan, 2020. Multiclass classification of dry beans using computer vision and machine learning techniques. Comput. Electron. Agric., 174: 105507.
R Kelley Pace and Ronald Barry. 1997. Sparse spa- tial autoregressions. Statisties & Probability Letters, 33(3)

---

Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. 2018. Data synthesis based on generative adversarial networks. Proc. VLDB Endow., (0110)0071-1088
Stef Van Buuren and Karin Stoothuis-Oudshooe 2011. mice: Multivariate imputation by chained equations in r. Journal of statistical software, 45:1-67.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luís Torgo. 2013. Openml: networked science in machine learning. SIGKDD Explor, 15(2):49-60) 60.
Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. 2016. The synthetic data vault. In 2016 IEEE Inter- national Conference on Data Science and Advanced Analytics, DSAA 2016, Montreal, QC. Canada, Octo- ber 17-19, 2016, pages 399-410. IEEE.
Eric Wallace, Yizhong Wang. Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know num- bers? probing numeracy in embeddings. In Proceed- ings of the 2019 Conference on Empirieal Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5306-5314, Association for Computational Linguistics.
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guang Lou, and Weizhu Chen. 2022. Reasoning like program executors. CoRR, abs/2201. 11473.
J. Ross Quinlan. 1987. Simplifying decision trees. Int. J. Man Mach. Stud., 27(3):221-22 -334.
Ruoxi Wang. Rakesh Shivanna, Derek Zhiyuan Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. 2021a. DCN V2: improved deep & cross network and practical lessons for web-scale learning to rank systems. In WWW '2: The Web Conference 2021, Virtual Event / I.ubljana, Slovenia, April 19-23, 2021, pages 1785-1797. ACM/ / TW3C2.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18):9
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn, Res.,
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2021b. TUTA: tree- based transformers for generally structured table pre- training. In KDD '21: The 27th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, pages 1780-1790. ACM.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR. abs/1910.01108.
Zifeng Wang and Jimeng Sun. 2022. Transtab: Learn- ing transferable tabular transformers across tables. CoRR, abs/2205.099 09328.
Ravid Shwartz-Ziv and Amitai Armon. 2022. Tabular data: Deep learning is not all you need, Inf. Fusion. 81:84990.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew, 2019, Huggingface's transformers: State-of-the-an natural language processing. CoRR, abs/ 1000/111..3
Gursewak Singh Sidhu. 2021. Crab age prediction.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019. Au- toint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Informa- tion and Knowledge Management, CIKM 2019, Bei- jing, China, November 3-7, 2019, pages 1161-1 170. ACM.
Xinyu Xing and Xiaojun Wan. 2021. Structure-aware pre- mne-triiing for table-to-text generation. In Find- ings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pages 2273-2278, Online. Association for Computational Linguistics.
Daniel J. Stekhoven and Peter Bühlmann. 2012. Miss- forest- non-parametrie missing value imputation for mixed-type data. Bioinform., 28(1) 1):11 12-118.
Lei Xu, Maria Skoularidou, Alfredo Cuesta- Infante, and Kalyan Veeramachaneni. 2019. Modeling tabular data using conditional GAN. In Advances in Neural Information Processing Systems 32: Annual Confer- ence Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7333-7343.
Beata Strack, Jonathan P DeShazo, Chris Gennings, Juan I. Olmo, Sebastian Ventura, Krzysztof J Cios, and John N Clore. 2014. Impact of hbalc mea- surement on hospital readmission rates: analysis of 70,000 clinical database patient records. BioMed research international, 2014.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data, In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, On- line, Association for Computational Linguistics.
Nan Tang, Ju Fan, Fangyi Li, Jianhong Tu, Xiaoyong Du, Guoliang Li, Samuel Madden, and Mourad Ouz- zani. 2021. RPT: relational pre-trained transformer almost all you need towards democratizing data preparation. Proc. VLDB Endow, 14(8)

---

Jinsung Yoon, James Jordon, and Mihaela van der Schaar. 2018. GAIN: missing data imputation us- ing generative adversarial nets. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsässson. Stockholm, Sweden. July 10-15, 2018, volume 80 of Proceedings of Mu- chine Learning Research, pages 5675-5684. PMLR.
CIGAN CopulaGAN
0.5g
0,06
GReaT
TAPTAP
0:85
0.4
0.6
Figure 6: Sampling diversity in terms of the coverage score averaged across datasets.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. 2020. VIME: extending the suc- cess of self- and semi-supervised learning to tabular domain. In Advances Neural Information Pro- cessing Systems 33: Annual Conference on Newral Information P Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
A Datasets
We provide the urls of the public datasets in Table
7. These datasets are publicly available, and their license permits usage for research purposes.
Tao Yu, Chien Shinn Wu, Xi Victoria Lin, Bailin Wang. Yi Chern Tan, Xinyi Yang, Dragomir R. Radev, Richard Socher, and Caiming Xiong. 2021. Grappa: Grammar Grammarr-uummented pre pre-traiming training table semantic parsing. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
B Additional Experiments
B.1 Distance to Closest Record
In order to demonstrate that TAPTAP generates synthetic samples similar to the original data instead of copying the original data, following the standard approach (Borisov al., 2022), we calculate each sample's 's distance to the closest record (DCR) in the original training data D. For each synthetic sample æ, its DCR is DCR(() = min (iinDDislane(aa 2t)]( E D). We use the L distance for numerical features. For categorical features, set the distance to be 0 for equal cate- gories and otherwise. We present the results of California Housing and HELOC in Figure 7 and 8.
Min Zeng. Beiji Zou, Faran Wei, Xiyao Liu, and Lei Wang. 2016. Effective prediction of three common diseases by combining smote with tomek links tech- nique for imbalanced medical data. 2016 IEEE In- ternational Conference of Online Analysis and Com- puting Science (ICOACS), pages 225-228 IEEE.
Tianping Zhang, Zheyu Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Wei Cao, and Jian Li. 2022. Openfe: Automated feature generation beyond expert-level performance. CoRR, abs/2211.12507
B.2 Sampling Diversity
We employ the coverage score (Naeem et al., 2020) to quantitatively evaluate the sampling diversity of TATTPAA TTPP and baseline methods. The coverage refers to the proportion of actual records that con- tain at least one synthetic record within its mani- fold. A manifold is defined sphere surrounding the sample, with radius of determined by the distance between the sample and its k-th nearest neighbor. We present the averaged coverage score in Figure
B.3 Running Time Comparison
We analyze the running time of TAPTAP, TAPTAP- distill, and baseline methods. The experiments are carried out single NVIDIA GeForce RTX 3090 with 24 GB RAM, 64 system RAM, and Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60GHz with 16 cores. For the privacy protection setting, we present the running time of training/fine-tuning and sampling separately. We present the results of the Adult Income dataset in Table 8. For the missing

---

Table The urls of test datasets.
Dataset
Link
Adult Income (AD) (Kohavi, 1996)
https://archive... ucss.uci...
(HE)
https://wwr. wtwpw//wwwgg..../
http://.......teeete//eeeell//// equity 1ine theloc
Califomia Housing (CA) (Pace and Rarry, 1997)
hetps://www http:///ww.aaaaaaa//////////aa//
Diabetes (DI) (Strack al., 2014)
hctps://ww.. htpps///wwwkgggll....
kagg prediction
Credit Scoring (CR) (Credit Fusion, 2011)
Loan (LO)
https//www hor.m/l....//////
Housing (DU)
https://ww/ http:///wwwaaaaalcooo//oa/taata/
Crab Age (AG) (Sicho, 2021)
hctps://ww.. Aaggglee..cc//ddtttsttt//ss/ssrr prediction
Medical Cost (ME)
hctps://wer htt////wkggll.... http://....o.ctaaaatt//etraaaaan
Gem Price (GE)
hetps://wwe wag.g//...ddaaaaa///oooaaaaaggee prac pecpeestttteeeeeeeeeeeeeeeeeeeee price--eeiii--iiiiionnn
Bean Type (BE) (Koklu Ozkan, 2020)
https://archvv wtt..///ww//.eeaattttt////rrrrre
Sick Record (SD) (Quinlan, 1987)
https opennl. wwtps//rgg//ttttteeeeeeeeeoeeeee wwt.m./.g....a....a.ataoooortree
value imputation setting, we present the rumning time of the California Housing dataset in Table 9, We can see that TAPTAP and TAPTAP-distil re- quires more running time than most of the baseline methods. While we enjoy the benefits of lever- aging LMs to achieve top performance, we also introduce the drawbacks of LMs in requiring more computational resources. However, there are im- portant real-world applications such healthcare finance where achieving better performance out- weighs saving computational time. In addition, the fine-tuning and sampling time can be reduced by using more computational resources.
data points in the "weight" column in the Diabetes dataset missing. However, the methods require at least one valid value for each training batch.
B.7 Imbalance Classification
Table shows the performance of our method and baseline methods the imbalance classification setting with LightGBM as the backbone. Smote- based methods fail on the Loan dataset because there are fewer than 10 minority class data, which results in the number of sampled data points being less than the number of neighbors(Chawla et al., 2002) required.
B.4 Privacy Protection
C Hyperparameters Optimization
Table 13 and 14 show the performance of our method and baseline methods in privacy protection setting with LightGBM, MLP, and Transformer as the backbone.
We use optuna (Akiba et al., 2019) to tune the hy- perparameters of our backbone models, i.e. Light- GBM, MLP, and Transformer. For each specific dataset and model, we first use the original data to tune the hyperparameters of the model. Then the set of hyperparameters used throughout all the experiments of the dataset all the methods for fair comparison.
B.5 Low Resource Regime
Table 15 and 16 show the performance of our method and baseline methods low resource regime setting with MLP and Transformer as the backbone. Note that both low resource datasets and high resource datasets are presented in the table.
C.1 LightGBM
When tuning the hyperparameters of LightGBM, the following hyperparameters are fixed:
B.6 Missing Value Imputation
1. boosting = "gbdt"
Table 17, and 18 show the performance our method and baseline methods missing value imputation setting using MCAR mechanism with LightGBM, MLP, and Transformer as the back- bone. Table 19, 20 and 21 show the performance of method and bascline methods in missing value imputation setting using MAR mechanism with LightGBM, MLP, and Transformer as the back- bone. MIWAE and HyperImpute fail on some datasets because one feature in the dataset contains too many missing values. For example, 96.9% of
early_ ftlly_stopping_rott round 50
3. estimatoor 1000
Other hyperparameters and the search space for tuning are in Table 10.
C.2 MLP
We follow the implementation in Gorishniy et al. (2022). We present the hyperparameters space for searching in Table 11.

---

Table 8: The running time in seconds on the Adult Income dataset different methods in the privacy protection setting. The number of fine-tuning steps for GReaT and TAPTAP was 10k. A total of 36k samples generated.
CTGAN
CopulaGAN
TVAE
GReaT-distill
GReaT
TAPTAP-distill
TAPTAP
Training/Fine-tuningg Time
873
846
360
3770
Sampling Time
9
11
3
895
1395
506
1185
Table 9: The running time in seconds the California Housing dataset of different methods in the missing value imputation setting. The number of fine-tuning steps for GReaTT and TAPTAP was 10k. A total 15k samples were imputed.
MIWAE
Hyperlmpute
GAIN
MICE
MissForest
Sinkhorn
TAPTAP-distill
TAPTAP
Running Time
210
175
8
336
47
565
1215
4008
Table 10: LightGBM hyperparameter space
Table 12: Transformer hyperparameter space
Parameter
Distribution
Parameter
Distribution
learning_rad rate
Uniform(0.01 (0.0m/0001...S
# Layers
Uniformint 1 .4]
num num_leaves
UniformInt 10, 100]
Embedding size
UniformIn[9664 591
min_child child_ weight
LogUniform[ S,1e-]
Residual dropout
[0. Uniform(O,0.22
min child samples
(niformInt(2.1000
Attetion dropout
Uniform[0,0.3
subsample
Uniform[0.5. 0.5,
FFN dropout
Uniform[0.00 [000.5]
colsample bytree
Uniform[0.5. (..
FFN factor
Uniform| [2/3,8/3)
Leaning rate
LogUniform] Ie-5, 1c-3)
Iterations
Weight decay
LogUnifome 1c-6, Ic-4)
Iterations
100
Table 11: MI. hyperparameter space
Parameter
Distribution
line methods of Random, SMOTE, ADASYN, Borderline, SMOTE+ENN, SMOTE+Tomek the imbalanced classification setting, we use the implementation in https://gite tttp://ggtttuu...e comb
Layers Layer size
Uniformant( 16]
Uniformlat| 1024]
Dropout
Uniform[0.0 (0.0.5)
Learning rate
(0, Uniform[0.0.51)
Weight decay
JogUnifor[[Se5.00m
# Iterations
100
We use the implementation of GPT2 and the distilled version of GPT2 in the huggingface plat- form (Wolf al., 2019). We pre-train TAPTAP and TAPTAP-distie for 80,000 steps. We finetune the TAPTAP, TAPTAP-distill, GReaT, and GReaT- distill model for 10,000 steps, except for the Credit Scoring (CR) and Sick Records (SI) datasets, which we finetune the model for 20000 steps. The batch size is 64 for all the datasets. In the privacy protec- tion, low resource regime, and imbalanced classi- fication setting, we use the one feature-value pair as prompt sampling method. We start sampling with the target feature following the previous ap- proach (Borisov et al., 2022). The missing value imputation setting does not require pscudo label generation, as the missing mechanism only drops the feature values and the labels are always pro- vided. In the imbalanced classification setting, we generate synthetic samples on the minority class until the number of samples is the same for the minority and majority class.
C.3 Transformer
We follow the implementation Gorishniy et al. (2022). We present the hyperparameters space for searching in Table 12.
D Reproducibility Details
For the baseline methods of CT-GAN, TVAE, and CopulaGAN in the privacy protection and low resource regime setting, we use the implementation in https://sdv dev/SDV/ user_ggudee//singee_ttae table/models GReaT-distill and GReaT, we use the implemen- tation in https://gite thub. great. For the baseline methods of GAIN, Hy- perlmpute, MICE, MissForest, MIWAE, Sinkhorn the missing value imputation setting, we use the implementation in https: http://gtttth.ccc com/ monderschaarlLb/yyperin For the base-
For

---

Table 13: The experimental results in privacy protection. "+ Ori" training with the original data. Below the backbone model is MLP with piece-wise linear encoding.
Metric
1.0
AD
HE
CR
ST
CA
ME
DU
Avg. Rank
MLP Ori
76.6
87.7
72.4
93.8
98.3
92.7
58.7
81.8
98.1
85.7
52,9
99.5
MIPP Synthetic Data by Models
CTGAN
64.9
84.3
65.4
6.6
94.7
65.7
53.8
46.4
83.5
-16.7
31.0
-10.6
6.17 + 0.62
CopulaGANN
63.2
84.1
64.0
93.5
94.7
62.3
53.9
31.2
83.6
10.3
31.4
-15.4
5.96 + 1.25
TVAE
64.9
82.7
71.7
92.4
95.7
70,9
56.1
60.8
93.8
-22.0
19.0
66.2
5.04 + 1.36
GReaT-distill
85.8
69.8
91.3
97.1
90.9
53.9
63.4
80.6
15.7
1.00
GReaT
73,9
86.7
71.0
91.5
97.4
90.7
57.6
72.0
96.6
84.7
49.2
69.5
3.25 + 0.97
TAPTAP-distl
77.0
87.4
72.3
93.8
97.5
92.2
57.2
80.5
98.1
86.7
53.3
77.6
1.83 1 0.58
TAPTAP
87.4
72.3
93.8
97.8
92.6
57.3
81.9
97.1
85.2
86.4
1.33 0.49
Table 14: The experimental results in privacy protection. "+ Ori" means training with the original data. Below the backbone model is Transformer with piece-wise linear encoding.
Metric
LO
AD
HE
CR
S1
BE
DI
CA
GE
ME
AG
DU
Avg. Rank
Transformer Ori
76.8
87.4
72.5
93.8
98.5
92.7
58.7
82.9
98.2
86.6
52.6
96.5
Transformer + Synthetic Data by Models
CTGAN
65.1
84.1
65.2
6.6
94.7
55.9
53.8
48.3
84.3
-14.3
24.5
-12.4
6.17 + 0.58
CopulaGAN
61.3
84.2
64.3
93.5
94.6
58.0
53.9
83.3
11.1
23.2
-16.1
6.08 + 1.24
TVAE
65.6
82.5
71.7
92.0
95.6
65.7
56.2
58.9
92.8
-2422
19.1
49.8
5,00 1 1.35 1.35
GReaT-distill
74.2
85.4
91.3
96.8
54.0
83.4
44.6
14.2
4.25 0.87
GReaT
72.3
86.5
68.7
91.3
97.2
90.3
57.6
71.9
96.4
85.6
48.1
60.4
3.25 + 1.14
TAPTAP-distil
76.7
87.2
72.2
93.8
97.3
92.0
57.2
81.5
98.1
86.9
53,5
63.1
1.75 0.62
TAPTAP
77.1
87.3
72.1
93.8
98.1
92.5
57.3
83.1
96.0
86.1
51.4
75.2
1.50 0.67
Table 15: The experimental results low resource regime. "+ Ori" means training with the original data. "+ Ori Synthetic Data" means training with the original data plus the synthetic data. Below the backbone model is MLP with picce-wise linear encoding.
Metric 1
LO
AD
HE
CR
S1
BE
CA
GE
ME
AG
DU
Avg. Rank
MLP + Ori
76.6
87.7
72.4
93.8
98.3
92.7
58.7
81.8
98.1
85.7
52.9
99.5
MLP
Ori Synthetic
by Models
CTGAN
76.4
87.4
71.1
84.8
96.2
93.0
58.5
80.2
95.6
69.2
50.3
83.9
6.08 + 1.00
CopulaGAN
76.5
87.5
71.4
93.8
97.4
93.0
58.5
80.0
95.1
70.6
51.9
97.9
4.83+1.27
TVAE
76.6
86.8
72.5
93.7
97.3
92.8
58.4
81.3
96.9
84.9
47.7
91.6
4.83 : 1.53
GReaT-distill
76.5
87.6
72.4
93.6
98.0
92.7
58.4
77.6
96.4
84.9
53.0
90.4
5.17 1.27
GReaT
75.8
87.6
72.1
93.5
98.2
93.0
58.5
80.1
97.9
85.8
53.4
91.4
4.08
1.44
TAPTAP-distil
76.8
87.7
98.5
93.0
58.6
85.9
54.2
99.5
0.49
TAPTAP
76.9
87.7
72.6
93.8
98.4
93.1
58.7
83.6
98.1
86.0
53.9
99.5
1.33 1 0.49

---

Table 16: The experimental results in low resource regime. "+ Ori" means training with the original data. "+ Ori + Synthetic Data" means training with the original data plus the synthetic data. Below the backbone model is Transformer with piece-wise wise linear encoding.
Metrie 1
L.
AD
HE
CR
SI
BE
DI
CA
GE
ME
AG
DU
Avg. Rank
Transformer Ori
76.8
87.4
72.5
93.8
98.5
92.7
58.7
82.9
98.2
86.6
52.6
96.5
Transformer + Ori + Synhhtic Data by Models
CTGAN
74.7
87.2
71.5
84.8
97.8
92.7
58.5
81.5
96.3
72.1
51.6
71.7
5.79 + 1.20
CopulaGAN
74.7
87.2
71.8
93.8
97.8
92.5
58.5
81.7
95.9
72.8
52.0
86.8
5.12 + 1.38
TVAE
76.2
86.8
72.8
93.7
97.4
92.5
58.4
82.0
97.2
85.7
47.3
80.0
4.83 + 1.90
GReaT-distill
76.1
87.5
72.0
93.6
98.3
92.6
58.4
77.9
96.6
86.2
52.4
79.0
5.00 + 1.13
GReaT
74.5
87.6
72.1
93.6
98.4
92.7
58.5
80.5
98.1
86.4
53.3
80.3
3.92 - 1.68
PdPttl-distill
76.2
87.6
72.5
93.8
98.5
92.8
58.6
83.7
98.2
86.9
53.8
98.2
1.83 + 0.58
TAPTAP
77.5
87.5
72.5
93.8
98.5
92.9
58.7
83.7
98.2
86.7
53.5
97.9
1.50 + 0.67
Table 17: The experimental results in missing value imputation. "+ M-Ori" means training with the original data processed by the MCAR mechanism. "+ M-Ori Synthetic Data" means training with the M-Ori data where the missing values are imputed by different models. Below the backbone model is LightGBM. X denotes the method cannot run successfully on the dataset due to too many missing values.
Metric
LO
AD
HE
CR
SI
BE
DI
CA
GE
ME
AG
DU
Avg. Rank
LightGBM M-Ori
73.3
86.2
71.3
93.7
97.0
91.1
57.4
68.0
93.1
66.6
44.2
83.5
LightGBM
M-Ori + Synthetic Data by Models
MIWAE
71.0
X
69.3
X
96.8
90.3
64.3
90.2
65.6
41.3
81.4
7.21 + 0.94
Sinkhorn
73.8
84.5
69.4
93.7
96.8
66.5
93.3
67.9
50.8
81.2
6.00 + 1.21
MICE
74.5
85.3
69.9
93.6
96.1
89.6
57.2
66.2
94.0
71.0
52.9
89.5
5.17 + 1.47
GAIN
85.4
86.4
74.3
62.8
67.3
44.6
85.1
4.67 2.64
MissForest
67.7
86.4
71.5
93.7
97.8
91.5
57.0
73.5
94.6
77.0
46.4
90.7
4.42 + 1.62
Hyperlmpute
69.6
88.0
71.0
91.7
97.7
92.7
80.8
96.3
79.8
57.3
92.2
3.46 + 2.50
TAPTAP-distil
75.2
87.3
72.4
93.7
98.3
93.4
57.3
80.5
94.6
70.0
53.9
70.2
3.00 1 1.91
TAPTAP
74.8
87.4
72.8
93.7
97.8
93.2
57.7
85.0
97.5
77.8
53.7
85.8
2.08 + 0.90
Table 18: The experimental results in missing value imputation. "+ M-Ori" means training with the original data processed by the MCAR mechanism. "+ M-Ori + Synthetic Data" means training with the M-Ori data where the missing values are imputed by different models. Below the backbone model is Transformer. X denotes the method cannot run successfully on the dataset due to too many missing values.
Metric 1
LO
AD
HE
CR
SI
BE
DI
CA
GE
ME
AG
DU
Avg. Rank
Transformer M-Ori
73.4
85.5
71.2
93.6
96.7
90.6
57.4
63.8
93.5
71.0
41.9
67.1
Transformer M-Ori + Synthetic Data by Models
MIWAE
72.7
X
68.7
X
96.3
89.8
X
61.0
90.9
68.8
42.8
74.0
7.46 1 0.66
Sinkhom
72.1
83.6
69.5
93.6
89.1
56.6
63.8
93.5
74.4
50.4
79.8
5.67
+ 1.56
GAIN
77.2
86.1
70.1
52.1
97.8
90.3
53.8
50.5
93.3
69.6
44.6
75.2
5.33 4 2.19 2.19
MICE
73.0
84.7
69.9
93.6
95.5
89.0
57.6
64.2
93.8
74.1
52.1
76.7
5.25 + 1.66
MissForest
73.0
83.9
70.7
92.8
97.3
91.6
57.3
74.6
94.7
46.3
82.8
4.00
+ 1.28
Hyperlmpute
75.3
86.7
69.8
83.6
97.2
92.8
X
77.7
96.4
80.0
56.8
85.5
3,46
1: 2.15
TAPTAP-distlll
74.6
86.9
72.3
93.6
98.0
93.3
57.2
79.0
94.6
75.0
53.2
68.4
2,92 + 1.93
TAPTAP
73.1
87.0
72.7
93.7
97.5
93.2
57.8
83.6
97.6
82.5
$2.4
78.6
1.92 1: 1.24

---

Table 19: The experimental results in missing value imputation. "4 M-Ori" means training with the original data processed by the MAR mechanism. "+ M--Ori+ Synthetic Data" means training with the M-Ori data where the missing values are imputed by different models. Below the backbone model is LightGBM. denotes the method cannot run successfully on the dataset due to too many missing values.
Metric
1.0
AD
HE
CR
S1
BE
CA
GE
ME
AG
DU
Avg. Rank
LightGBM M-Ori
77.1
86.9
72.1
93.7
97.3
91.8
58.5
80.1
93.7
50.5
52.0
82.9
LightGBM M-Ori + Synthetic Data by Models
Sinkhorn
77.1
86.3
69.9
93.8
97.3
91.2
58.7
78.6
93.5
49.8
50.3
61.3
6.08 + 1.83
MIWAE
78.2
86.4
69.9
X
97.3
91.6
79.1
92.7
48.1
51.8
79.0
5.88 + 1.68
MICE
77.1
86.7
70.4
93.8
96.6
91.2
58.3
79.4
93.7
62.8
52.4
93.2
5.33 + 1.78
GAIN
78.8
87.0
75.4
97.0
97.2
90.0
69.7
88.3
45.9
55.0
74.0
4.92 1 3.09
MissForest
77.0
87.1
68.7
95.9
97.9
92.3
58.5
80.1
93.8
84.2
51.9
59.0
4.67 + 2.19
HyperImpute
77.1
90.0
70.7
94.5
85.8
51.4
60.9
3.96 + 2.38
TAPTAP-distill
77.3
87.6
72.5
93.8
98.3
92.8
58.5
81.0
93.9
73.1
53.0
83.3
2.83 + 0.94
TAPTAP
77.3
87.5
72.6
93.8
98.0
93.1
58.6
83.9
97.0
77.7
53.1
79.1
2.33 1 1.15
Table 20: The experimental results in missing value imputation. "+ M-Ori" means training with the original data processed by the MAR mechanism. "+ M-Ori + Synthetic Data" means training with the M-Ori data where the missing values imputed by different models. Below the backbone model is MLP. X denotes the method cannot run successfully on the dataset due to too many missing values.
Metric 1
LO
AD
HE
CR
SI
BE
DI
CA
GE
ME
AG
DU
Avg. Rank
MLP + M-Ori
77.3
86.2
72.0
93.6
96.4
91.7
58.4
76.9
93.5
53.0
52.1
76.9
MLP + M-Ori Synthetic Data by Models
Sinkhorn
77.3
85.3
69.9
93.7
96.3
91.5
58.4
76.5
93.2
45.5
51.4
76.7
6.29 + 1.39
MIWAE
77.8
85.6
69.6
X
96.7
91.5
X
76.8
92.2
46.9
52.1
74.2
6.12 = 1.71
MICE
77.2
86.0
70.1
93.7
95.7
91.0
58.3
77.1
93.3
61.5
$1.9
97.2
5.50
GAIN
78.1
85.7
75.3
97.0
95.5
87.6
55.3
61.6
78.3
46.9
54.5
70.1
+
MissForest
77.3
86.1
70.4
96.6
97.3
92.2
58.4
83.6
52.3
75.9
4.04
1.18
Hyperimpute
77.3
88.6
70.8
94.4
98.1
92.6
X
80.6
95.5
86.1
51.2
77.2
3.50 + 2.42
TAPTAP-distill
77.3
87.0
72.5
93.8
97.6
93.4
58.5
79.2
93.6
73.8
52.5
84.2
2.83 + 1.03
TAPTAP
77.3
87.0
73.1
93.8
97.4
93.2
58.6
81.2
97.0
77.4
52.6
81.7
2.46 = 1.34
Table 21: The experimental results in missing value imputation. "+ M-Ori" means training with the original data processed by the MAR mechanism. "+ M-Ori + Synthetic Data" means training with the M-Ori data where the missing values are imputed by different models. Below the backbone model Transformer. denotes the method cannot run successfully on the dataset due to too many missing values.
Metric
L.
AD
HE
CR
DI
CA
GE
ME
AG
DU
Avg. Rank
Transformer M-Ori
76.0
86.2
72.2
93.7
58.4
77.9
93.6
$1.9
72.2
Transformer
M-Ori Synthttie Data by Models
MIWAE
76.8
85.6
69.6
96.9
91.6
X
77.8
92.5
49.0
51.7
69.9
6.33 + 1.42
GAIN
76.4
85.0
74.3
97.4
96.0
87.5
55.3
63.4
79.3
48.1
54.3
71.0
5.83 1: 3.01
Sinkhom
76.6
85.1
69.8
93.7
96.6
91.6
58.4
77.1
93.3
48.3
51.7
74.9
5.67
+ 1.30
MICE
76.8
86.1
70.0
93.7
96.4
91.3
58.3
77.6
93.5
63.2
51.8
88.6
5.17 + 1.70
MissForest
76.9
86.3
70.6
96.7
97.7
91.9
58.4
78.7
92.7
84.9
51.6
70.9
4.08
1 1.78
Hyperimputc
76.4
88.8
70.1
94.3
92.4
81.0
95.8
87.7
50.4
76.0
3.96
+ 2.60
TAPTAP-distll
77.0
86.9
72.2
93.8
98.5
92.9
58.5
79.6
93.8
76.4
52.1
80.2
2.58 +: 1.16
TAPTAP
76.8
86.9
73.0
93.8
97.7
93.0
58.6
81.7
97.1
79.0
51.8
71.1
2.38 + 1.33

---

Original
CTGAN
TVAB
300
300
8 Sample
200
# Sampps
200
Sample
200
100
100
100
0
10
15
20
10
15
20
10
15
20
DCR
DCR
DCR
CopulaGAN
GReaT
TabGen
300
300
Samples es
200
# Samples
# Samples
200
200
100
100
100
0
5
10 DCR
15
20
10
15
20
10
15
20
DCR
DCR
Figure 7: Distance to closest record (DCR) distribution the California Housing dataset. "Original" denotes the DCR of the original test set with respect the original train set. The experimental results illustrate that each method does not copy samples from the train set.
Original
CTGAN
TVAE
200
150
Sumples
# Samples
#t Samples
100
100
100
50
100 DCR
150
200
0
50
100
150
200
%
100
150
200
CopulaGAN
GReaT
TabGen
200
200
# Samples
Somples
# Sumples
100
100
100
0
50
100 DCR
150
200
0
50
100
150
200
%
50
100
150
200
Figure 8: Distance to closest record (DCR) distribution of the HELOC dataset.


Reviews:
[{"review_id": "75wuvVlovU", "reviewer": "Reviewer_PTdH", "summary": "", "questions": "- Question A: In the experiment of missing value prediction (Table 4, Table 17-21), some methods failed due to massive missing values. However, TAPTAP seems robust due to the permutation function (L306-L323). Does the permutation function somehow a data augmentation module, or what it does is just shuffling the data?\n- Question B: In section 3.5, how is the percentage of data prompt decided? Is it data-specific, or it depends on which task is solving? For example, use (1) Feature name as prompt for data privacy tasks.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review!\n\n> Question A: In the experiment of missing value prediction (Table 4, Table 17-21), some methods failed due to massive missing values. However, TAPTAP seems robust due to the permutation function (L306-L323). Does the permutation function somehow a data augmentation module, or what it does is just shuffling the data?\n\nThe permutation function serves as a kind of data augmentation in TAPTAP. Each time a training sample is feed into the model, the order of its features is randomly shuffled. Through this process, the permutation function reconstructs the feature order independence. Consequently, TAPTAP can flexibly sample from any provided feature-value pairs in a robust way, and thus perform well in the missing value imputation scenario.\n\n\n> Question B: In section 3.5, how is the percentage of data prompt decided? Is it data-specific, or it depends on which task is solving? For example, use (1) Feature name as prompt for data privacy tasks.\n\nThe selection of data prompting strategy depends on the characteristics of the task at hand (i.e., the quantity of feature values that need to be sampled), and hence is task-dependent rather than data-specific. For example, tasks such as privacy protection and low resource regimes necessitate the sampling of all feature values, leading to the adoption of the 'feature-name-as-prompt' strategy. Conversely, the task of missing value imputation requires the sampling of absent feature values, given the presence of known feature values. As a result, it employs the 'multiple-feature-value-pairs-as-prompt' strategy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Four challenging tasks, especially Privacy Protection and Missing Data Imputation, are crucial in medical data engineering as patient data cannot be shared across hospitals. In addition, the legacy software in the hospital often makes ETL a barrier to introducing grounded AI in this field. Hence, a method for generating high-quality training data like this work is worth exploring.\n2. The authors provide a comprehensive report and implementation details (section 3.3) on applying the proposed method for resolving tasks (1)-(4), including using different backbone models (i.e., MLP, LightBGM, and Transformer) on 12 datasets that fairly report the pros and cons of baselines and TAPTAP. The reproducibility of this work is good.", "reasons_to_reject": "1. Though mentioned in L86-L91, linking the purpose of tabular prediction and the experiments takes a lot of work in the first pass due to the intense information in this paper. Not until L212-L214, it's hard to catch the main contribution of TAPTAP is to improve the quality of training data, which is helpful for tasks (1)-(4).\n", "questions_to_authors": "- Question A: In the experiment of missing value prediction (Table 4, Table 17-21), some methods failed due to massive missing values. However, TAPTAP seems robust due to the permutation function (L306-L323). Does the permutation function somehow a data augmentation module, or what it does is just shuffling the data?\n- Question B: In section 3.5, how is the percentage of data prompt decided? Is it data-specific, or it depends on which task is solving? For example, use (1) Feature name as prompt for data privacy tasks.\n", "paper_topic_and_main_contributions": "The authors proposed TAPTAP(Table Pretraining for Tabular Prediction),  a method that aims to improve the quality of the training data for tabular prediction. After pretraining language model in the public tabular datasets via static prompt (e.g., <column> is <value>), fine-tuning LM, and data sampling, the resulting synthetic data can help resolve four common challenges: (1) Privacy Protection, (2) Low Resource Regime, (3) Missing Value Imputation, and (4) Imbalanced Classification. The experimental results show that the proposed TAPTAP and TAPTAP-distill significantly improve the performance of tasks (1)-(4) over 12 publicly available datasets.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f9eJiWd6ei", "reviewer": "Reviewer_YvRN", "summary": "", "questions": "A. See the above Reason to Reject 1.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review!\n\n> It lacks a qualitative analysis of the quality of the synthesized table examples. The tabular prediction tasks require learning a distribution across the feature values and labels, but how can you guarantee that the synthesized feature values in the augmented tables are beneficial rather than noisy for the learning of downstream models?\n\nWe measure the quality of the synthesized samples by their performance in the application scenarios. There is substantial evidence in our experiments that the synthesized feature values are not just noise but substantive contributors to the learning performance of downstream models. For example, on half of the privacy protection datasets, models trained with synthetic data delivered performance similar with those trained using the original data. Furthermore, when additional synthetic features (or samples) generated by TAPTAP are integrated into the training data, the machine learning models show great improvements, outperforming those trained with original datasets in the missing value and imbalanced classification scenarios.\n\n> Some details are unclear in the main body: the exact base model for TAPTAP, the description of TAPTAP-distill, etc.\n\nThank you for pointing it out. TAPTAP uses the original GPT2 [1] with 355M parameters, while TAPTAP-distill uses the distilled version of GPT2 [2] with 82M parameters. We will include this information in the experimental setup section in the revised version.\n\n[1] Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners[J]. OpenAI blog, 2019, 1(8): 9.\n\n[2] Sanh V, Debut L, Chaumond J, et al. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter[J/OL]. CoRR, 2019, abs/1910.01108. http://arxiv.org/abs/1910.01108.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed data generator model is novel and has good generalizability to various tabular prediction tasks.\n2. The design of the proposed method, including input encoding and data augmentation, provides useful empirical insights that may benefit the research in this field.\n3. The experiments are thorough, showing the effectiveness of the proposed method in a wide range of scenarios.", "reasons_to_reject": "1. It lacks a qualitative analysis of the quality of the synthesized table examples. The tabular prediction tasks require learning a distribution across the feature values and labels, but how can you guarantee that the synthesized feature values in the augmented tables are beneficial rather than noisy for the learning of downstream models? ", "questions_to_authors": "A. See the above Reason to Reject 1.", "paper_topic_and_main_contributions": "Table pretraining has been well-explored to solve the joint reasoning of tables and text, but its effectiveness for tabular prediction tasks has not been verified. This paper presents the first successful trial of table pretraining for tabular prediction by proposing a TAPTAP framework to enhance downstream tabular prediction tasks by synthesizing high-quality table examples. They collected a large-scale pretraining corpus to enable the pretraining, and systematically evaluated the framework on 12 datasets across four downstream task scenarios: privacy protection, low-resource regime, missing value imputation and imbalanced classification. Experimental results have shown the benefits of the generative pretrained model for different backbone models (e.g., GBDT models or Transformer which are trained for downstream tasks).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Some details are unclear in the main body: the exact base model for TAPTAP, the description of TAPTAP-distill, etc. "}, "extra_text": {}}, {"review_id": "HoJBPCGrdx", "reviewer": "Reviewer_2Qac", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review!\n\n> This paper sort of overclaims its contribution as \"the first attempt to apply table pre-training to tabular prediction\". There were many attempts in tabular pre-training to improve tabular prediction [1,2,3,4,5]. In my opinion, the major contribution of this paper lies in the cross-table generative pre-training and then generates synthetic tabular data for the target task to augment the downstream machine learning prediction models.\n\nWe sincerely appreciate the reviewer’s thoughtful comments and critique. We agree that there were other previous studies that performed tabular pre-training. The major difference between TAPTAP and previous studies on tabular pre-training is that, TAPTAP performs cross-table pre-training using a large number of tables on language models to leverage the knowledge embedded in language models, and previous works (such as TabNet, VIME, and SCARF) usually perform single-table pre-training (or few tables with lots of overlapped columns, such as TransTab) on models specifically designed for tabular data. There are other studies that perform cross-table pre-training on language models (such as TaPaS, TaBERT, and TaPEx) for joint reasoning over text and tables (such as the TableQA task), yet TAPTAP is the first attempt to apply cross-table pre-training on language models for the tabular prediction task.\n\nWe will refine the claim for better clarity in the revised version according to your suggestions. We will also include an extended discussion on the differences between TAPTAP and the tabular pre-training studies referenced by the reviewer in the revised version of our paper.\n\n> The position of this paper seems to be suboptimal. While the paper emphasizes its capability in tabular prediction, the main strength of this paper, in my perspective, lies in its tabular data generation capability. From the observation made in Tables 3 and 5, the lift obtained by augmentation tabular data seems to be marginal over the other baselines. By contrast, in Table 2, it is witnessed that the proposed method reaches the lowest performance gap between the model trained on real data and trained on synthetic data, showing its power in generating synthetic data. It is suggested the author dives deeper into its generation capability and perform more in-depth of the fidelity, utility, and privacy of the synthetic data.\n\nWe greatly appreciate your keen observations about the positioning of our paper in the context of its strength in tabular data generation. We agree that the main strength of this paper is the tabular data generation capability. We determine the current positioning of this paper because a key purpose of tabular data modeling is to boost the performance of tabular prediction, and the merits of tabular data generation are substantiated by its role in improving the performance of tabular prediction in different challenging scenarios (such as privacy protection, missing value imputation, etc). We will follow your suggestions and include these discussions in the revised version.\n\nWhile the marginal lift over baselines in the data augmentation experiments may not seem impressive, we note that improving performance with synthetic data is a challenging problem. Despite this difficulty, TAPTAP still manages to outperform other state-of-the-art techniques, demonstrating its capabilities in synthesizing useful tabular data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors conducted a large-scale experiment over a bundle of 450 tabular datasets collected from OpenML and Kaggle. It would benefit the community if the authors could offer this data for future research in tabular learning.\n\n- The TapTap method that has been proposed can be used for pre-training across various tables and fine-tuned to generate for targeting tabular data. The setting is reasonable and brings the promise of copying the success of generative pre-training on text to on tabular datasets.", "reasons_to_reject": "- This paper sort of overclaims its contribution as \"the first attempt to apply table pre-training to tabular prediction\". There were many attempts in tabular pre-training to improve tabular prediction [1,2,3,4,5]. In my opinion, the major contribution of this paper lies in the cross-table generative pre-training and then generates synthetic tabular data for the target task to augment the downstream machine learning prediction models.\n\n- The position of this paper seems to be suboptimal. While the paper emphasizes its capability in tabular prediction, the main strength of this paper, in my perspective, lies in its tabular data generation capability. From the observation made in Tables 3 and 5, the lift obtained by augmentation tabular data seems to be marginal over the other baselines. By contrast, in Table 2, it is witnessed that the proposed method reaches the lowest performance gap between the model trained on real data and trained on synthetic data, showing its power in generating synthetic data. It is suggested the author dives deeper into its generation capability and perform more in-depth of the fidelity, utility, and privacy of the synthetic data.\n\n\n---\n\n[1] Sercan O Arık and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, volume 35, pages 6679–6687, 2021.\n\n[2] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. VIME: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033–11043, 2020.\n\n[3] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. SCARF: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022.\n\n[4] Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. SubTab: Subsetting features of tabular data for self-supervised representation learning. Advances in Neural Information Processing Systems, 34, 2021.\n\n[5] Wang Z, Sun J. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 2022, 35: 2902-2915.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The target of this paper is to perform generative pre-training over a bunch of tabular datasets and then synthesize tabular data for downstream tasks. The synthetic data are then labeled by a classifier trained on the real data. Subsequently, the labeled synthetic data and the real data are combined to train a final classifier for the downstream task. \n\nThe authors emphasize the utility of this data synthesis approach in the following aspects:\n\n- Privacy-preserving tabular prediction: replacing the real data with the synthesized data to build the predictive ML models.\n\n- Few-shot tabular prediction: augmenting the real data to boost the predictive ML models in low-data scenarios.\n\n- Missing data imputation: the generative pre-training model can help impute the missing values in the target table.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper titled "TAPTAP: Generating High-Quality Training Data for Tabular Prediction" presents a method called TAPTAP for improving the quality of training data for tabular prediction tasks. The proposed method involves pretraining a language model on public tabular datasets using static prompts, fine-tuning the language model, and data sampling. The resulting synthetic data helps address four common challenges in tabular prediction: privacy protection, low resource regime, missing value imputation, and imbalanced classification. The paper presents experimental results on 12 publicly available datasets that demonstrate the effectiveness of the proposed method in improving the performance of these tasks. Overall, the reviewers think that the work has done well in this task, and also has some concerns about the writing, claim. For example, the position of this paper has been questioned by one reviewer, some contribution might be overclaimed, and more experimental explanations and discussions are needed, too.
Decision: Accept-Main