Language Models are Realistic Tabular Data Generators
Authors: Vadim Borisov
Kathrin Sessler
Tobias Leemann
Martin Pawelczyk
Gjergji Kasneci
Venue: ICLR 2023 poster (2023)
Keywords: tabular data
tabular data generation
large language models
transformers
probabilistic modeling
deep neural networks

Abstract: Tabular data is among the oldest and most ubiquitous forms of data. However, the generation of synthetic samples with the original data’s characteristics remains a significant challenge for tabular data. While many generative models from the computer vision domain, such as variational autoencoders or generative adversarial networks, have been adapted for tabular data generation, less research has been directed towards recent transformer-based large language models (LLMs), which are also generative in nature. To this end, we propose GReaT (Generation of Realistic Tabular data), which exploits an auto-regressive generative LLM to sample synthetic and yet highly realistic tabular data. Furthermore, GReaT can model tabular data distributions by conditioning on any subset of features; the remaining features are sampled without additional overhead. We demonstrate the effectiveness of the proposed approach in a series of experiments that quantify the validity and quality of the produced data samples from multiple angles. We find that GReaT maintains state-of-the-art performance across numerous real-world and synthetic data sets with heterogeneous feature types coming in various sizes.

Published as a conference paper at ICLR 2023
LANGUAGE MODELS ARE REALISTIC TABULAR DATA GENERATORS
Vadim Borisov!; Kathrin Scßler"; Leemann', Martin Pawelczyk', Gjergji Kasneci! University of Tübingen, Tübingen, Germany
Technical University of Munich, Munich, Germany
ABSTRACT
Tabular data is among the oldest and most ubiquitous forms of data. However, the generation of synthetic samples with the original data's characteristics remains a significant challenge for tabular data. While many generative models from the computer vision domain, such as variational autoencoders or generative adver- sarial networks, have been adapted for tabular data generation, less research has been directed towards recent transformer-baser large language models (LLMs), which are also generative in nature. To this end, we propose G GReaT (Generation of Realistic Tabular data), which exploits an auto-regressive generative LLM to sample synthetic and yet highly realistic tabular data. Furthermore, GReaT can model tabular data distributions by conditioning on any subset features; the re- maining features are sampled without additional overhead. We demonstrate the effectiveness of the proposed approach in series of experiments that quantify the validity and quality of the produced data samples from multiple angles. We find that GReaT maintains state-of-ther -htt performance across numerous real-world and synthetic data sets with heterogeneous feature types coming in various sizes.
INTRODUCTION
Tabular data is one of the most common forms of data in machine learning (ML) over 65% of data sets in the Google Dataset Search platform contain tabular files in either CSV XLSS formats (Ben- jelloun et al., 2020). However, due to the expensive nature of data collection processes, tabular data sets (i) are often class imbalanced, (i.e., tabular data sets tend to have long-tailed label distributions (Cao et al., 2019)), (ii) contain critical person-related information and cannot be shared due to pri- vacy protection or socio-ethical principles (Gascón et al., 2017), and (iii) often come with impurity issues such as noisy or missing values which impede the application of modern ML algorithms (Lin & Tsai, 2020). Synthetically generated data has the potential to alleviate these three important is- sues. Therefore, the generation of realistic artificial tabular data has received considerable attention in recent years (Choi et al., 2017; Park et al., 2018; Xu et al., 2019; Borisov et al., 2021).
Apart from real-world impurity issues, there also exist various technical problems that make the generation of synthetic data difficult. Typically, tabular data contains various feature types, such as categorical features (c.g., name, count countryoffrrgiin jobTitle) and numerical features age, income). ). The categorical variables (i.c., words or clauses) may frequently contain most of the information. For example, the highly used Adult Income data set consists of seven numerical and eight categorical variables (Dua & Graff, 2017). This heterogeneity of feature types and values leads to three core challenges in tabular data preprocessing and modeling:
features (c.g.,
Extensive and lossy preprocessing. For most of the existing tabular data generation methods, extensive data preprocessing of tabular data required, which usually includes the following steps: (i) categorical data encoding into numbers, (ii) data scaling normalization, (iii) replacing missing values, and (iv) removing outliers and smoothing. These data transformation steps may result in the loss of important information the introduction of artifacts that not present in the original data. As an example, the categorical encoding into numeric values may introduce an artificial ordering into
"Equal contribution
Corresponding authors: wewtariinsssseerreeeee tum.de,
@uni-tuchingen.de

---

Published as a conference paper at ICLR 2023
Original
TVAE
CopulaGAN
18.
Latirode
36-
35.
-125.0
-22.5 -13000 -1755 -1150
-
125.
-1255
13000
1175 -1500
-125.
-1225
-100 -1775 -1150 Longitude
Longitude
Longitude
CTGAN
Distill-RReaT (ours)
GReaT (ours)
42
Latinde
34
34-
34-
-12500 -12555
-1200 -1175
-1150
125. 122.5
-1300 Longitude
-1755
-1500
-125. -1224
-1300 -1755
-1150
Longitude
Longitude
Figure 1: A comparison of the original and generated samples for the California Housing data set (Pace & Barry, 1997), which contains characteristic information about different properties in California, USA. We show joint histogram plots of the highly interconnected variables Latis ude and Longitude. The black outline indicates the true boundary of the state of California.
previously unordered values (Borisov al., 2021). Therefore, the problem of lossy preprocessing may strongly influence the quality of generated data result.
Context knowledge for coherent semantics. Almost all common synthetic data generation meth- ods transform tabular data into fully numerical representation. However, tabular data sets fre- quently consist of variables that are contextually interconnected. In the Adult Income data set (Dua & Graff, 2017), the features age, marital- saarital-statte atuuss, and education ion have clear coherence relationship: There is a certain minimal legal age for marriage, and it challenging to get Ph.D. young age. Such context knowledge should ideally be considered when generating realistic synthetic data samples. We refer to this common issue as the contextual knowledge problem.
Arbitrary conditioning. A versatile model that can generate data for large variety of applications should be able to synthesize data conditioned on an arbitrary set of variables. This allows impu- tation of any missingness pattern in the data oversampling of arbitrary subsets. Currently, the majority of the methods do not provide extensions to arbitrary conditioning and require the gener- ative model to be re-trained according each specific set of features to be conditioned on (Mirza & Osindero, 2014). We refer to generators that allow for conditional generation with any specified feature combination as supporting arbitrary conditioning.
Most modern deep-learnnng approaches for tabular data generation build on generative models trans- ferred from the computer vision domain (Borisov et al., 2021), such as Variational Autoencoders (VAEs, Kingma & Welling, 2013) or Generative Adversarial Networks (GANs, Goodfellow et al., 2014). However, deep learning models have equally revolutionized the field of natural language pro- cessing (NLP, Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020). Modem large language models (LLMs) are often constructed in the form of ruto-regressive density models over sequences of words (Radford et al., 2019; Bengio et al., 2000). This begs the question which extent success- ful architectures for NLP are apt to the tabular data generation task.
Carrying this thought further, present novel method for probabilistic data generation that covers the outlined core challenges and results in state-of-the- performance (see Fig. for a qualitative example). We argue that pretrained seff-attention-based L.MSs (Vaswani et al., 2017) suitable
2

---

Published as a conference paper at ICLR 2023
for the probabilistic modeling of heterogeneous tabular data sets after these data sets have been ap- propriately transformed into a textual representation. We do so by constructing syntactically correct sentences based on feature names and row values without losing information or introducing artificial orderings and thus mitigate the issue of lossy preprocessing. This step, to which we refer as textual encoding, maintains substantially more information than usual transformations. Since we include the variable names in the encoding, a model trained on this data can directly access contextual infor- mation. To be able make sense of this information, we suggest using established and pretrained language models to perform the generation task. This could be possible path towards tackling the zontextualization problem. Finally, we introduce a feature order permutation step to shuffle the feature order in the textual encodings. Training model on such data will result in versatile gen- erator that supports arbitrary conditioning. Specifically, our work offers the following contributions relative to the existing literature on the generation of synthetic tabular data:
Novel paradigm. We propose the first approach for realistic heterogeneous tabular data modeling and generation utilizing a transformer-decoder network architecture. Thereby, we connect tabular and textual data modalities via textual encoding scheme.
Arbitrary conditioning. When trained on textual encodings with random feature order permutations, our model inherits its arbitrary conditioning power from the LLM, which can model the data distribution conditioned on any given subset of features and sample the remaining features.
Extensive experimental results. We show that our Generation of Realistic Tabular Data (GReaT) obtains statc-of-the-an generative performance on a variety of data sets across several measures. We have open-sourced our experimental results, making them available as strong benchmarks for the benefit of the community.
Python package. Finally, we provide an easy-to-use Python implementation of the GReaT model, where takes only three lines of code to generate new synthetic samples. Access to the package is provided via pip install be -eegrett.
RELATED WORK
While the generation of images and text is extensively explored (Karras et al., 2020; Subramanian al., 2017), the generation of synthetic tabular is less commonly considered in the recent machine learning literature. Classical methods for tabular data modeling include Bayesian networks, in par- ticular those based on the Chow-Liu approximation (Chow & Liu, 1968) or statistical tools such as copulas (Kamthe et al., 2021). More recent methods for probabilistic tabular data modeling utilize generative adversarial networks (Choi al., 2017: Park al., 2018; Mottini al., 2018; Xu al., 2019; Koivu et al., 2020) or variational autoencoders (Xu et al., 2019; Ma et al., 2020; Vardhan & Kok, 2020; Darabi Flor, 2021).
The mixed structure of discrete and continuous features along with their different value distributions still poses significant challenge. CTGAN, the current state-of-the- approach by Xu al. (2019), 1)),, places special focus on the conditional distributions between the features to generate semantically meaningful data. For non-Gaussian feature distributions, the authors propose a mode mode-specific nor- malization technique. However, the one-hot-encoding scheme used to encode the modes and the categorical features in this work can aggravate the "curse of dimensionality" problem (Bellman, 1966) in the presence of high-cardinality variables. Furthermore, it does not profit from contextual information. We thoroughly compare our method to their approach in the experimental evaluation section.
In parallel development, the area of natural language processing was dominated by recurrent neural networks (RNNs) before self-attention-basee neural networks (Vaswani et al., 2017) revolutionized this field. Based on their technique, various auto-encoding models (Devlin et al., 2018; Sanh et al., models (Raffel
2019; Lan et al., 2019) for tasks like sentence classification, et al., 2020; Lewis et al., 2020) for translation or summarizing, and auto-regressive models (Radford al., 2019; Brown et al., 2020) for natural language generation tasks showed the strength of the self-attention mechanism.
'https:// hht
3

---

Published as a conference paper at ICLR 2023
In the light of these successes, transformer-basee models have also been devised for tabular data classification (Arik & Pfister, 2019; Somepalli et al., 2021; Kossen et al., 2021) and learning joint representations of tabular and textual data (Yin et al., 2020). Padhi et al. (2021) investigated the generation of multi-variate time series data, using a transformer architecture based on BERT (Devlin al., 2018) 8). However, there no prior works generating highly realistic non-sequential tabular data with the help of attention-basee LLMs. Our work is the first to rigorously explore this path, which leads to state-of-the-ar performance.
Orignal lobular dala set
'Age is 39, Education is Bachelors, Occupation is Adm-clerical,
Age
Education
Occupaton
Gender
Income
Gender is Male, Income is s SOK."
G Gender Female, HS-grad, Occupation Exec-managerial
"Age
19
Bachelors
Admicierical
Male
5OK
2 SOK...
H5-grad
E Fxec-managerial
Female
"Age Education Occupation Handler-cleaners,
Bachelors
P Profispeciaity
Female
Gender Female,
50K."
(b)
[B08]
"Eduration is Bachelors, Income is s 50K, Age is 39, Occupation is Adm-clerical, G Gender Male.". "Income 50K, Occupation Exec-managerial. Age 50, Education HS-grad, Gender Female.", "Occupation Handler-deaners, Education is 11th, Age is 53, Income is 50K. Gender Female."
Pretrained Generative Large Language Model
(c)
[s00]
The GReaT data pipeline for the fine-tuning step. First, a textual encoding step transforms tabular data into meaningful text (a). Subsequently, feature order permutation step applied (b), before the obtained sentences can be used for the fine-tuning of large language model LLM (c). The toy tabular data set inspired by the Adult Income data set (Dua & Graff, 2017).
Figure 2:
3 GREAT: GENERATION OF REALISTIC TABULAR DATA
This section presents the GReaT approach for fully-conditional tabular data generation using transformer-based neural networks. GReaT consists of two major stages: (1) the fine-tuning of a pretrained large language model (LLM) and (2) sampling from the fine-tuned LLM to generate synthetic tabular data. We illustrate the sam- pling procedure in Fig. 3. In the following, we describe each component of the fine-tuning and sampling steps in detail. We conclude this section with a brief summary of our approach.
textually encoded tabular data set shown in Fig.
3.1 GREAT FINE-TUNING
Textual encoding. Standard pretrained generative LLMs expect sequences of words as inputs. Hence, to apply an LLM on tabular data, we have to convert each row of our data set into a tex- tual representation. To this end, propose the textual encoder.
Definition (Textual encoder) Given a tabular data set of m columns with feature names fi. f' and rows of samples ..,..., we let the entry Vijj E (l,...,u,,5 E {1...., m) represent the value of the j-th feature of the i-th data point. Taking the feature name and value into account, each sample s, of the table is transformed into textual representation t, using the following subject-prediee objecc transformation:
li.j [J}r "is" Va.js 4.]
Vi (1,..,n)::) (1, m), Vi E {1, n],
t; [1,,1, li,,,... Limm
(2)
where ti.g, the textually encoded feature, is clause with information about a single value and its corresponding feature name, and denotes the concatenation operator.
Remark For the scope of this work, we treat the target variable as a regular feature in our formu- lations.
4

---

Published as a conference paper at ICLR 2023
Inpur sequences (Arhitrary condilloring)
Age'
[. Age is 26,")
(A)
Telenbor
Finetuned Pretrained Generative
Large Language Model
Education is Masters, Age 59,"
(
[e0s]
De-takeriizer
(b)
yynhhet: tabular data set
'Age 23, Occupation is Adm-clerical, Income is s 50K, Gender Missing, Education 11th." "Age is Income S0K, Occupation Sales, Education HS-grad, Genderis Fernaee
Age
Education
Occupation
Gender
Income
(c)
23
11th
Adm-clerical
Missing
50K
HS-ggaed
Sales
Femele
50K
"E "Education is Mastars, Age 59, Occupation Other-service,
Otherssaricce
50K
Gender Male, Income 50K"
Figure 3: The sampling procedure of the proposed method for synthetic data generation. In order to generate new data points using pretrained LLM, it is necessary to transform either single feature name or an arbitrary combination of feature-value pairs into text (a). Subsequently. the input is completed by the fine-tuned LLM (b) and be transformed back into tabular format (c). In comparison to other state-of- off-----tthtrt-- approaches, GRea'T allows arbitrary conditioning on feature subsets without model retraining, i.e., the sampling can be performed by conditioning on any feature name or combination of feature names and values.
Fig. 2 (upper panels) shows illustrative example of the proposed encoding technique. This result could be a sequence like "Occupation is doctor, Gender is female, Age is 34,", which requires min- imal preprocessing and does not suffer from information loss. While in standard natural language sequences the word order is crucial, in our case there is no preferred order between the individual features.
Random feature order permutation. By transforming a tabular feature vector into a sequence using the textual phjject-predicate-ohjeett encoding scheme, pseudo-positional information is artifi- cially introduced into the transformed tabular data sample. However, naturally there is no spacial ordering relationship between features in tabular data sets (Borisov et al., 2021). To reconstruct the feature order independence, we randomly permute the encoded short sentences ti,j of the full textual representation &; by permutations k.
Definition (Feature order permutation function) Formally, the result of applying a random feature order permulation k to tú where each ki E {1...., m) is arbitrary and kj # liz for j jr, defined as (kk) [fr,k k..,,...., tg Vi E nt....,n.
When using shuffled orders of the textually encoded features, we fine-tune our generative language model on samples without order dependencies. Moreover, using permutations of the above form is highly beneficial as they allow for arbitrary conditioning in tabular data generation, which is further discussed in the following Subsection 3.2.
Fine-tuning pretrained ruto-regressive language model. Finally, we describe the fine-tuning procedure of pretrained LLM to the encoded tabular data for generation tasks. We suppose tex- tually encoded tabular data set T = :=1.......... that was transformed into text by the proposed encoding scheme. Let k; be randomly drawn permutations and n denote the number of rows.
To be processed with an LLM, the input sentences t E T need to be encoded into sequence of tokens from a discrete and finite vocabulary W. These tokens can be character, word or subword (Sennrich et al., 2015). Thus, t T is represented by
encodings such the a sequence of tokens (wi,..., myy) TOKENIZE(t) with tokens wi, ........ ,W, E W, where j denotes the number of tokens required to describe the character sequence t.. Commonly, the probability of natural- natural-language sequences is factorized in an auto-regressive manner in LLMs (Jelinck, 1980; Bengio al., 2000). It represented product of output probabilities conditioned on previously observed tokens,
p(t) = pl wi....... ...., w3) =
pl (wye W1, ..., Wk-1)-
(3)
5

---

Published as a conference paper at ICLR 2023
In this formulation, it becomes evident that LLMs formally need to be highly capable predictors for follow-up tokens given an arbitrary-length sequence of preceding tokens. Indeed, the model is trained to output probability distribution over possible next tokens we from an input sequence wi,.-,,kk1 of arbitrary length. The entire model usually fitted by optimizing the parameters to maximize the probability Iht p(t) of the entire training data set.
As a result, an end-user can choose any existing generative language model for tabular data mod- eling and exploit the vast amount of contextual knowledge present in these models (Roberts al., 2020). For instance, in generative transfomer-decoder LLM architectures (e.g., the GPT models (Radford et al., 2018: 2019; Brown et al., 2020)) word-embeddings are obtained from large corpus of text (e.g.. GPT3 model trained on 45TB of textual data (Brown et al., 2020)). By learning from such large collection of data, Transformers are able to build robust contextualized representations of language (Liu et al., 2021). Fine-tuning enables the model to leverage this contextual information in combination with the feature and category names to boost the models capabilities manner that is similar to transfer learning by learning bidirectional representations (Raffel et al., 2020).
3.2 GREAT SAMPLING SYNTHETIC DATA
Having obtained fine-tuned auto-regressive model of the textual training data set that returns categorical output distribution z q( [[W...... wk-1) over possible follow up tokens for an input sequence wi...., Wk-1, we can apply several sampling strategies. Usually, the next token w is sampled by weighted choice sampling with temperature parameter T > 0 from the output of the LLM,
e
p(w|w1... wk-1) Ew'ew /T=
(4)
We note that the auto-regressive paradigm offers the possibility of sampling from token distributions p( ():/)a)) with arbitrary preconditioning Wick- When we use random feature order permuta- tions at train time, we can also start the textual sequence with any possible combination of features and values at inference time. As a result, the GReaT method is particularly flexible and could pos- sibly be used
variety of real-world problems such as missing value imputation (Kachuee al., 2020) or generation of realistic counterfactual explanations (Pawelezyk et al., 2020). Moreover, the sampling of the conditional distribution comes at no additional cost.
Sampling and extraction of synthetic tabular data. Therefore, the setup provides several ways to sample new tabular data points using the GReaT method. We initialize the model with certain conditions and let the LLM sample the remaining tokens complete the feature vector (in its textual representation). In total, we propose three options of preconditioning:
Feature Name Preconditioning, Only a feature name, but no value is provided as an initial condition. This type of conditioning is able to generate samples from the entire joint data distribution pVV...., Vv), where V denotes the random variables representing the n features in the data set.
Name- Value Pair Preconditioning, In this case, single feature name and value are provided. Starting from this input, GReaT will complete the sample. This approach will generate samples from the distribution p( (A) IV; v.). Because modeling a single feature distribution is usually tractable (by frequentist estimation for categorical features or by fitting parametric density estimator for continuous features). we can first sample value for V. and then apply Name ValueValue Pair Preconditioning to sample the remaining features. Thereby, we can also sample the entire data distribution.
Multiple Name- Pair Preconditioning. We also provide multiple Name-Vale pairs =i, By providing the textual encoding of this condition, we are able to sample from the distri- bution of the remaining features effectively ((V( (V, ,%,,............................ V,, ).
=%, as precondition to the LLM to realize arbitrary conditioning.
We illustrate the GReaT sampling possibilities in Fig. 3 and Fig. that underline the high flexibility of the proposed approach for synthetic data generation. We apply commonly accepted pattern- matching algorithms using regular expressions to convert the generated textual feature representa- tions back tabular format (Aho, 1991). We dismiss the respective samples in the rare case where

---

Published as a conference paper at ICLR 2023
Octginal Test
TVAE
CTGAN
CopulaGAN
GReaT
Samples
Summplee
Samplee
Sampler
ree
Samples
ve
Samples
10 DCR
10 DCR
DCR
DCR
DCX
DCC
Figure 4: Distance closest record (DCR) distributions for the California Housing data set with respect to the original train set. "Original Test Data Set" shows the DCR between the original test set and the original train set. This experiment shows that the proposed method does not "copy" samples from the training set but rather generates new synthetic samples close to the original samples.
the required format is violated. Generation rates of invalid samples were monitored and found to be consistently below 1 %. The main cause of this rare event lies in the generation procedure of the pre-trained LLM, which chooses the next token based on a probability distribution. For example, the Occupatin in variable in the Adult dataset has specific support, e.g., ["Adm-clerical", "Exec- managerial", "Prof-specialty"...
}. However, rare cases, the pre-trained LLM might leave this support and output, for example, 'Adm clerical' (without dash), or mix up these categories and return 'Adm-mana aamm-managerial" To lower the chances of this unwanted effect, we suggest reducing the sampling temperature T (Eq. 4), which sharpens the probability distribution. In practice, simple validation function can quickly reject those infrequently invalid samples.
A brief summary of the strengths of the GReaT method. GReaT comes with several significant advantages over related approaches: It (i) allows the end-user to have full probabilistic control over the sampling procedure by its arbitrary conditioning power; it (ii) utilizes the knowledge from large text data bases to obtain better representation that includes context knowledge; (iii) the proposed approach is easy use, since does not require a preprocessing of the data values. There no need to specify discrete or numerical variables beforehand and the information loss due to preprocessing therefore kept at its bare minimum.
4 EXPERIMENTAL EVALUATION
In this section, we empirically demonstrate the performance of the proposed GReaT approach us- ing multiple qualitative and quantitative experiments. Lastly, for better reproducibility, we provide information on the packages and parameters for the selected LLMs.
Data sets. For the evaluation of the proposed algorithm, we utilize six real-world data sets that come from various domains. They also come in different sizes, reaching from less than 1,000 to over 100,000 samples. We also consider three synthetic data sets with varying numbers of features. Key characteristics of each data set are presented in Table 7. We split all data sets into 80% train and 20% test sets to avoid any data leakage. All models trained fine-tuned the same training data samples. To demonstrate the power of our synthetic data generation framework to work out-of- the-box with real-world data, we apply zero data preprocessing, e.g., feature names and values used as they are presented in original data sets. In contrast, other baselines require extensive data preprocessing.
Baselines. As baselines for our experiments, we use three deep learning-based methods. CT- GAN (Xu et al., 2019) is based on a generative adversarial network (GAN) (Goodfellow al., 2014) for tabular data that allows to condition the generation process only a single discrete feature. The same authors proposed TVAE (Xu et al., 2019). a variational autoencoder (VAE) for tabular data. The CopulaGAN model from the Synthetic Data Vault (SDV) framework (Patki et al., 2016) is ap- plying the Gaussian copulas to simplify the underlying CTGAN. We analyze the main properties of the chosen baselines in Tab. 10.
We compare those baselines to the proposed method with two different pretrained transformer- decoder LLM models of various sizes. The smaller, distilled version of GPT-2 (Sanh et al., 2019) has 82 million learned parameters. We term the corresponding tabular data generator Distill-GReaT. An original version of GPT-2 by Radford et al. (2019) has over 355 million trainable parameters we refer to this version simply as GReaT. A description of the architecture of the selected generative L.MMs can be found in Table 8. We apply name-value pair preconditioning start sampling.
7

---

Published as a conference paper at ICLR 2023
For the evaluation of synthetic tabular data, select four measures, all of which have been featured in multiple previous studies on synthetic tabular data generation (Borisov et al., 2021).
Machine learning efficiency (MLE). Since the generated data set should be able to replace the real data in a training process, this measure evaluates the performance of discriminative models trained on synthetic data sets. Therefore, the models are tested real test data, and the scores are compared to the original performance when the models were trained on the original, real training data set. Table and Table present the results for the machine learning efficiency of proposed generative models compared to the baseline models. To be independent of the concrete discriminative model, we evaluated the synthetic data sets using Linear/Logistic Regression (LR). Decision Tree (DT) and Random Forest (RF) (Ho, 1995). We observe that either GReaT and Distill- -eeaT outperform all competitors and entail considerable performance improvements.
Dataset
Original
TVAF
CopulaGAN
CTGAN
Distill-RReaT
GReaT
LR
82.72.0.00
79.5810.00
73.30.10.00
73.30:00.00
78.53:1..00
80.10:10.00
Travel (f)
89.01
816811125
73.6140:26
73.30:00.00
77.38:1051
83.56:00422
RF
85.0310.53
816681119
73.30.0.00
71.4120.53
79.90.10.33
84,30
LR
96.69:0.00
04.70::0.00
94.57:10.00
94.44200.00
96.56-10.00
97.72::0.23
Sick (1)
DT
98.94+00.29
95. 05.39:t00.5 18
93.77=00.01
92.05==0.41
95, 95.39-40:44
97.72:00233
RF
99.28:00066
94.91:10.06
94.57:0.01
94,57::0.00
97.72-10.10
98.30:00.13
LR
71.80=0.00
71.04+00.00
42.03-0.000
57.72=00.00
70.58::0.00
71.90=:0.00
HELLOC (1)
DYT
81. 900000022
76.39++0.00 10
42. 12.36-+0.10
61.34+00.99
81.40-00.15
79. 10++007
RF
83.1920.21
77.24:0:225
42.35:20.44
62.35:20.35
82.14:10.33
80.93:0.25
LR
85.00000.00
80.53-+0.00
80.61:00.00
83.20:1:000
84.65-10.000
84.77
Adult Income (1)
DT
85.27=0001
00
76.29+00006
81,32++0.02
84.49++0.04
RF
85.9340.11
83.48820.1
80.46::0.21
83.53
85.25:10.07
85.42:
LR
58.76.10.00
56.34110.00
40.27.10.00
50.93
57.33-10.00
$7.34:10.00
Diabetes (1)
DT 57.29:0.03
53.30::0.09
08.50
49.73=00.2
54,10:10.01
55.23::0.04
RF
59.00
55.17:10.10
37.59.0.31
52.23
58.03-00.16
58.34:.0.09
I.R
0.40+0000
0.65++0.00
0
0.61+0000
0.57-0.00
0,34-0.00
Californa Housing (1)
DT
0.32:0.01
0.45=0001
1.19=0.01
0.82:0001
0.43=0001
0.39:00001
RF
0.21+0m
0.35-+0.11
0.99-0.01
0.62+0001
0.32-0001
0,28-0001
Table I: ML efficiency experiment. The best results are marked in bold, the second bbot results underlined. Six real-world data sets used, Travel Customers, Sick. HELOC, Adult Income, Diabetes, and California Housing. Each data set is evaluated on three discriminative ML models, Linear//Logistee Regression (LR), Decision Tree (DT), and Random Forest (RF). For classification tasks the accuracy score is reported, in the case of regression the mean squared error is used. Results are averages over five trials with different random seeds (cf. Appendix B for additional measures).
CopulaGANN
TVAE
CTGAN
Distill-GReaT
GReaT
Travel
18.1910.33
72.80.1..00
71.26.10.00
85.8410.43
78.68.00.66
Sick
92.32.1..16
99.76.10.03
99.99
13.0010.24
63.77.1..1
HELOCC
97.83+0.12 12
100.00+00.00
99.99+0.01
68.30-10.47
69.15-+0.36
Adult Income
88,54=00.99
88,49+00.8
97,23++0.10 10
69.79--011 17
62.84:0008
Diabetes
99.85
98.31
17.8410.14
99.82100.01
0..10
California Housing
85.48-+0.16 16
85.0444.27
82.98++0.27
18
70.68-+0.30
Average
90.37=10:03
90.73+0.03
89.88+00033
78.82+0005
69.57=0.05
Table 2: Discriminator measure (accuracy in %). Lower accuracy values indicate that the discrim- inator cannot differentiate between fake records and real samples. The accuracy of perfectly in- distinguishable data set would be 50%. Best results are bold, second-best results are underlined. Results are averages over five trials with different random seeds.
Distance to closest records (DCR) histogram. To verify that the generated data is similar to original samples while not being exact copies, this measure computes the distance to the clos- est record in original training data set Ttrain- For each synthetic record 8yen. it is given by DCR(s = min Distanccesssss of the differences. We set the difference to be 0 for equal for categorical features and to I other- wise. In the best case, all DCR scores are non-zero and their distribution is close to that of DCRs computed with points from the original test data set Trest- Visualizations of the distribution of the minimal distances can be found in Fig. 4 and Appendix B. All utilized models generated unseen samples in close proximity to the originals, and no significant difference was observed.
ss)[s; E ]. As a distance measure, we use the -n-rom
8

---

Published as a conference paper at ICLR 2023
Adult
HELOC
Calfomia
Travel
Distill-RReaT
discr. (1)
MLB (t)
discr(())
(1)
diser. (L)
MLE
discn(()
MLE (1)
w/o permutation
61.18=0.16
85.71
79.18:0.23
81.97
76.47:0.26
0.261:001
61.62:10.67
83.77.00.33
with permutation
69.77=0.09
85,25=0007
68.29:0.34
82.14=0.13
76.09=0.14
0.33+0001
85,6 85,65:+0.38
80.31=0.53
w/o pretraining
99. 1400
84,15:00005
99.51#0001
76.32=0:20
85. 10#0.22
0.34
62.98:00.1
81,47:0.42
with pretraining
69.77 0.09
85.25
68.29:00.34
82.14:0.13
76.09
0.33
85.65-40.88
80.31=0053
Table 3: Results of experiments with and without permutation, as well as with and without pre- training based on discrimination (diser.) and ML efficiency (MLE, measured by the accuracy of a Random Forest model) on four real-world data sets. In all experiments, we used Geeill with the same training and pretraining setup; the only difference is the input data and pretraining. Results are averages over five trials with different random seeds.
Discriminator measure. To check whether the generated data cannot be easily told apart from the original data, we train a Random Forest discriminator (with hyperparameter tuning) on a mix of the generated train set (with label 0) and the original train set (with label 1). We then report the test accuracy on a test data set, which contains equal shares of samples from the generated test set and the real test set. Scores are shown in Table 2 and demonstrate the superior performance of the GReaT approach, which on average (across all used data sets) decreases the discriminator performance by 16.2 % over other competitive baselines for tabular data generation.
Bivariate joint distribution plots. We qualitatively compare the generated feature distributions by the baselines and the GReaT approach to that of the original data. As an illustrative example, present joint density plots for the Longi tude tude and Lat tuuee features in the California Housing data set in Fig. 1. While CTGAN and CopulaGAN fail to model the strong dependency between these variables (indicated by out-of-distriuttion samples having both high latitude and longitude or low values for both), Destill-GReaT and GReaT yield densities that align well with the ground truth boundaries. TVAE shows mediocre performance but still places density mass outside the bounds.
Effects of pretraining and permutations. Having obtained impressive results with the complete setup, we investigate the role of the individual components towards its success. We compare the full model (with permutation and pretraining step) to a model without permutation and a model without pretraining. Tab. 3 presents MI efficiency and discriminator results for the modified models. On all but the very small "Travel" data set, we observe pretraining to help boost the performance. This might be due to the context knowledge made available to GReaT through the extensive adaptation to large text corpora. The results regarding the feature order permutation step are mixed - the MLE performance decreases with permutations, but the results for the discriminator metric are inconclu- sive. With permutations, the learning problem is undoubtedly more demanding. It provides models with the ability to perform generation based on arbitrary conditioning. However, we conjecture that it might also increase performance in some cases hecause it does not introduce any possibly unnatural, fixed feature ordering into the tabular data.
CONCLUSION
In our recent work, investigate how state-of-the-art generative language models can be leveraged to synthesize highly realistic tabular data samples. Instead of following the usual path of encod- ing heterogeneous tabular data in numerical format, we devise a textual encoding strategy and represent it with sentences that capture each record's semantics. The resulting transformeer-eccoder network fine-tuned on this data exhibits unprecedented generative performance and outstanding flex- ibility at the same time. We term our method Generation of Realistic Tabular data (GReaT). GReaT unites several remarkable characteristics that address key problems in tabular data modeling: First, minimal preprocessing is required, which is efficient and results in the least possible information loss, thereby tackling the issue of possibly lossy preprocessing. Second, leveraging random feature order permutations, we exploit the capability of arbitrary conditioning and are thus equipped with full control over the probabilistic sampling procedure for tabular data. Finally, through pretraining, incorporate contextual and semantic knowledge extracted from terabytes of textual data for a more authentic tabular data synthesis. In conclusion, we see our work as a door opener leading to yet-undiscovered possibilities in the domain of heterogeneous data generation.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Various critical applications rely on heterogeneous tabular data (c.g., healthcare, finance, and so forth) and would profit from free data sharing. However, this always requires ethics and privacy considerations. The present work proposes a novel synthetic tabular data generation approach based on pretrained large language models. Before sharing any data (including data generated by the proposed method), the authors thus strongly encourage possible owners of proprietary data sets to verify that reverse identification impossible is prevented by regulatory means. Other than that, sec further ethical questions related to this work.
REFERENCES
Alfred V Aho. Algorithms for finding patterns in strings, handbook of theoretical computer science (vol. a): algorithms and complexity, 1991.
Serean O Arik and Tomas Pfister.
TabNet:
Attentive
interpretable
tabular learning-
2019.
Samuel Assefa, Danial Dervovie, Mahmoud Mahfouz, Robert Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on AI in Finance, pp. 1-8, 2020.
Richard Bellman. Dynamic programming. Science, 153(3731):34-37, 1966.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. neural probabilistic language model. Advances in neural information processing systems (NeurIPS), 13, 2000.
Omar Benjelloun, Shiyu Chen, and Natasha Noy. Google dataset search by the numbers. In Inter- national Semantic Web Conference, pp. 667-682. Springer, 2020.
Vadim Borisov, Tobias Leemann, Kathrin SeBler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:21100018 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 13:1877-1901, 2020.
Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake Van- derPlas, Arnau Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Min- ing and Machine Learning, pp. 108 122, 2013.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-nnn margin loss. Advances in neural information processing systems (NeurIPS), 32, 2019.
Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. Gen- erating Multi-label Discrete Patient Records using Generative Adversarial Networks. Machine learning for healthcare conference, pp. 286-305, 2017.
CKCN Chow and Cong Liu. Approximating discrete probability distributions with dependence trees. IEEE transactions Information Theory, 14(3):462-467, 1968.
Sajad Darabi and Yotam Flor. Synthesising multi-modal minority samples for tabular data. arXiv preprint arXiv:2 105.0000000 105.080044 2021.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18.0.04800000 2018.
10

---

Published as a conference paper at ICLR 2023
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Shashank Rajput, Michael Gira, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-intern interfaced fine-tuning for non- language machine learning tasks. arXiv preprint arXiv:2206.0665 06500, 2022.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httt:://archive
ics. wui.cccee edduml.
Adrià Gascón, Phillipp Schoppmann, Borja Balle, Mariana Raykova, Jack Doerner, Samee Za- hur, and David Evans. Privacy-preserving distributed linear regression on high-dimensional data. Proc. Priv. Enhancing Technol., 2017(4)3345538) 364, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Mikel Hernande7, Gorka Epelde, Ane Alherdi, Rodrigo Cilla, and Debbie Rankin. Synthetic data generation for tabular health records: systematic review. Neurocomputing, 2022.
Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume pp. 278-282. IEEE, 1995.
Frederick Jelinek. Interpolated estimation of markov source parameters from sparse data, In Proc. Workshop Pattern Recognition in Practice, 1980, 1980.
Mohammad Kachuee, Kimmo Karkkainen, Orpaz Goldstein, Sajad Darabi, and Majid Sarrafzadeh. Generative imputation and stochastic prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Sanket Kamthe, Samuel Assefa, and Marc Deisenroth. Copula flows for synthetic data generation. arXiv preprint 100.002,,,,, 2021.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/VVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8110-8119, 2020.
Diederik P Kingma and Max Welling, arXiv:1312.611,, 2013.
Auto-encoding variational bayes
arXiv preprint
Aki Koivu, Mikko Sairanen, Antti Airola, and Tapio Pahikkala. Synthetic minority oversampling of vital statisties data with generative adversarial networks. Journal of the American Medical Informatics Association, 27611:11677-167, 2020.
Jannik Kossen, Neil Band, Clare Lyle, Aidan Gomez, Tom Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In Advances in Neural Information Processing Systems, 2021.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1909.1199.. 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequens pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871 7871-7880, 2020.
Wei-Chao Lin and Chih-Fong Tsai. Missing value imputation: a review and analysis of the literature (2006-2017). Artificial Intelligence Review, 53(2):1487-159) 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.1358. 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1711011711.05101. 2017.
11

---

Published as a conference paper at ICLR 2023
Chao Ma, Sebastian Tschiatschek, Richard Turner, José Miguel Hemándeez-obato, and Cheng Zhang. Vaem: a deep generative model for heterogeneous mixed type data. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets.
arXiv preprint
arXiv: 17784, 2014.
Alejandro Mottini, Alix Lheritier, and Rodrigo Acuna-Agost. Airline Passenger Name Record Gen- eration using Generative Adversarial Networks. arXiv preprint urXiv:1807.06657. 2018.
R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33 (3):291-297, 1997.
Inkit Padhi, Yair Schiff, Igor Meinyk, Mattia Rigotti, Youssef Mrouch, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3569. IEEE, 2021.
Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Young- min Kim. Data synthesis based generative adversarial networks. Proceedings of the VLDB Endowment, 11110)1071-1083, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bus E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8033 Curran Associates, Inc., 2019.
N. Patki, R. Wedge, and K. Veeramachaneni. The synthetic data vault. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 390-410, Oct 2016. doi: 10. 1109/DSAA.2016.49
Martin Pawelczyk, Klaus Broelemann, and Gjergji Kasneci. Learning model-agnostic counterfactual explanations for tabular data. In Proceedings of The Web Conference 2020, pp. 3126-3132, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(14)):166) 2020.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.089/00, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistiBBERT, distilled version of BERT: smaller, faster, cheaper and lighter. In NeurlPS EMC² Workshop, 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 100.09990000999, 2015.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-traine pre-training. arXiv preprint arXiv:2106.01344 2021.
Beata Strack, Jonathan P DeShazo, Chris Gennings, Juan L Olmo, Sebastian Ventura, Krzysztof J Cios, and John Clore. Impact of hbale measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. BioMed research international, 2014, 2014.
12

---

Published as a conference paper at ICLR 2023
Sandeep Subramanian, Sai Rajeswar, Francis Dutil, Christopher Pal, and Aaron Courville. Adver- sarial generation of natural language. Proceedings of the 2nd Workshop on Representation Learning for NLP. pp. 241-251, 2017.
Vivek Harsha Vardhan and Stanley Kok. Generating ing oblivious variational autoencoders. In Proceedings of the Workshop on Economics of Privacy and Data Labor at the 37 th International Conference on Machine Learning (ICML), 2020.
synthetic tabular data us-
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems (NeurIPS), 30, 2017.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do NLP models know numbers? probing numeracy in embeddings. In Empirical Methods Natural Language Pro- cessing, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of the-art art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38-388455 45, 2020.
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2019.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguisties, pp. 8413-8426. Association for Computational Linguistics, July 2020.
A DISCUSSION
Processing numerical values with LLMs in the GReaT approach. Since we convert hetero- geneous tabular data into text (Sec. 3), continuous and discrete numerical values are represented as character sequences. While this might seem counter-intuitive at first, multiple independent studies capable of understanding and processing numerical
have shown that transformer-basee models data encoded such a way (Wallace et al., 2019; Brown et al., 2020). This is even true in the one- shot setting for inference tasks (Dinh al., 2022). The impressive performance of GRea'T that en- compasses the numerical features aligns with these previous observations. Having said that, smarter encodings for numerical values also be considered a possible path further improvement.
Unification of multiple modalities through transformer models. Along with the usual numerical values, tabular data frequently contains textual metadata, e.g., feature names, named categories ("male", "female"). and open text features (e.g., remarks). The recent evolution of transformer neural networks now permits to holistically unite this information, which was processed separately in the past, and learn context-specific, robust, and meaningful representations. In the case of tabular data, information comes from the textual and numerical data modalities. The GReaT approach and the results presented in this work provide an initial clue of the range of possibilities and opportunities that may lie this line of research.
B ADDITIONAL EXPERIMENTAL RESULTS
B. FURTHER MLE MEASURES
Next to the accuracy measures (Tab. 1) we also report the ROCAUC score and the score for the MI. efficiency experiment (Sec. 4).
13

---

Published as a conference paper at ICLR 2023
INAE
CqpilAAN
CIGGN
GReat
ROCALI
F1
ROCAIIC
F1
ROCALC
F1
ROCADC
F1
ROCAUI
F1
ROCALI
F1
30:14-010
28.03100
12.00:000
71300010
S2.00
KFF
04 IS2102
$4.69.3944 66
60.25 49
51.561831
1200000000
666421.12
85.372135
$7,901022
79.2929445
LR
N6:40:00:00 00
94.94:3006 88
93 356550000
77-766500
01 100000000
75:5=3300
91 20
05.25 60
93.47:550M
S1
66-952000
53533000
50
95.72-0155
71,25
91.506000
97.37
1RR
TM E3:0000
71
77:00000 80
77 17:00-0000
26
T1 1100
43:00:000
61 M0:0000
00:71-000
57 070000000
TT 081000
70 00:00:00
71407:008
DT
89 57-0044
81
01
19
17:011
36-000
81.40-4155
RF
00.520013
03.15
25
35.60:535
05.34
02.12:9133
1.X
00 18.000
64.55..300 80
87.1520000
$1 1L.R
$1.92:010
57.35.010
65, 19.10100M1
6460:1100
AT
DT
64.31:00:0 51
6466328344
$2 TRLDon
7141:21M
2622211444
$4.4::016
50. 607621042
83.5755055
RF
91
85:004
21:11:010
08.73500 00
14:000
77:55:0119
11
55.47:0010 16
00000000
$1.57
64.85-0844
65:022:000
5461-10.0
00.35
52.9020000
50:2:000
70:000
57.24-310
48, 48.45000000
54.18:900
64:19:0000
54.22:100
DF
52
91 1002
53.55.05 0.00
56 14
31 12
12.0003
55.52 03.
47. 10.49.007
TUUS
56 01
61.27 1043
53.16 11982
KF
85 1005
53.87.0300 30
00.07.1125
32. 11
40.051027
50.41
$1.09.0.14
40.3 40.20/007
53 23.001 1000017
KA75 1815
54.2011415
Table 4: Additional results of the ML efficiency experiments measuring the ROCAUC score and the Fl score for the five real-world classification data sets, Travel Customers (TR), Sick (SI). HELOC (HE), Adult Income (AD) and Diabetes (DI). Each data set is evaluated on three discriminative ML models, errear//Logistie Regression (LR), Decision Tree (DT)), and Random Forest (RF). The best results are marked in bold, the second-best results are underlined. Results are averages over five trials with different random seeds.
Original Test Data Set
TVAE
CTGAN
200
210
200
# Samples 50-
1:50
# Samples
1:50
Samples
1:50
100
100
50-
50-
1.5
150
100
200
DCR
DCR
DCR
CopulaGAN
Distil-GReaT
GReaT
200
200
200
Samples
# Samples
# Samples
1500
150
150-
100
100
100
0
50
100
130
200
0
50
100
150
200
50
100
150
200
DCR
DCR
DCR
Figure 5: Distance to closest record (DCR) distributions for the HELOC Data set with respect to the original train set. "Original Test Data Set" shows the DCR between the original test set and the original train set. According to this experiment, the proposed method does not "copy" samples from the training set but rather generates new synthetic samples close to the original samples.
B.2 AVERAGE NEGATIVE LOG-LIKELIHOOD METRIC FOR SYNTHETIC DATA
The generated data should be likely under the training data's 's distribution. Therefore, we generate three common synthetic data sets, and compute both the likelihood of the sampled data under the training data's density (. This can however be prone to overfitting, which is why we addi- tionally deploy the likelihood fitness metric proposed by Xu et al. (2019) (Lrnat). To compute this metric, a parametric density model (BNs and GMMs respectively) is fitted to the generated data. Then the likelihood of the original test samples is computed. The results in Table indicate that LLMs are comparable with state-of-the-an deep neural networks when modeling high -ige-dimensional mixture distributions.
B.3 DISTANCE TO CLOSEST RECORD RESULTS
We compare the distribution of the minimal distances of the generated samples to the training data set. Figure 4 shows the distribution for the California Housing data set and Figure 5 for the HELOC
14

---

Published as a conference paper at ICLR 2023
GMM
Asia (BN)
Alarm (BN)
Model
Layn
Liest
Layn
Crest
Layn
Ltest
Identity
-4.403
-4,403
-2.265
-2.265
-11.739
-11.739
CopulaGAN
-4.406
-4.676
-6.321
-3.129
-22.438
-16.828
TVAE
-4.205
-4.817
-2.418
-2.289
-11.919
-12.074
CTGAN
-4.427
-4.723
-3.999
-2.528
-20.804
-15.248
Distill-GReaT
-4.372
-5.124
-1.925
-2.355
-6.941
-12.689
Table 5: Average Log-likelinne ligeliie of synthetic data samples density model derived from the original data (Ley)) and of the original test data on the model derived from the syntethic data (Ltest)- The best results are marked in bold, the second-best results underlined.
Original
CopulaGAN
14
14
mmm
17
edlccation num
69
20
80
20
80
10
CTGAN
Dittll
GReaT
15.0
12.5
edlcationn Jarcation.umm muin
ediccatinn num
000
10.0
7.5
75
0.0
Figure 6: comparison of the original and generated data sets for the Adult Income data set using joint plots for two related features Age and eeuucationNun
data set. The results indicate that the generated samples are close to the original ones without coping them exactly.
B.4 ADDITIONAL QUALITATIVE ANALYSIS RESULTS
In Figure 6 we additionally compared the joint feature distribution of two exemplary features from the Adult Income data set, Age and Educat onNum. The joint plots computed using a kernel density estimator.
B.5 RUNTIME COMPARISON
We also analyse the training/fine-tuning and sampling time between baseline models and two versions of the GReaT method using only two GPUs (Sec. C). Table summarizes our time- benchmarking results. To make fair comparison, corresponding implementations. Also, we utilize the same DL framework PyTorch (Paszke et al., 2019), and the same number of epochs 200 as well as the sampling size 1000, for each model.
used the latest available versions of the
15

---

Published as a conference paper at ICLR 2023
While the proposed model requires significantly time for the fine-tuning step compared to other baselines, Distill-GReaT and GReaT models demonstrate top performance for the vast majority of benchmarks across various datasets 1Tab.1, Tab.2, Tab. 4). Also, there critical applications such as healthcare (Hernandez et al., 2022) or finance (Assefa et al., 2020), where high quality synthetic tabular data is required, regardless of the possible computation cost, therefore in experiments introduced two data sets from medical machine learning field (Tab. 8). Lastly, with more compu- tation resources, the proposed model's sampling and fine-tuning times can be drastically decreased.
TVAE
CopulaGAN
CTGAN
Distill-GReaT
GReaT
training/ fine-tuning time
0:46 min
2:30 min
1:10 min
1:35
9:10 h
sampling time
0.28 sec
0.119 sec
0.045 see
4 sec
17 sec
Table 6: run time comparison of all generative models of our study. Selected models were trained/fine-tuun for 100 epochs and 1000 samples were generated.
Datasel
Domain
#Samples
#Num
#Cal
Task
#Classes
(TR)
Travel Customers
Churn
954
2
4
Clussification
2
(SI)
Sick
Medical
3772
22
Classification
2
(HE)
HELOC
Financial
9,871
21
2
Classification
(AD)
Adult Income
Social
32,561
8
Classification
(DI)
Diabetes
Medical
101,766
39
Multi-Class
3
(CH)
California Housing
Real Estate
20,640
8
0
Regression
Alarm
Synthetic
20,000
0
37
Asia
Synthetic
20,000
8
GMM
Synthetic 6,000
2
0
Table Details of the real-world and synthetic data used in the experimental evaluations. #Num and #Cat columns indicate numbers of numerical and categorical features in each data set.
#Parameters
#Layers
#Heads
Embedding Size
Context Size
Distill GPT-2
82M
12
768
1024
GPT-2
355M
24
1024
1024
Table 8: Structural details about the pretrained large language models used in study.
C
REPRODUCIBILITY DETAILS
Reproducibility details. We utilize pretrained generative language models from the established HuggingFace framework (Wolf et al., 2020). Its routines also used for the fine-tuning and sam- pling steps. We open-sourced our implementation.
We fine-tune the Distill-GReaTT model for each data set for 200 epochs, except for the Califor- nia housing and Diabetes (Strack al.. 2014) data sets, for them, we fine-tune them for 100 epochs. The GReaT baseline fine-tuned for 110, 310, 400, 255, 150, 85, epochs for Cal- ifornia Housing, Adult Income, Travel, Home Equity Line of Credit (HELOC), Sick (Dua & Graff, 2017), and Diabetes data sets, respec- the GPU memory limi-
I.R
DT
RF
max Iter
max depth
max depths
estimators
Travel Castrmers
100
HELOC LCC
500
12
Adult Income
1000
12
85
Diabetes
500
California Housing
Figure 7: The hyperparameter configuration of the evaluation models for the MI. efficiency experi-
tively. Depending tations, we vary the batch size from 8 to 124. For the sampling step, set the temperature
ments.
16

---

Published as a conference paper at ICLR 2023
Travel Customers
intpst//www keggl con/ de seet i't geshvii 4/2 pur-trave wtw.weewleeeeeeeee custoner- ohunn- pred ct
Sick
httpss/ http///wpppppp. opennnpaann.... ozg/ astp//earchhtyye-aaaaaaa http:///raaa.tttooooooooeaaaaaaa nestive
HELOC
hctpo:/ /www. sagglo. celleo.ccooiii www./oo///nteeeeooe dtt.d/aaattttateeeoooaddaats /t/.
-onoa- 1ino-of cfcrreithhlooe
Adult Income
bi pu 2/ /arehiiiiie www.oo.......................... wdd///ldd///ll /at/n/////aaae a a/Adols?
Diabetes
1et hi.kpt http://a..aa...coooooo//ttaaaa diabeles readniasion prediot Jon
California Housing
https:/ https////////..// httpss//www..asptt.ccccaattttttt http:///wweegeeeeec/eeooceeeeeee htwpaaooooosooottteoooee
Table 9: URLs for real-world data sets of the study.
parameter T 0.7 for all experiments and data sets. We sample new synthetic data using the name- value pair preconditioning (Sec. 3), starting with the target feature for all data sets (see an example in the supplementary materials).
We utilize the A optimizer (Loshchilov & Hutter, 2017) for the proposed generative models, with the learning rate 5 x 10-5 We plan to share trained weights for the GReaT models. The baseline models are trained for 200 epochs for each data set.
Our hardware setup consisted of NVIDIA 2080RTX GPUs with 12 GB RAM each, 126 GB system RAM, and AMD Ryzen 3960X with 24 cores, we use the Ubuntu 20.04 operation system.
For the ML efficiency and discriminator experiments (Sec, 4) additionally linear/logistic regression, decision tree, and random forest models from the Scikit-Learn package (Buitinck et al., 2013), we report the exact hyperparameters for each model in Table 7. For the discriminator measure experiment (Table 2), we tune hyperparameters for each data set using the 5-fold cross- coss-valliation.
Besides providing important information for the experiment reproducibility, we also share the syn- thetic data sets generated by our GReaT method.
Single Variable
Multi-Variable Conditional Sampling
Usage Context (Variuble Names)
Transfer Leurning
Data
Conditional Sampling
Prepossessing
CopulaCiAN TVAE
X
Scaling, Encoding
X
Scaling, Encoding
CTGAN
X
Scaling. Encoding
GReaT
/
Format conversion
Table 10: An analysis of the main properties of the synthetic tabular data generation frameworks
17

---

Published as a conference paper at ICLR 2023
Age
value
prob.
ageiimnome eOOk edocation HS HS-asooo)
0.05
33
0.03
Income
48
0.03
value
prob.
s50K
Iecomeeelssatioon HS schac
>50K
0.26
Aaggoincome 50K, oducation (HS-scnoo))
value
prob.
Education
0.07
value
prob.
58
HS-school
0.17
41
0.04
Some-collage
0.14
Bachelors
0.12
Afacoeeelccatonn oi Besiverra)
value
Leseelacomee sducation Bagheiess
0.07
value
prob.
40
<50K
>50K
0.49
Plggliacone HaK oducation in Bechebus)
59
0.04
value
prob.
0.17
69
0.15
61
0.11
Figure 8: Example of the arbitrary conditioning using the GReaT approach Adult data set. For this experiment, we select only three variables Education, Income, and Age from the Adult Income data set (Dua & Graff, 2017). However, the proposed method be scaled to the arbitrary number of conditions. The results obtaining by changing the input textual sequence, e.g., Education is HS-school, Income is >50K, Age is, after we obtain the ditional discriminate distribution o(Agelincome => 50K, education = HS school). Inter- estingly, the GReaT method successfully learned that there is only two options for the Income variable. The arbitrary sampling is supported by our Python programming framework.
18


Reviews:
[{"review_id": "sMaU8WOibW", "reviewer": "Reviewer_gUA4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the use of large-scale autoregressive generative language models (LLMs) for generating tabular data. Several recent works have similarly attempted the task of generating tabular data using popular computer vision methods such as Variational Autoencoders and Generative Adversarial Networks, but require an initial preprocessing step of encoding the categorical columns and (optionally) transforming the numerical columns, which often have a large impact on the performance of the generative model. In this paper, a different approach is taken by using a transformer-decoder network, a popular architecture in the NLP field. In this approach, the columns are first converted to text using a textual encoding scheme, the feature order is permuted to allow arbitrary conditioning and sampling, and fine-tuning is performed using a pretrained generative LLM.", "summary_of_the_review": "The proposed method is a breath of fresh air compared to previous work on GANs and VAEs for generating tabular data. Transformer architectures are increasingly dominating various fields, and this approach seems promising for future deep-learning-based synthesis of tabular data, but how to protect the privacy of the data synthesised from such models still needs to be studied in depth.", "strength_and_weaknesses": "Strengths:\n- The proposed approach is easier to handle because it removes the bottleneck of the first preprocessing step for the numeric and categorical columns \n- The textual subject-predicate-subject encoding scheme, together with the random permutation of the feature order, provides the end user with full probabilistic control over the sampling procedure of the generative model\n- The paper shows significant improvements over the state of the art evaluated with the proposed methods\n\nWeakness:\n- Lack of related works. The authors claim that there are no previous works using pre-trained LLM to generate tabular data. However, a quick Google search revealed that there are some recent works (2021) that have considered a similar architecture for generating tabular data, including time series data. Granted, the pre-processing steps are different and this work is new in that regard. Regardless, the authors should highlight such related work and differentiate their work or justify the omission \n- Bad samples. The authors state in Section 3.2 that some synthesised samples that violate the resulting patterns for tabular data have been discarded, but fail to provide an explanation of why such a violation occurs or provide examples of such samples to guide the reader's imagination\n- Practicality: the proposed methods require significantly longer fine-tuning (~9hrs) compared to other baseline methods (< 2mins). The authors have not commented on this or justified why such a computationally intensive approach is preferable\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n1tsm_ybZX", "reviewer": "Reviewer_HxcB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "summary_of_the_review": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "strength_and_weaknesses": "Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kdBelh5BfCZ", "reviewer": "Reviewer_7WMZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel approach to generate synthetic tabular data using large language models. ", "summary_of_the_review": "The authors did find a novel application of large language models. However, it's unclear if the gains of the proposed method can justify the 100x increase in the computation cost.\n", "strength_and_weaknesses": "Strengths:\n\nThe authors proposed an interesting way to use large language models. It's a novel use case.\n\nWeaknesses:\n\n1. The computation cost for the proposed approach seems profitably expensive. In table 6, both GReaT and Distill-GReaT required two order of magnitudes bigger compute budget comparing to other baselines. This leads to serious scaling concerns. \n\n2. Have the authors consider using few-shot evaluation from the large language models as the alternative way to generate synthetic tabular data? It would be cheaper than fine tuning such a big model using hundreds of epoches.  With proper prompting, large language models should be able to generate desired outputs.\n\n3. I am not sure the significance of the proposed method. It's not clearly to me the significance of the deltas in those measures reported in the experiment section. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3IDVS75cHH", "reviewer": "Reviewer_EaHN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes GReaT (Generation of Realistic Tabular Data), a large language model fine-tuned to generate samples for tabular datasets. First, GReaT encodes examples in a tabular dataset into sequences of text, then GReaT fine tunes a large pretrained language model using the textual representation of the tabular data. Once tuned, GReaT can generate entirely new examples, or sample missing values using arbitrary conditioning.\n\nExperiments on 4 tabular datasets (3 classification and 1 regression) demonstrate GReaT generally outperforms existing tabular data generators on a wide and comprehensive range of evaluation metrics.", "summary_of_the_review": "GReaT leverages large language models to generate samples for tabular data in a way that avoids many of the lossy data preprocessing steps required by existing approaches, resulting in high-quality contextually-relevant tabular examples. Overall, the number of datasets evaluated is rather low, however, the positives currently slightly outweigh the negatives.", "strength_and_weaknesses": "Strengths\n---\nGReaT operates directly on the tabular dataset, avoiding lossy data preprocessing steps such as 1) encoding of categorical data (e.g., one-hot encodings); 2) data scaling or normalization; 3) missing value imputation; and 4) outlier removal. These data preprocessing methods can often result in loss of information and ultimately reduce the quality of generated samples.\n\nGReaT is a simple and novel approach for generating samples for tabular datasets, the most common type of data used in industry. This is a potentially significant tool that can help combat noisy or missing values, or can be used to balance imbalanced datasets.\n\nGReaT fine-tunes a large language model using random feature order permutation, allowing sampling with arbitrary conditioning. This also allows GReaT to leverage the vast amount of contextual knowledge learned during pretraining of the large language model to better generate appropriate values for certain features given the values of other features.\n\nIn comparison with existing methods, the experimental results suggest GReaT generates samples that 1) better substitute as training data, 2) are less discernible with the actual training and test data, and 3) qualitatively produce more appropriate values than existing methods.\n\nWeaknesses\n---\nExperiments are run on just 4 rather small tabular datasets, limiting claims of GReaT's generalizability.\n\nThe datasets used contain a relatively small number of features; thus, I am unsure of how GReaT would perform for much higher-dimensional tabular datasets resulting in significantly longer input sequences for fine-tuning.\n\nOne of the motivations for GReaT is the fact that tabular datasets may contain sensitive or personal information, but wouldn't a high-quality data generator potentially reproduce some of this personal data since it is trained on that information?\n\nMinor Weaknesses\n---\nWhat is the train-test split percentage mentioned in the \"Data sets.\" paragraph in Section 4?\n\nIn Figure 3, the occupation \"Adm-clerical\" in the first row of the bottom-left table does not match the first input sequence of the textual representation where the occupation is \"Transport-moving\".\n\nIn the paragraph \"Baselines.\" in Section 4, \"...on a only...\" -> \"...on only a...\".\n\nTable 1 caption, \"...in case of the regression...\" -> \"...in the case of regression...\".\n\nIn the paragraph \"Baselines.\" in Section 4, \"...an variational autoencoder...\" -> \"...a variational autoencoder...\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster