CTSyn: A Foundation Model for Cross Tabular Data Generation
Authors: Xiaofeng Lin
Chenheng Xu
Matthew Yang
Guang Cheng
Venue: ICLR 2025 Poster (2025)
Keywords: Foundation Model
Tabular Data
Synthetic Data Generation
Primary area: generative models

Abstract: Generative Foundation Models (GFMs) have achieved remarkable success in producing high-quality synthetic data for images and text. However, their application to tabular data presents significant challenges due to the heterogeneous nature of table features. Current cross-table learning frameworks struggle because they lack a generative model backbone and an effective mechanism to decode heterogeneous feature values. To address these challenges, we propose the Cross-Table Synthesizer (CTSyn), a diffusion-based generative foundation model for tabular data generation. CTSyn comprises two key components. The first is an autoencoder network that consolidates diverse tables into a unified latent space. It dynamically reconstructs table values using a table schema embedding, allowing adaptation to heterogeneous datasets. The second is a conditional latent diffusion model that generates samples from the learned latent space, conditioned on the table schema. Through large-scale pre-training, CTSyn outperforms existing table synthesizers on standard benchmarks in both utility and diversity.  These results position CTSyn as a promising framework for synthetic table generation and lay the groundwork for developing large-scale tabular foundation models.

Published as a conference paper at ICLR 2025
CTSYN: A FOUNDATION MODEL FOR CROSS TABU- LAR DATA GENERATION
Xiaofeng Lin', Chenheng Xu', Matthew Yang", Guang Cheng'
'Department of Statistics and Data Science, University of California Los Angeles CA, USA Departmennt of Computer Science, University of California Los Angeles, CA, USA
[bernardo1998, chenhengx0101, ymatt24, hugp:///ow.....t....naanneeeneee guangchengJeucla.ee.
ABSTRACT
Generative Foundation Models (GFMs) have achieved remarkable success in pro- ducing high-quality synthetic data for images and text. However, their application tabular data presents significant challenges due to the heterogeneous nature of table features. Current cross-table learning frameworks struggle because they lack a generative model backbone and an effective mechanism to decode hetero- geneous feature values. To address these challenges, we propose the Cross-Table generative foundation model for tabular
Synthesizer (CTSyn), a diffusion- data generation. CTSyn comprises two key components. The first is autoen- coder network that consolidates diverse tables into a unified latent space. dy- namically reconstructs table values using table schema embedding, allowing adaptation to heterogeneous datasets. The second is conditional latent diffusion model that generates samples from the learned latent space, conditioned on the ta- ble schema. Through large-scale pre-training, CTSyn outperforms existing table synthesizers on standard benchmarks in both utility and diversity. These results position CTSyn as promising framework for synthetic table generation and lay the groundwork for developing large-scale tabular foundation models.
INTRODUCTION
Generative Foundation Models (GFMs) have revolutionized fields such as Computer Vision (CV) and Natural Language Processing (NL (oommaaaa et al.) 2021 He et al.) [I 2023 Touvron et al. 2023 Ramesh et al. 2022; Rombach et al. (022). Trained on vast datasets (Merity ct Deng al. 2009 Schuhmann et al. 2022) and with versatile model backbones /Vaswani 2017 Ho et al. 2020). these models excel across a diverse range of domains and tasks. They generate valuable synthetic training examples to enhance the performance of various down-
et al.
stream applications (Kirillov et al. 2023 Li et al. 2023b Moor et al. 2023 Trabucco et al. 2023
Zhang al. 2023a).
GFMs also hold immense potential for generating tabular data, a modality essential to many real- world applications (Dash et al., [219 Borisov al. [2022] Shwartz-Ziv & Armon 2022). Despite the ubiquity of tabular data, obtaining high-quality samples for modeling remains a challenge. Al- though tabular data synthesizers have increasingly gained attention (Xu et al., 2019 Kotelnikov et al. 2023 McKenna et al., 2022), they yield little performance improvement in downstream mod- els when real data limited EElo & Averbuch- Elor 2022 Manousakas & Aydöre 2023). 8023).The This lim-
itation stems from fundamental constraint: synthesizers cannot generate information beyond what present in the original training data. Tabular GFMs have the potential to overcome this limitation
by leveraging diverse pre-training data.
Despite these opportunities, implementing tabular GFMs remains particularly challenging and largely overlooked due to the heterogeneity of column structures, feature sets, and value ranges (On- ishi et al. 2023 Huang et al. 2020; Borisov et al. 2022 Zhu et al. 2023 van Breugel van der Schaar 2024). Existing methods for transferable tabular learning either model tables with language
models Ye et al.
2024 Wang & Sun [2022] Heggelmann et al. 2023 Yan et al. 2024) or attempt to
leam unified latent space across datasets (Wang & Sun 2022
Onishi al. 2023 7hu al. 2023

---

Published as a conference paper at ICLR 2025
Ye et al. )023
These methods either lack generative capability rely pre-trained language models that process tables as unstructured text, distorting structural information and metadata that critical for effective tabular modeling.
To address these limitations, we propose CTSyn, a foundational model specifically designed for generating heterogeneous tables.
Unified table representation and reconstruction: We developed cross-tabular autoen- coder that embeds heterogeneous table rows, projects them into unified latent space, and decodes tabular values using table metadata guidance. This approach enables model training across diverse tabular formats, overcoming data-specific structural constraints while preserving structural information table-consistent manner.
Generative foundation model: Our versatile conditional diffusion transformer backbone efficiently samples from the unified latent space, enhancing flexibility and applicability across diverse tabular domains.
Cross-tabular pre-training: We conduct extensive cross-tabular pre-training on large- scale web-scale dataset containing 5 million rows. With diverse data domains covering common table applications, this pre-training serves as a foundation for various downstream generation tasks.
Through extensive benchmarking with real-world datasets, we demonstrate that CTSyn extends the gee-training/fin-ttniiii paradigm tabular data generation, achieving state-of-the-ar (SOTA) per- formance on low-data regimue. Crucially, by effectively leveraging prior knowledge and incorporat- ing table metadata, our model unlocks unprecedented potential in synthetic tabular data generation and holds immense promise for extending to various tabular tasks such as regression and classifica- tion.
RELATED WORK
2.1
TRANSFERABLE TABLE REPRESENTATION
Self-supervised learning can significantly improve representation quality for various downstream tasks (Gururangan et al. 2020 Yuan et al. 202 Wei et al., 2021 Chen et al. 2024). In the tabular supervised
domain, methods like VIME Yoon et al.] 2020) train an encoder using a combination reconstruction loss and mask-array prediction loss, while SCARF (Bahri et al. (022) employs con- trastive loss by utilizing randomly corrupted feature vectors as positive pairs. Subtab (Ucar et al.] 2021) and SSP (Chitlangia et al., 2022) integrate contrastive and reconstruction losses. However, these approaches do not produce transferable representations across tables, as they rely on data- specific feature encodings and structures.
Xtab (Zhu et al. 2023) and TabRet (Onishi et al. 2023) introduce transformer-based backbones with separate data-specific featurizers projection heads for each downstream task. These models achieve transferability at the expense of increasing model complexity as the number of datasets and tasks grows.
Pre-trained Language Models (PLMs) can be used to unify representation dimensions of heteroge- neous features. TransTab (Wang & Sun 2022) extends Subtab's methodology by tokenizing and then encoding column names and categories, creating latent space that can be shared across tables. By combining tokenization with a masked-vale -vauee prediction objective, transformer based models can be trained to perform predictive tasks across tables (Ye et al. 2024 Yak et al.) 2023 Yang et al.) 2024 Yan et al. 2024). However, such methods include neither a generative model backbone nor fixed-dimensional row representation and decoder that can be integrated with other generative models.
Another line of research involving PLMs converts tabular features to sentences and models regres- sion/classification problems as NLP tasks (Dinh et al. 2022 Hegselmann et al.) 2023 Borisov et al.) 2023 Liu et al. 2022 Zhang al. 2023c b). Despite enabling transfer learning, these methods face challenges in accurately modeling continuous values and tend to overlook the intrinsic structural
properties of tables (van Breugel & van der Schaar 2024).
2

---

Published as a conference paper at ICLR 2025
E
Metadata
Tokenize Embed
Pre rre-trained LM
FFN
33
Em
Age
Sex
Encoder JiF)
Decoder giBi
Similartyy
Male
33
Male
E
Z
Probabilities
Class
Normaliza
Encoder
Decoder
Diffusion:
Latent Space
Z
Z+
Real Table
Age
Sex
Diffusion
33
Male
Encoder
Process
Metadata
Denolsing
Z
Synthetic Table
Age
Sex
Transformer
33
Male
Decoder
Backbone
Conditioning
Pro Pre-trained LM
Age
Sex
Figure 1: Overview of the proposed CTSyn framework.
2.2 SYNTHETICC TABULAR DATA GENERATION
Synthetic Tabular Data Generation (STDG) has long been studied by statisticians (Nowok et al.j 2016 Reiter 2005 Chawla et al., 2002). Recent advances in deep generative models have signifi-
cantly pushed its boundaries (Che et al. 2017 Kim et al. 2021 Figueira & Vaz 2022).
In particular, CTGAN and TVAE (Xu et al. 2019) combine conditional generation with Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), along with model-specific normalization, handle highly imbalanced and non-Gaussian columns. CtabGAN+ (Zhao al., 2021 2024) proposes a solution for handling mixed-type and long-tailed variable distributions. Au- todiff (Suh al. (023) and Tabsyn (Zhang al. (0244) use combination of latent diffusion and data-specific autoencoder structures. While these are the most similar works to ours, they lack crit- ical transferable encoding and decoding capabilities. TabDDPM (Kotelnikov al., 2023) achieves the current state-of-the-ad in tabular generation by employing separate diffusion processes for nu- merical and categorical columns.
Some models also synthesize data with Differential Privacy guarantees (Jordon et al. 2018 Zhang al. 2017 McKenna et 2022]. Despite their effectiveness modeling column distributions, none of the above methods is able to effectively boost the training of machine learning models with synthesized data, greatly limiting their usage in data augmentation (Manousakas & Aydöre 2023).
GReaTT (Borisov et al. 2023) and Tabula İZhao et al.) 2023) generate tables using PLMs by treating table rows as natural language text. Despite showing evidence of transferability, they not consider cross-table pre-training and generation. Additionally, they introduce the risk of producing out-of- bound examples due to unconstrained sampling output tokens and face the well-known challenge of modeling numeracy in discrete token space (Wallace et al. 2019)
METHODOLOGY
In this section, we outline CTSyn, our approach overcoming the challenges of developing a tabular GFM. Figure provvides an overview of the proposed framework.
3.1
FEATURE EMBEDDING
Let an observation (row) in a mixed-type table be represented
(C1, F1, C22 F2, ... Cpy Fp);
3

---

Published as a conference paper at ICLR 2025
where C₁ for i ....... are the feature names, and for == 1.2...., . feature values, which can be either numerical or categorical. The parameter p denotes the number of features in this row. Let mmeta be the text metadata describing the context of the table.
the corresponding
To facilitate knowledge transfer across tables, it is crucial to develop a unified representation that preserves information at the cell, observation, and table levels. Using techniques such as one-hot encoding and min-max normalization results in the loss structural and contextual information: ta- bles with entirely different contents can end up with identical representations. Thus, we tokenize and embed all levels of information into vectors of consistent dimensions to facilitate unified modeling.
The first step in this process is consolidating all text metadata, column names, and categorical values, ensuring that integer class labels and abbreviations are expanded into their full textual forms to preserve their original meanings. Then, we create embeddings follows:
(x:))
if m;; is categorical, Quantile if my is numerical
cm LM(m),
ec.- LM(c,, ex,
where LM is pre-trained text embedding model that is invoked only once for each unique category column name. Quantile is quantile transformer (Pedregosa al., 2011) fitted to the dataset, mapping numerical values to a uniform distribution. 1 is a vector of ones with the same dimension as MyM, ensuring that numerical and categorical embeddings maintain consistent dimensionality.
Finally, we interleave all embeddings and flatten them into a sequence:
E = ,
Cz,). (Cez, Cz.)..... (Cep, C2,,] Cz,,) R PP22MM
(1)
where MLM is the dimensionality of the language model embeddings. Each step in the sequence formed by concatenating the column name embedding ec, with the corresponding column value embedding ex,, This interleaving creates cell-level representations that encapsulate both column type and value context, eliminating the need learn the relative positions of column names and values in the sequence. This design aligns with the permutation-invee invariann property of tabular data.
3.2 AUTOENCODER FOR HETEROGENEOUS TABLES
Encoder: To facilitate efficient learning of the diffusion model, we use an encoder model f to further compress the input sequence E into fieddddimensionl latent vector:
z (E) M
where l the latent dimension, and Mage is the size of each latent vector. The encoder is based on the Perceiver Resampler Yuaan et al.) 222 consisting of multi-head attention (MHA) blocks and linear layers. The learnable latent parameters serve as queries, while the keys and values consist of the concatenation of the latent queries and the flattened input sequence E.
In each layer of the encoder, a cross- aoss-tttention operation is performed where the latent queries iter- atively attend to both the input sequence (in the first layer) and the latent representations (in subse- quent layers). Formally, the output of one attention block is given by:
(++
FFN Z ( + +MHA(q= = ,, k== ,''))==')))
where ( is the latent representation at layer 1, MHA(-) represents the multi-heat attention op- eration, and FFN is a feedforward network. At the first layer = 0). the keys and values are the concatenation the latent queries and the input sequence, i.c., kv Zz (0) E].
Note that do not include positional embeddings in the sequence E to maintain the permutation invariance property tabular features.
Following the variational autoencoder (VAE) framework, use two separate encoders, where each output serves as either the mean vector μ E R Maz or the log-van varvinnaece vector log E A Ms, respectively. The encoder outputs parameterize the latent distribution, and for each input, sample the latent vector z using the reparameterization trick, given the predicted mean μ and log- variance log 2

---

Published as a conference paper at ICLR 2025
Decoder with Meta Guidance: To enable cross-tabular training, the decoder must handle varying column orders and combinations of mixed-type columns while ensuring permutation invariance. Since positional embeddings excluded during encoding, approach uses cross coss-attention based transformer decoder, where the column order the decoder is guided by explicit embeddings of the target column names.
We encode the column names using a PLM as:
E
[Ces: Cez.
Ceply
where each Ec; represents the embedding of column ei. These embeddings serve queries for the decoder.
The table metadata embedding 6m is concatenated with the latent variables z (derived from the encoder) to form the keys and values. The decoder g(-) operates by cross-attending to E. and [ems The order of the output from the decoder is thus determined by the order of columns in E., and the model learns to dynamically extract cell information from the row latent representation. The output of the decoder is: h s( EE (emi ((E)) E IR Maccoded
Table reconstruction: The decoded embeddings are used reconstruct the table cell values.
For numerical variables, the decoding process transforms the embedding back into scalar values using a linear layer followed by a sigmoid activation function, producing the predicted value:
f num
Sigmoid (innea((h)))
For categorical columns, we use a loss based on cosine similarity to accommodate unseen categories in real-world applications. Inspired by Yaak al. 2023), compute the cosine similarity by first refining both the reconstructed embeddings and the PLM embeddings of all real categories using linear layer. Then, calculate the cosine similarity between these embeddings, apply softmax to the similarities, and use the resulting distribution as the predicted class probabilities:
P(ajtt) Softmax (CosineSim (Linear(h,)) Linear(C)))
where h4 and hy are the latent representations of the i-th numerical cell and j-th categorical cell, respectively, and C represents the set of embeddings for all possible categories in the column. These predicted probabilities and values are then used to reconstruct the original table by mapping the latent space the appropriate categorical or numerical values for each cell.
Training VAE: Following the 3-VAE setup (Higgins et al. 2017) the overall objective is a combi- nation of numerical and categorical reconstruction losses and KI -eggeaalarizztton the latent space:
C
Cmum
nUm
DUM
+
Σ Lea (1 (( + (N (0, 1)), 1=1
1=
A=1
where Znum is the MSE loss for numerical variables, Lest is the cross-entropy loss for categorical variables. DKL is the KL-divergence between the learned latent distribution A N(H,, of and the stan- dard Gaussian (0,, 1), and 3 is weighting factor balancing the reconstruction and KI--divergence terms.
Implementation: We use four cross cooss-attention layers for both the encoder and decoder, with & = 16 latent dimensions and Mags = 64. All VAE models this paper are trained using the Adam optimizer with initial learning rate of 0.0002. The learning rate is multiplied by 0.95 if the validation loss does not improve for 10 consecutive epochs.
We 3-VAE setup, starting with 3max 10-2 and gradually decrease 3 by multiplying it by 0.7 when the reconstruction loss does not improve for consecutive epochs, until reaching a minimum value of 10
We construct training batches to contain samples from the same source table, improving training efficiency by reducing inter-domain contrast, unrealistic domain shifts not representative of real-world applications. We use GTE-large (Li et al.) (023a) as our text embedding model, as provides robust representations for metadata and categorical features. features.
5

---

Published as a conference paper at ICLR 2025
3.3 CONDITIONAL DIFFUSION MODEL FOR LATENT VECTOR GENERATION
For cross-tabular generation, a conditional latent diffusion model is preferred because table schemas can vary significantly across domains, with different column types and distributions. An uncondi- tional model would struggle to generalize across these diverse formats, making fine-tuning more difficult for domain-specine tasks. We follow the Denoising Diffusion Probabilistic Model (DDPM) formulation to train a conditional diffusion model, with the objective specified below. The input latent variable z is derived from our VAE.
For the denoising objective, utilize the o-parameterization strategy, which is more effective for latent diffusion than the classic noise prediction strategy. We condition the embedding generation on the embedding sequence [em, Ee], which encompasses the schema of the desired table. The model is trained with the following loss function:
((0) = (Zres de 1120
( (Var + vI are, t, [ems Ecl) - -Zug 2ugl12
where Bng is the latent variable from the target sequence, and or is the noise schedule. Classifier- free guidance used to improve sample quality, with conditional and unconditional networks jointly trained, where conditioning is dropped with probability of 0.1 during training.
Following the specifications in Lovelace et al. (024), our diffusion model uses a pre-LayerNorm transformer architecture with 12 layers, hidden dimension of 768, learnable absolute positional encodings, and a GeGLU activation function. The noise level is conditioned via a sinusoidal time embedding, which is processed by an MLP and added to the input sequence. Adaptive layer nor- malization is applied to each feedforward layer, conditioned on the time embedding. We use the AdamW optimizer with learning rate of 0.0001, cosine annealing scheduler, batch size of 256, and 250 sampling steps.
4
EXPERIMENT
4.1 EVALUATION SETUP
In this section, we evaluate the performance of CTSSnn in representing tables and generating infor- mative and diverse synthetic tabular data. Our primary research questions are: 1. Does pre-training on large, general datasets improve the quality of synthetic data generation for downstream tasks? 2. How does the inclusion of metadata and table schema help CTSyn create effective and transferable table representations?
Baselines: We compare our method against wide array of baselines in synthetic data genera- tion. These include modified SMOTE (Chawla et al., 2002). CTGAN and TVAE (Xu et al. 2019). TabDDPM (Kotelnikov et al.
2023).
TabSyn (Zhang al.) 2024). AIM (MeKenna et al. 2022
PATE-CTGAN (Jordon et al. 2018). and GRea'T (Borisov et al. 2023). The implementation of
baselines is detailed in Section
B
We use a filtered version of the Open'Tab dataset Ye et al. 2024) as our pre- training set. The filtering follows the strategy outlined in (Yan et al. (2024). which excludes duplicate tables, tables containing free-text, date-time, or personally identifiable information (PII) columns, tables with fewer than 10,000 rows, and tables with categorical columns in integer label format that cannot be mapped back to their original string representations. After filtering, the pre-training set comprises 86 tables with a total of 5.01 million observations. Note that while the multi-modal representation framework of CTSyn is extensible to include text and date-time variables, we focus on numerical and categorical variables in this work (an extension to text and date-time is demonstrated in Appendix F)
Dataset Construction:
For downstream benchmarking, we evaluate eleven real-world datasets widely used in the tabular synthesis literature (Suh et al.) 2023 Kotelnikov et al.) 2023 Zhang et al., 2024) (see Table I To
avoid any data leakage, we manually verify that none of the datasets used for pre-training appear in the downstream benchmarks; further details these datasets can be found in Appendix C For each downstream dataset, we randomly split the data into fine-tuning set (80%) and a held-out test set (20%). The fine-tuning set is then randomly shuffled, and few-shot subsets are created by

---

Published as a conference paper at ICLR 2025
selecting the first 30, 50, 100, 200, and 500 rows, respectively. This procedure allows assess the performance of synthetic data generation in low-data scenarios. For each few-shot subset, a separate generator is trained, and three synthetic tables generated over three independent trials. All synthetic tables generated for a given downstream dataset are evaluated using the same held-out test set. Synthetic data modeled from different subsets are all sampled to 500 observations to ensure a fair comparison and highlight impact of training data size.
For CTSyn, we pre-train the autoen- coder for 300 epochs and the diffu- sion model for 200,000 steps. For fine-tuning, we train the conditional diffusion model and decoder network of the autoencoder while freezing the encoder to maintain alignment in the latent space. We fine-tune the de- coder for 100 epochs and the diffu- sion model for 10,000 steps.
Dataset
Rows
Target
Num Cols
Cate Coll
Faults
1941
Classification
34
Wilt
4839
Classification
5
HTRU2
17898
Classification
8
News
39644
Regression
60
Bcan
13611
Classification
16
Obesity
2111
Classification
8
Titanic
714
Classification
6
Insurance
1338
Regression
Abalone
4177
Regression
8
Shoppers
12330
Classification
16
The text data describing table context was generated by query- ing the ChatGPTT-40 (OpenAl, 2024) model with the first 10 rows of each
Metadata:
Indian Liver Putient
579
Classification
9
2
Table 1: Summary Statistics of Downstream Datasets
table and prompting it to generate a
description of the table's context, domain, and potential use cases. The prompt and metadata gener- ation process is detailed in Appendix G
4.2 STATISTICAL FIDELITY
Model
Shape
Corr
Precision
Recall
SMOTE
0.96 (0.02)
0.91 (0.03)
0.68 (0.04)
0.02 (0.003)
CTGAN
0.81 (0.04)
0.73 (0.02)
0.57 (0.03)
0.014 (0.002)
TVAE
0.88 (0.03)
0.89 (0.04)
0.35 (0.02)
0.01 (0.001)
AIM
0.63 (0.05)
0.70 (0.02)
0.01 (0.001)
0.03 (0.003)
PATECTGAN
0.15 (0.01)
0.47 (0.03)
0.01 (0.001)
0.02 (0.002)
Ta
0.93 (0.03)
0.93 (0.02)
0.59 (0.04)
0.027 (0.004)
TabSyn
0.97 (0.01)
0.93 (0.02)
0.69 (0.03)
0.003 (0.001)
GReaT
0,90 (0.03)
0.64 (0.04)
0.68 (0.03)
0.005 (0.001)
CTSyn
0.94 (0.02)
0.95 (0.02)
0.64 (0.04)
0.075 (0.006)
Table 2:
Statistical Fidelity Metrics. Scores are averaged across benchmark datasets.
We evaluate the similarity between real and synthetic tables based on the marginal distributions of columns, column- wise correlations, and sample-level coverage.
For column distributions, we use the Saatnoggoro-Smminoy (KS) test for numerical columns and Total Variation Distance (TVD) for categorical columns, subtracting them from one so that higher values indicate better similarity. For column-wise correlation, we apply Pearson's 's correlation for numerical columns, a contingency similarity metric for categorical columns, and a combined method for mixed types (Dai, 2023). For sample-level coverage, we measure precision and recall to quantify the overlap between real and synthetic data (Alaa et al. (022).
Table 2 presents the average similarity of column distributions and correlations across benchmark datasets. CTSyn consistently matches or exceeds state-of-the-art baselines. While methods like TabSyn and SMOTE excel in maintaining lower-order statistical similarity due to their focus on replicating training data distributions, CTSSn demonstrates superior performance in capturing com- plex relationships between columns. This is particularly evident in its higher correlation scores and significantly better recall, indicating its ability to preserve important structural relationships within the data, as well as benefiting from the regularization effect of pre-training.
4.3 MACHINE LEARNING UTILITY
To evaluating machine learning utility of synthetic data, we fit the following classifiers: logistic regression, Naive Bayes, decision tree, random forest, XGBoost (Chen & Guestrin 2016), and Cat-
7

---

Published as a conference paper at ICLR 2025
Regrossion
Real
TRADOM TR20g OPeeT Craym
TaI0OPM
0.0
Shout
0.00
Syrth
0.00-
100
200
300
402
303
100
200
300
402
502
Classification F1-Scores
(h) Regression RMSE
Figure 2: Downstream Machine Learning Utility on Classification and Regrssion Datasets, on syn- thetic data from different generators.
Boost (Prokhorenkova al. 2018), and then evaluate them on the holdout test set. Figure reports the average classification Fl -ccre (for classification datasets) and regression root mean squared er- ror (RMSE) across models and shots. For clarity, only the top five performing models are shown. We report scores for the remaining models in Appendix
We observe that for the low-data regime (See- dat et al.) with < 100, CTSyn consistently outperforms all baselines and real data at
Model
PCT
Authenticity
SMOTE
0.82 (0.24)
0.88 (0.03)
the corresponding scale. The performance gap widens in the 100-200 shot range. This indi- cates CTSyn's ability to leverage pre-training data to assist training when real data limited. Note that GReaT failed to generate text that fol- lows the tabular format for N 100. However, as the data size increases further, the advan- tage of CTSyn diminishes. We conjecture that this phenomenon is due an ineffective trans- fer learning setup and leave the exploration of transfer learning for tabular GFMs beyond the simple pre- pre-trining/fine-tum paradigm to fu- ture work.
CTGAN
0,84 (0.04)
0.91 (0.29)
TVAE
0.84 (0.01)
0.95 (0.31)
AIM
1.00 (0.00)
1.00 (0.00)
PATECTGAN
1.00 (0.00)
1.00 (0.06)
TabDDPM
0.85 (0.07)
0.87 (0.16)
TabSyn
0.80 (0.03)
0.93 (0.03)
GReaT
0.74 (0.21)
0.79 (0.03)
CTSyn
0.90 (0.02)
0.97 (0.06)
Table 3: Privacy scores of synthesized data. Best
scores of non-DP synthesizers are bolded.
4.4 DIVERSITY AND PRIVACY
We evaluate the diversity and privacy of the synthesized data using two metrics: the proportion of synthetic examples with L2 distance closer the test set (PCT) compared to the training set (Platzer & Reutterer 2021), and authenticity scores (Alaa et al. 2022). which assess the likelihood that synthetic data point genuinely generated rather than memorization of real data. Lower PCT or authenticity values suggest that synthetic data points are too close to the training set, raising concerns about potential memorization risk. powerful generator can easily memorize training data, achieving falsely high fidelity and utility without truly generating new samples, thus harming downstream model generalization and breaching individual privacy.
Table presents the diversity scores. CTSyn achieves the highest PCT and authenticity scores compared to ofate-of-the-ar models like TabDDPM and TabSyn, indicating that CTSyn produces more distinct synthetic data. CTSyn's high PCT scores comparable to those of AIM and PATE- CTGAN, which incorporate Differential Privacy (DP) mechanisms to reduce proximity to real data. However, these DP models have demonstrated poor fidelity and utility in previous sections. CTSyn,
8

---

Published as a conference paper at ICLR 2025
by contrast, achieves balance between data utility, diversity, and privacy, further supporting the claim that pre- pre-training acts as regularization.
10522
LOAC
19KC3 TacSyn
LOVE
"M003PP
OifeaT
crSyn
-130
75
-50
30
28 -100 90 175
Figure 3: T-sne plot of Indian Liver Patient dataset from different synthesizers.
We further illustrate the diversity of synthesized data using 2D t-SNE projection Figure 3 for the Indian Liver Patient dataset. Among all synthesizers, CTSyn generates the most diverse data distribution, covering wider region around the test set. In contrast, baseline models tend to overfit to regions surrounding certain data points. This diversity, enabled by pre-training, explains CTSyn's superior utility, as the diverse pre-training data serves as implicit regularization, promoting better generalization.
4.5
ABLATION STUDY
Variants
In-distribution MSE
Unseen Columns
Permuted Columns
Acc
MSE
Acc
MSE
Acc
Column & Meta
0.0004
0.94
0.0063
0.76
0.0005
0.94
Column Name Only
0.0007
0.93
0.0068
0.68
0.0006
0,93
Meta Only
0.0008
0.91
0.0082
0.61
0.0009
0.91
PE Only
0.006
0.87
0.07
0.54
0.08
0.67
Table 4: Reconstruction performance of different autoencoder settings.
Importance of Metadata. We evaluate the key factors for effective cross-tabular representation and reconstruction by testing different autoencoder variations: removing column name embeddings in decoder guidance, removing both column names and metadata embeddings while using positional encoding to encoder on positions seuqnce E. These models are trained on the same pre-training data and tested on three scenarios: (1) in in-distribution data from the validation splits, (2) unseen data from downstream test datasets, and (3) Permuted data, where column order is randomly permuted.
Table presents the results in terms of Mean Squared Error (MSE) and categorical accuracy. Re- moving column names primarily affects categorical column reconstruction, emphasizing the need for contextual representation. The replacement of metadata with PE significantly worsens perfor- mance, particularly when handling permuted column order. This highlights a fundamental distinc- tion between tables and unstructured data like text: tables have permutation sermutation-invariant structures, and relying on positional information, as PE, is ineffective.
We further observe that while retaining column names alone results in some performance degrada- tion compared to the full model, the impact is less severe than dropping column names entirely. This outcome highlights the importance of column names in capturing table structure and semantics. We

---

Published as a conference paper at ICLR 2025
conjecture that this robustness arises from our diverse pre-training dataset, where column names act as proxies for metadata.
Model
Pretrained
Shape
Corr
Synth F1
Synth RMSE
PCT
0.94
0.95
0.84
0.13
0.97
CTSyn
0.96
0.90
0.80
0.17
0.90
0.90
0.71
0.79
3.75
0.88
GReaT
0.90
0.64
0.83
0.18
0.79
0,88
0.82
0.75
0.23
0.90
TabSyn
0.97
0.93
0.84
0.14
0.93
Table 5: Impact of transfer learning on different models.
Impact of Pre-training. We compare different models' potential to leverage pre- pre-training data. We repeat the experiments with CTSyn trained without pre-training, while GReaT and TabSyn also pre-trained before training on downstream benchmarks. For GRea'T, we pool all pre-training data into dataloader to train meettrained distilled GPT-2 and use the resulting model as initialization for downstream benchmark training. For TabSyn, since its model structure is data-specific, we first train separate VAE networks for each pre-training table, pool, and zero-pad all embeddings the same length, and use them to train latent diffusion model. During downstream training, the VAE embeddings are padded to the same dimension in pre-training.
As shown in Table the performance of CTSyn degrades without pre-training, though it remains on par with other state-of-the-an models across all dimensions. On the other hand, the change in performance of GReaT and TabSyn with pre-training is mostly negative, indicating their inability to effectively translate knowledge across tabular domains and further reinforcing the need for a model that comprehensively encodes tabular structure information like CTSyn.
CONCLUSION
In this paper, we introduced CTSyn, a pioneering framework within the realm of Generative Founda- tion Models (GFMs) for tabular data. Through extensive experimentation real-world datasets, we demonstrated that CTSyn effectively represents heterogeneous tabular data while leveraging knowl- edge from diverse pre-trained tables to enhance synthetic data in low-data regimes.
To the best our knowledge, CTSyn the first method to integrating large-scale pre-training in tabular data generation via lateng diffusion. Our results highlight its ability to capture complex relationships between table columns, maintain permutation invariance, and generate synthetic data with high fidelity and diversity, surpassing existing state-of-the approaches. By incorporating metadata and schema- based conditioning, CTSyn bridges a critical gap in cross-table generalization, allowing for more robust and transferable table representations.
Despite these advances, challenges remain in fully optimizing tabular generative models for broader applications. Future work should explore the potential of conditional diffusion mechanisms and generative pre-training to further enhance tabular tasks such as predictive modeling, imputation, and multi-table learning. Additionally, investigating transfer learning paradigms beyond simple tre-training/full fine-tuning could improve adaptability across diverse tabular domains, unlocking new frontiers synthetic data generation such as in-context generation and parameter efficient fine tuning for tabular diffusion model.
REFERENCES
Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning, pp. 290-306. PMLR, 2022.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022.
URLhttps: http:::::///peeeeveeeeeee www.wwwww.wweeeweeteeeeee
10

---

Published as a conference paper at ICLR 2025
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions Neural Net- works and Learning Systems, 2022.
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Lan- guage models are realistic tabular data generators. In The Eleventh International Confer- net forum?id-
ence on Learning Representations, 2023. URL CEygmONDeI
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer, Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Zhengping Che, Yu Cheng, Shuangfei Zhai, Zhaonan Sun, and Yan Liu. Boosting deep learning risk prediction with generative adversarial networks for electronic health records. In 2017 IEEE International Conference on Data Mining (ICDM), pp. 787-792. IEEE, 2017.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo. Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. International Journal of Computer Vision, 132(():208-225 2024.
Sharad Chitlangia, Anand Muralidhar, and Rajat Agarwal. Self supervised pre-training for large scale tabular data. 2022.
Sabyasachi Dash, Sushil Kumar Shakyawar, Mohit Sharma, and Sandeep Kaushik. Big data in healthcare: management, analysis and future prospects. Journal of big data, 6(1):1-25, 2019. 2019
Synthetic Data Metrics. DataCebo, Inc., 10 2023.
URL https://docs adv. dev
/dmmetrics/ Version 0.12.0.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfaced fine-tuning for non- language machine learning tasks. Advances in Neural Information Processing Systems, 11784, 2022.
Yotam Elor and Hadar Averbuch-Elor. To smote, or not to smote? arXiv preprint arXiv: 2rXiv:2201.08522 2022.
Alvaro Figueira and Bruno Vaz. Survey on synthetic data generation, evaluation methods and gans. Mathematics, 10(15):2733, 2022.
Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables. arXiv preprint arXiv: 160400677 2016.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 8342-8360, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 5549-5581, PMLR, 2023.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karinn. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06675 2020.
James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. In International conference learning representations, 2018.
Jayoung Kim, Jinsung Jeon, Jaehoon Lee, Jihyeon Hyeong, and Noseong Park. Oct-gan: Neural ode-based conditional tabular gans. In Proceedings of the Web Conference 2021, pp. 1506-1515, 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304. 02640,23333 2023.
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference Machine Learning, pp. 17564- 17579. PMLR, 2023.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-ssquencce pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7888 2020.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, arXiv preprint arXiv:2308.03: 032811, 2023a.
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large lan- guage models for text classification: Potential and limitations. In The 2023 Conference Ëm- pirical Methods in Natural Language Processing, 2023b.
Guang Liu, Jie Yang, and Ledell Wu. Ptab: Using the pre-trained language model for modeling tabular data, arXiv preprint arXiv:2209.0800 2022.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation. Advances in Neural Information Processing Systems, 36, 2024.
Dionysis Manousakas and Sergül Aydöre. On the usefulness of synthetic tabular data generation. arXiv preprint arXiv:2306. 15636, 2023.
Ryan McKenna, Brett Mullins, Daniel Sheldon, and Gerome Miklau. Aim: an adaptive and iterative mechanism for differentially private synthetic data. Proceedings the VLDB Endowment, 15 (11):2599-2612, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 160.v:1609.0770 2016.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelli- gence. Nature, 616(7956):259-266 2023.
Beata Nowok, Gillian M Raab, and Chris Dibben. synthpop: Bespoke creation of synthetic data in Journal of statistical software, 74:1-26, 2016.
12

---

Published as a conference paper at ICLR 2025
Soma Onishi, Kenta Oono, and Kohei Hayashi. Tabret: Pre- Pre-training transformer- based tabular mod- els for unseen columns. ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
OpenAl. Gpt-4 technical report. arXiv preprint
2023.
OpenAl. Chatgpt (got-40-2024-08-06) [large language model]. 2024.
URL https:/
alatform openai wtt.ccommmmoo/m. l /ppp 400400/0000p---222000--000000000 2024-08-2-
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Couraaeeau M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2836 2011.
Michael Platzer and Thomas Reutterer. Holdout-based empirical assessment of mixed-type synthetic data. Frontiers in big Data, 4:679939, 2021.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorohev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.66125 2022.
Jerome Reiter. Using cart to generate partially synthetic public use microdata. Journal of official statistics, 21(3):441, 2005.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-1069 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25225 2022.
Nabeel Seedat, Nicolas Huynh, Boris van Breugel, and Mihaela van der Schaar. Curated Nm: Syn- ergy of llms and data curation for tabular augmentation in low-data regimes. In Forty-first Inter- national Conference on Machine Learning.
Ravid Shwartz--Zi Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
Namjoon Suh, Xiaofeng Lin, Din-Yin Hsich, Merhdad Honarkhah, and Guang Cheng. Au- todiff: combining auto-encoder and diffusion model for tabular data synthesizing. CoRR, Abb/2330.15449, 2023. httll:hts////// /oi.. org //lg//10.. 48550/ arxiv. 2310. 15479.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307 009888888.... 2023.
Brandon Trabucco, Kyle Doherty, Max A Gurinas, and Ruslan Salakhutdinov. Effective data aug- mentation with diffusion models. In The Twelfth International Conference on Learning Repre- sentations, 2023.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for supe-supe representation learning. Advances Neural Information Processing Systems, 34:18853-18865, 2021.
Boris van Breugel and Mihaela van der Schaar. Why tabular foundation models should be a research priority. arXiv preprint arXiv:2405.00 147, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2025
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do nip models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP) pp. 5307-5315, 2019.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902-2915, 2022.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in down- stream tasks? analysis of head and prompt tuning. Advances in Neural Information Processing Systems, 34:16158-1 10615811617771 60,0,22000 2021.
Lei Xu, Maria Skoularidou, Alfredo Infeate, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in neural information processing systems, 32, 2019.
Scott Yak, Yihe Dong, Javier Gonzalvo, and Sercan Arik. Ingestables: Scalable and efficient train- ing of Ilm-enabled tabular foundation models. In NeurlPS 2023 Second Table Representation Learning Workshop, 2023.
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. The Twelfth Interna- tional Conference on Learning Representations, 2024. URL /tttps://operreiiiie /eeeerervieeeee//e
Serumiid=anzIzeee
Yazheng Yang, Yuqi Wang, Guang Liu, Ledell Wu, and Qi Liu. Unitabe: A universal pretraining protocol for tabular foundation model in data science. In The Twelfih International Conference on Learning Representations, 2024.
Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Towards cross- table masked pretraining for web data mining. Proceedings of the ACM on Web Conference 2024, pp. 4449-4459, 2024.
Han-Jia Ye, Qile Zhou, and De-Chuan Zhan. Training-free generalization heterogeneous tabular data via meta-repreesentation 2023.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela Van der Schaar. Vime: Extending the suc- cess of self-and semi-supervised learning to tabular domain. Advances Neural Information Processing Systems, 33:11033-11044 2020.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021a.
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6995- 7004, June 2021b.
Boyu Zhang, Hongyang Yang, Tianyu Zhou, Muhammad Ali Babar, and Xiao Yang Liu. Enhancing financial sentiment analysis via retrieval augmented large language models. In Proceedings of the Fourth ACM International Conference on in Finance, pp. 349-356, 2023a.
Han Zhang, Xumeng Wen, Shun Zheng, Wei Xu, and Jiang Bian. Towards foundation models for learning tabular data. arXiv preprint arXiv:2310.07338. 2023b.
Hengrui Zhang. Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score- based diffusion in latent space. In The Twelfth International Conference on Learning Represen- tations, 2024. ht..st//:/// //tp: htt.eeew.eeeotteeeeeeeeee wow.umiid=:aay222.
Jun Zhang, Graham Cormode, Cecilia M Procopiue, Divesh Srivastava, and Xiaokui Xiao. Privbayes: Private data release via bayesian networks. ACM Transactions on Database Systems (TODS), 42(4):1-41, 2017.
14

---

Published as a conference paper at ICLR 2025
Tianping Zhang, Shaowen Wang, YAN Shuicheng, Li Jian, and Qian Liu. Generative table pre- training empowers models for tabular prediction. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023c.
Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Chen. Ctab-gan: Effective table data syn- thesizing. In Asian Conference on Machine Learning, pp. 97-112. PMLR, 2021.
Zilong Zhao, Robert Birke, and Lydia Chen. Tabula: Hamesssing language models for tabular data synthesis. arXiv preprint arXiv:2310.1272 2023.
Zilong Zhao, Aditya Kunar, Robert Birke, Hick Van der Scheer, and Lydia Y Chen. Ctab-gan+: Enhancing tabular data synthesis. Frontiers in big Data, 6:1296508, 2024.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: cross-table pretraining for tabular transformers. In Proceedings of the 40th International Confer- ence on Machine Learning, pp. 43181-43280 2023.
15

---

Published as a conference paper at ICLR 2025
Appendix
A BROADER IMPACT AND LIMITATIONS
foundational table generator like CTSyn can significantly enhance various application domains, especially where real data is scarce, sensitive, expensive to obtain, by providing high-qualitg synthetic tabular data. In healtheare, for example, CTSyn can generate synthetic patient records that maintain statistical fidelity to real data, enhancing the robustness and generalization ability of machine learning models by augmenting datasets with synthetic data.
CT also facilitates data collaboration between parties, such advertising companies and social media websites. Through conditional generation, CTSyn can augment one party's 's dataset with es- sential columns for business analysis without violating privacy laws that prohibit linking individual data points across parties.
However, CTSyn's performance relies heavily on clean, large-scale tabular datasets. The quality of generated data depends on the training data, and any biases or errors can be propagated. This risk can be mitigated by carefully curating high-quality datasets for different domains. Additionally, despite pre-training reducing memorization of downstream data, individuals included in the pre- training data still face privacy risks, complicating the safe gathering of large datasets. This can be mitigated by properly anonymizing or adding noise public pre-training datasets to ensure privacy before they are used for pre-training.
The requirement of semantically meaningful category names also present challenges for acquiring large-scale training data, as normalized values must be carefully sanitized and converted back to raw form.
B BASELINES IMPLEMENTATION
We use the official implementation at http:://ithub..om///dddd/CCCCCCt bedding dimension =128, generator dimension=(256.5500 discriminator dimension =(256,256), generator learning rate=0.0002, generator decay =0.000001, discriminator learning rate =0.0002, discriminator decay =0.000001, batch size=500, training epoch 300, discriminator steps=1, pac size 5.
CTGAN:
We use em-
TVAE: We used the official implementation https:/ddoss..dv.ee///d We used default parame- ters: class dimensions =(256, 256, 256, 256), random dimensions=100, 64 channels, 12scale=le-5, batch size=500, training epoch 300.
TabDDPM: We used the official implementation at We used 2500 diffusion steps, 10000 training epochs, learning rate 0.001, weight decay 1e-05, batch size = 1024.
We use the code implementation http://hiihbhccooaaaaaa/1222///p 12358/private-pgmm, with default parameters: eesilon=3,delta=ll-99 ,aax model size=80
AIM:
PATE-CTGAN: We adapted the implementation posted at:
dulkblobbmmainnsyyttbssnnyttt which combines the PATE (Jordon et al. 2018) learning framework with CTGAN. We use epsilon = 3, 5 iterations for student and teacher network, and the same value for other parameters which are shared with CTGAN.
GReaT: We used the official implementation at
great seeatt/tree/mme im We used batch size of 32. During pre-training, we began with a pre- trained distilgpt2 model and training for 2 millions steps on the combination of pre-training data. We train 200 epochs for each dataset during finetuning.
TabSyn: We use the official implementation at /ittttt://// http::///gitub..//// com/ amazon science/ tabsyn, with default parameters. For pre-training with heterogeneous VAE embeddings, we train its VAE model for each pre- pre-training dataset, zero-pad all embeddings to the same dimension, and then ire-train& diffusion model such padded embeddings. During downstream training, the VAE embedding of the downstream datasets are padded to the same dimension as in the pre-training. The pre-trained TabSyn is loaded and diffusion training proceed with it initialization.
16

---

Published as a conference paper at ICLR 2025
The original SMOTE algorithm designed to upsample minority classes. We extend to perform interpolation for all classes. For each generation, we first randomly select one target class using empirical class frequency probability. Then randomly sample one example from the selected class, and generated interpolated examples using number of nearest neighbour 5. The interpolation weight 0.5.
SMOTE:
C BENCHMARK DATASETS
We provide the URL for the sources of each downstream benchmark set considered in the paper.
abalone (
active (Multi class)
2. Bean (UCI) http://astiiiiuuuuuuudlcalllol/d
class)
3. faults (UCT): http:///tttii...ccccooo////////e (Multi class)
4. HTRU http://///tiiiiiisss//////////// class)
5. indian liver patient (Kaggle) http://wwwwwaaaagkcccoommmooeeew
class)
http://www..lgggcoooooosooooocec (Regges 6. insurance (Kaggle) sion)
7. News (UCI): wttp://wwhhccuuuooocaaaa//////ia (Regression)))
8. Obesity (Kaggle): http://wwwwwaaaooooooooooooaaaoa ml (Multi class)
9. Shoppers (Kaggle): http://wwwww.goooooooooo./////op (Binary class)
Titanic (Kaggle): http:///ww..aagleeemm////iiiiaia class)
11. wilt (OpenML) http:///wwwwww/ooooooooyyyyy//// active (Binary class)
D PRETRAINING DATASETS
We show the TpenTab files included in our pre-training, as well as their summary statistics. The classification type dataset are shown in table O and regression datasets in table
17

---

Published as a conference paper at ICLR 2025
File Name
Categorical Cols
Numerical Cols
2736_Shippinn Shipping
10999
6
1366 bankmarkettinn
41188
11
10
0944. SensorraataResource
100000
25
01 B...ooorrigee..eessioolllll
100000
0062
100000
10
1
0673 (NG(baseball)
100000
16
1046 jungle chess 2pes endgame
44819
6
complete
0677 0677_COME COMETTMC.SAMPP 1681 1998
89640
5
15007
12
4
28155
3
pulsar traia.triin
12528
9
0666 C066_BNGGprimarryy
100000
18
0
M.00.BNG(trrastttccccer.mmmmmm ort.rattteaacnce.ooommaaoo000000 (0vo_BN
100000
1
100000
1375_M Seeiic--amma--elssopp-----ee
19020
10
0063 Bwl.BBGGeddttg.ooooee g.nominll,, (atmiinal,1000000 1000000)
100000
21
0
1431 1431_ _Bejjing Multi-Site Sitei--itt-Aiii Air-Quality
100000
16
bodyPerformance
13393
10
term dem.deppsit.o. www..eeoo..uuusribeeeeeeeeeeee subscribed33
31647
8
1465 credit
16714
0077 BNGhheart-saalog.nonnt 1000000)
100000
14
0
0761 BNG(autos, 1000, 0142 BNG(hreast-w)
100000
16
39366
9
0105 kropt
28056
3
campaign33
12870
2149_ 19__ecctricityy
38474
8
2701 1772.BNG((ymph.50000.
24780
0
8
0772.
100000
16
0059 nomiioll....miill..o 1000000)
100000
1egg.lettr-ccalleeg--uaaaadd
10000
16
0639 jml
10885
21
fusion experiment
100000
17
1690 Malware- on-Headers
43293
5
0137 BNG(labor)
100000
8
1020 110__Ru__oo wwt.wwll........e
information www.k_infoommationn
88588
7
0070 0070_BN BNG(B nominal,
100000
classifying documenn.typpee to enhanc
scarcch and recommendations. in dig ital libraries_datase
11539
5
0747 BNG(letter,5000.]
100000
16
Warehouse_bloc
10999
7
1579_ 1579_Magic'Ter
13376
10
O160_BNG(hhppaee BNG((hepatiiss)
100000
14
6
1981 1981_Higgs
100000
25
1674_adult
48842
6
2687_Diabetes I 13US
71090
8
0057 (NG(mushroom)
100000
0
0074 BNG(tic-tac-toe)
39366
0
BNG(vehicke.nominll 000 univ.ai.Tes. Data deabtt.cllysttrainn bank
0078
100000
19
0
28000
5
100000
2
11162
10
7
Firewall_Run BwwewallRRulleCCM CtueeCllasiffiattin
100000
11
Booo_AArreuttureDiiae
88858
4
0711 1ttgger
100000
(074.BNG(wine)
100000
14
mushroom
12960
0
0968 BNG(segment)
100000
0
45211
8
Table 6: Classification Task Files
18

---

Published as a conference paper at ICLR 2025
File Name
N
Categorical Cols
Numerical Cols
1860_ Worldwidde-Coop-rrooeee
21165
2711 -eddemellll..cm charges
100000
4
0690 (NG(breas((umor)
100000
6
4
2743.Tall T4loTollo
100000
12
MAMe_datase
37407
4
4
2664 diamonds
53940
10692
0693 Beiee/iGGGweeeeeeeee
)
100000
0
12
2e7_elevators 2677 fifa
16599
17
19178
28
1940_scattlccrime6
52358
5
3
10361
0
1905 Sww-Delhi-Rennal-Liisee
17890
5
9
1415 beijing-pm2.5 1199_TTmilnadu-CC www.-rrropppoeccce stats
43824
11
13266
10000
9
1466 post-operative
65532
11
Airline Airline_Delay Delay-Causee Cause
100000
17
Hoouse-Rent-in Rent-in-Indian
10692
5
8
calities credit_card _efuuull
10000
2
2
1781 SDSS-16
100000
1
17
20640
O
1595
Oranges-vs-Graaeffiii
10000
1
5
[100_exerssses
15000
6
1245 Produection-rrxssscciioss----
50625
0
13
-Doublet-Mooee
2136 1.6bung-aaxigrree-dee221116
100000
0
10
06884 BNG(autoPrice)
100000
0
16
Seele-Commlete-Stoceee 1107
Stock-Datta190022222
10015
0
6
16755
2659_video transcoding
68784
2
17
Table 7: Regression Task Files
19

---

Published as a conference paper at ICLR 2025
E
NUMERICAL RESULTS FOR UTILITY
Model
30
100
200
500
Full
Real
0.79
0.79
0.82
0.83
0.88
0.90
SMOTE
0.67
0.74
0.76
0.77
0.83
0.85
PATECTGAN
0.31
0.27
0.32
0.34
0.37
0.40
AIM
0.44
0.48
0.55
0.62
0.52
0.57
CTGAN
0.41
0.52
0.52
0.54
0.63
0.64
GReaT
0.76
0.80
0.84
0.85
TVAE
0.75
0.77
0.79
0.79
0.82
0.84
TabDDPM
0.77
0.79
0.81
0.81
0.84
0.85
TabSyn
0.76
0.78
0.81
0.82
0.85
0.86
CTSyn
0.79
0.81
0.83
0.84
0.84
0.86
Table 8: ML utility for classification Senchmarks.Columeee represent training examples(shots) pro-
vided.
Model
30
50
100
200
500
Full
Real
0.24
0.23
0.21
0.17
0.15
0.14
SMOTE
0.48
0.24
0.22
0.16
0.13
0.11
AIM
10274.55
> 10k
> 10k
>> 10k
> 10k
110.14
PATECTGAN
> 10k
>> 10k
>> 10k
>> 10k
>> 10k
10k
CTGAN
0.97
0.25
0.28
0.19
0.21
0.20
TVAE
0.60
0.50
0.47
0.33
0.34
0.28
GReaT
0.30
0.25
0.23
0.20
0.19
0.16
TabDDPM
0.35
0.30
0.27
0.21
0.20
0.18
TabSyn
0.27
0.23
0.22
0.19
0.16
0.13
CTSyn
0.22
0.18
0.17
0.15
0.14
0.12
Table 9; ML. utility for regression benchmarks. Columns represent training examples(shots) pro- vided.
VAE FOR TEXT AND DATETIME VARIABLES
In our main experiment, we omitted free text and date-time variables to align with prior research in tabular transfer learning Yan al. 2024). However, CTSyn can handle these data types with proper decoder. We conducted preliminary VAE reconstruction experiments on the 'ebay reviews' and '0875 nfl games' table from Open incorporating textual and date-time columns. We train a VAE model from scratch jointly on these two data frames. For text encoding and decoding, we followed (Lovelace al.) 2024), encoding text with a pre-trained frozen BART-base model (Lewis) et al. (2200) followed by 4 layer Perceive resampler encoder-decoder network, and train the entire VAE and text encoder with cross-cntroly loss on token prediction. Date column cyclically en- coded (Guo& & Berkhahn 2016) capture periodicity and decoded with a linear layer with circular loss function. The reconstruction performances are as follows:
Metrie
Numerical (MSE)
Categorical (Accuracy)
Text (ROUGE-1, eBay)
Date-Time (Cireular Loss, NFL)
Score
0.0005
92%
0.80
0.021
Table 10: Performance metrics across different data types.
These findings demonstrate CTSyn's potential to handle diverse data types without compromising performance. Further studies can be explore the potential of text pre-training in assisting categorical reconstruction, well text decoding method for table summarization tasks.
20

---

Published as a conference paper at ICLR 2025
G METADATA GENERATION
To generate metadata for each table, we used GPT-4o (g01-40-2004-08--00) The metadata serves as textual description capturing the context of the table, aiding in cross-table learning and improving generalization across diverse tabular datasets. Each table's metadata was generated by providing GPT-4o with the first 10 rows of the dataset and prompting it to generate concise yet informative description. Below, we detail the exact procedure, including an example prompt and the correspond- ing metadata generated for the Indian Liver Patient Dataset.
G.1 QUERYING GPT-40 FOR METADATA
For each table, we queried GPT-4o using structured prompt. The prompt included:
A brief instruction specifying that the model should describe the table.
The first 10 rows of the dataset, formatted as a tab-separated table.
A request for summary that captures the table's context.
G.2 EXAMPLE PROMPT
The following prompt was used for query GPT-4o in metadata generation:
You are analyzing a dataset called (dataset_name)
Below are the first 10 rows of this dataset along with the correspending
column names:
Columns: (column names)
Data: Ffirst_row_of_dataa latooo___aaa____eeeee feamffrrmm_____
Based on the above data, please do the following:
1. Identify which columns
likely categorical.
2. Determine whether all categorical columns are represented in text form.
If a categorical column has integer values like 0 or 1, then it is not in text form. But if a categorical column has values like male/female then it is text form.
3. Output the result as JSON string with:
"dtypes": dictionary mapping column names to "categorical" or muuerrical.,
"metadata": A short description of the dataset background. "nontextcolunns"" A 1ist of categorical columns not in text form.
The output should be JSON string and should not contain anything else.
G.3 GENERATED METADATA EXAMPLE
Upon submitting this prompt to GPT-4o, the model generated the following metadata:
"The dataset contains numerical data related liver function and alcohol con- sumption. It is likely used for medical research purposes, particularly to analyze liver-related enzyme levels and overall liver
the effects of alcohol consumption health. Each column represents specific biochemical parameters from liver tests as well as the reported number of alcoholic drinks consumed by an individual."
H LATENT SPACE VISUALIZATION
We conducted a qualitative analysis comparing the encodings generated by the Transformer-VAE used in TabSyn with those from the encoder module of our VAE on an insurance dataset. Latent vectors of data points were color-coded based categories. The results demonstrate that model maintains clear separation of groups in the encoding space, providing evidence that our shared latent space enhances model capacity while maintaining interpretability for heterogeneous tables.
21

---

Published as a conference paper at ICLR 2025
-17
(a) Tsne plot on Region column.
(b) Tsne plot on Sex column.
Figure 4: Tsne visualization of embedding space created by pre-trained VAE model, on columns of Insurance dataset.
ENCODING UNINFORMATIVE FEATURES
Metric
Original Data
Uninformative Column Names
Shape
0.94 (0.02)
0.92 (0.03)
Corr
0.95 (0.02)
0.93 (0.03)
Precision
0.64 (0.04)
0.60 (0.05)
Recall
0.075 (0.006)
0.070 (0.007)
Table 11: Comparison of metrics between Original Data and Uninformative Column Names.
Non-informative abbreviated column headers(such as "XI", "VARI") is a notable concern for tabular foundation models, especially as such models are expected to be pre-trained on web-scale dataset with limited curation in a self-supervised manner. To evaluate the robustness of CTSyn against such noisy data, we conducted an experiment where all column names in downstream bench- marks used during finetuning real-world scenarios with non-descriptive headers in finetuning data. The results show slight de- cline in statistical fidelity; however, CTSyn maintains high level of performance, demonstrating its robustness to uninformative column headers.
replaced with single English letters (A-Z). This setting simulates
J COMPUTATION
Our training are completed on an Amazon AWS g5. 1xxlare instance, with 192 GB system memory, Nvidia A10G GPU with 4 x 24 GB GPU memory. The pre- pre-training time of CTSyn, GReaT and
TabSyn are shown in the table [2]
Model
VAE
Generation
CTSyn
12 hours
12 hours
GReaT
50 hours
TabSyn
86 x 0.5 43 hours
24 hours
Table 12: Pre-training computation cost. Note that TabSyn requires training table-specific autoen- coders.
22


Reviews:
[{"review_id": "qidVQYYO9L", "reviewer": "Reviewer_nmzh", "summary": "The authors propose CTSyn, specifically targeting generating heterogeneous tables. The author propose to do this by first using a cross table VAE, which can embed tables with different types of rows and projects them in a common latent space. Subsequently, they employ a DDPM based diffusion model (With classifier free guidance via a pre-trained LM) to generate samples for different types of tables. They also show the benefits of pre-training on a large number of diverse tables -- so as to serve as a starting point for different downstream tasks. The authors present results on multiple downstream tabular datasets with classification and regression tasks - and compare with multiple baselines -- with regard to fidelity, ML utility, and privacy.", "questions": "Please address the weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. While there have been prior works employing auto-encoders + latent diffusion models towards tabular data synthesis (e.g. https://arxiv.org/abs/2310.09656), prior works dealing with heterogenous table synthesis have been limited.\n2. The authors provide good comparison against baselines on a good range of datasets -- for fidelity, privacy and ML utility of generated data.", "weaknesses": "1. While a common latent space across tables provides a strategy to work on heterogeneous tables, it certainly limits the ability to interpret what the embeddings in the space mean - and the authors have not studied this aspect (to clarify, this is different from the privacy plots)\n2. The pre-trained LM to emit embeddings for rows - individually for each column type, while preserving tabular structure - raises questions about scalability to enterprise tables - which have thousands of columns associated with each table (and can become even bigger due to joins, lineage additions, etc). Since this directly impacts the dimension (Eq 1, Page 4) - it can have an impact on the representative power when reduced to a common low dimensional space (with tables with much smaller number of columns)\n3. Many tables often have short form/ abbreviated or cryptic tabular headers - where the tokenizer of the pre-trained LM can suffer - the authors can potentially add a study on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FGQHj7L9BO", "reviewer": "Reviewer_GE4V", "summary": "This paper introduces CTSyn, a foundational model designed to generate heterogeneous tabular data. CTSyn leverages an autoencoder that consolidates diverse tables into a unified latent space and reconstructs data based on the provided table schema embedding, allowing it to adapt dynamically to various table formats. Through large-scale pre-training, CTSyn outperforms existing data synthesizers and demonstrates superior performance compared to real data in low-data scenarios.", "questions": "1. As part of the ablation study, I am also interested in seeing the results of experiments with only the column name condition applied.\n2. While reviewing the provided code in the supplementary material, I encountered issues running vae.sh smoothly. Would it be possible to provide an updated .sh script that can reliably reproduce the main experimental results or specify the exact package versions used? Thank you for your help in ensuring reproducibility.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A strength of this work is the successful application of a straightforward approach to map embedding vectors of categorical variables back to their original space. Although simple, this method effectively demonstrates that returning to the original categorical space can be achieved without complex transformations, providing a useful baseline for handling categorical data embeddings.", "weaknesses": "A limitation of this work is that it primarily proposes a method for handling individual variables within a framework similar to LSGM [1] that trains a diffusion model in latent space. As such, the approach lacks substantial novelty and may have limited impact, given that it focuses on variable handling within an established generative model framework rather than introducing fundamentally new techniques.\n\nMinors. Typos in line 129 (specific) and 266 (The)\n\n[1] Vahdat, Arash, Karsten Kreis, and Jan Kautz. \"Score-based generative modeling in latent space.\" Advances in neural information processing systems 34 (2021): 11287-11302.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sdgJJFbbtZ", "reviewer": "Reviewer_mmN9", "summary": "Authors propose Cross Table Synthesizer (CTSyn), a tabular data generation pipeline that leverages diffusion models and variational autoencoders to encode and generate data from diverse types of tables. Specifically, CTSyn comprises a pre-processing step where the table meta-data, table column names and column values are all embedded using Language Models (and quantile transformer in the case of continuous column values) into a common embedding space. Each table row is then represented using the meta-data, column-name and column-value embedding vectors. This sequence of embeddings is supplied to a $\\beta$-variational autoencoder ($\\beta$-VAE) for embedding multiple diverse types of tables in a common embedding space. Finally, a diffusion model is trained (conditioned upon the latent representation output from the encoder in the $\\beta$-VAE) to generate synthetic table data. Results demonstrate that synthetic data generated by the proposed technique achieves good statistical similarity (evaluated using column-wise metrics) to real data. Further, results also demonstrate that the representation learned by CTSyn is statistically similar to the training data while not exactly replicating the training data. Overall, the experimental comparison to baselines and research questions investigated demonstrate the prowess of the proposed synthetic tabular data generation technique.", "questions": "1. Why are `free text` and `date-time` containing tables dropped (line 280)? Does this have to do with the inability of the existing pipeline to encode linearly increasing data (e.g., dates / times) or was there some other reasoning?\n2.  How was PE generated to be employed with different tables? Although the motivation of column order invariance is clear in the tabular data generation context, what is the intuition behind treating column embeddings and positional encoding as substitutes, i.e., could they not have complementary strengths (e.g., could PEs be designed in a way to encode information not captured in column embeddings, table metadata embedding?) due to which the model might benefit from usage of both PE, col. embeddings in conjunction with metadata embeddings?\n3. How does the proposed CTSyn method perform in the context of missing column values in the training / fine-tuning tables? Has this investigation been conducted?\n4. What is the intuition behind the result in Table 2a where F1 scores achieved by a classifier trained on CTSyn data outperforms the performance of a classifier trained on real data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and the proposed technique is detailed clearly. \n- The results clearly demonstrate the prowess of CTSyn in terms of matching the training data at least as captured by column-wise statistical metrics (Table 2). \n- Figure 2 and Table 3 demonstrate the ability of CTSyn to maintain privacy of training data (compared to other non differentially-private synthetic tabular data generators) while still learning useful representations. This demonstrates that the model is capable of balancing learning rich / accurate representations of the tabular data without replicating training data.", "weaknesses": "- The training process employing a diffusion model requires costly pre-training and fine-tuning hence scaling the modeling pipeline to large tables (e.g., 100s of columns, millions of rows) may be challenging.\n\n- One crucial facet of the paper that is lacking clarification is a description of the meta-data for the various tables employed. \n\n- Further, as the current model is termed as a foundation model for tabular data generation, it is crucial to demonstrate its effectiveness on noisy training / fine-tuning tabular data. The noise-free nature of datasets (at least during fine-tuning) cannot be guaranteed and an investigation of the robustness of the proposed technique to noisy training data is necessary but not presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aiIlyAlP64", "reviewer": "Reviewer_PYmU", "summary": "The paper presents CTSyn, a generative model designed for synthesizing tabular data by unifying diverse datasets into a single latent space.\nThe proposed architecture combines a cross-table variational autoencoder model with a conditional diffusion model to generate flexible synthetic data for various domains. Extensive pre-training across real-world datasets enables CTSyn to outperform existing models in data diversity and fidelity, also enhancing downstream machine learning tasks, especially in low-data regimes, even *surprisingly* surpassing real data.", "questions": "- The statement *\"synthesizers cannot add information not included in the original training data\"* may not fully capture the recent advances in synthetic data generation that utilize pre-trained LLMs for transfer learning. Specifically, methods like TabTab and LaTable, which are pre-trained on extensive tabular datasets, leverage LLMs' vast prior knowledge, extending the potential of data generation beyond what’s strictly in the training dataset. This aligns with approaches like GReaT, which explicitly incorporate pre-trained LLMs to enable effective transfer learning across tabular domains. To further clarify the novelty of CTSyn, it would be helpful if the authors addressed these points, including a comparison to methods that already exploit LLM-based architectures for tabular data synthesis. Additionally, on Line 099, the authors mention that existing methods struggle to capture \"intrinsic structural properties of tables\" and continuous values when using LLMs. Could the authors provide more context on this limitation? Recent studies indicate that fine-tuning can improve LLM performance for modeling continuous values, potentially overcoming some of these structural challenges [1]. \n\n - CTSyn employs quantile transformation for handling numerical values, but it’s unclear why this choice was made over alternative scaling methods, such as min-max or standard normalization. Quantile transformation can improve handling of skewed distributions but might also introduce artifacts in certain cases. Could the authors elaborate on the rationale behind selecting quantile transformation? \n\n- Eq. 1 introduces the sequence $\\mathbf{E}$, representing tokenized and embedded table values, but the meta embedding $e_m$ is excluded. Since $e_m$ likely contains crucial table-specific metadata, incorporating it directly into $\\mathbf{E}$ might strengthen the alignment between table content and contextual information\n\n-  Table 2 presents a *column-wise comparison* of synthetic and real data, which provides some indication of CTSyn’s data fidelity. However, this approach does not fully address the challenges of synthetic data generation, particularly regarding joint modeling of column relationships and complex dependencies. For synthetic data to be *realistic*, it must ideally preserve not just marginal distributions but also higher-order dependencies and potential causal relationships between columns. Could the authors consider additional evaluation metrics, such as multivariate measures or metrics assessing the preservation of conditional distributions, to more rigorously demonstrate CTSyn’s capability for joint data modeling? \n\n- In Fig. 2, the authors show that synthetic data generated by CTSyn outperforms real data in certain tasks, which is a quite surprising result in my view. How can authors elaborate on that? Please provide detailed explanation of the experiment settings here. \n\n- What is the final parameter size for the CTSyn model?\n\n\n[1] Akhtar, Mubashara, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, and Elena Simperl. \"Exploring the numerical reasoning capabilities of language models: A comprehensive analysis on tabular data.\" arXiv preprint arXiv:2311.02216 (2023).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Relevance. CTSyn focuses on the important area of generating high-quality tabular data, which is essential for improving machine learning models. Synthetic data can help when real data is limited or sensitive, such as in healthcare and finance.\n\n- Novel Architecture Design. The model features a new design that combines a cross-table variational autoencoder with a conditional diffusion model. This approach effectively maintains data quality and flexibility across different types of tables, significantly enhancing the capabilities of previous methods in this field.\n\n- Solid Baseline and Dataset Choice. CTSyn compares its performance with SOTA models and uses large, diverse datasets from the real world.", "weaknesses": "- CTSyn cannot handle datasets where columns include free text (e.g., in medical records with text entries), limiting its application to purely structured numerical or categorical data. \n\n- The evaluation lacks certain robustness tests for synthetic data quality, such as a discriminator test to assess whether a surrogate model can distinguish between real and synthetic data, or metrics like \"distance to closest record\" for authenticity. Additionally, the experimental setup for machine learning utility (e.g., cross-validation specifics) needs more transparency and detail to fully validate CTSyn's claims of enhancing downstream tasks.\n\n- A more comprehensive ablation study is needed to assess the impact of each component in CTSyn’s architecture, such as the specific roles of the cross-table variational autoencoder and the conditional diffusion model in enhancing data fidelity and adaptability. Including these would offer clearer insights into model design choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)