Title: TabMT: Generating tabular data with masked transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Tabular Data

Deep Learning

Generative Modeling

Transformers

Masked Transformers

Synthetic data
Abstract: Autoregressive and Masked Transformers are incredibly effective as generative models and classifiers.
    While these models are most prevalent in NLP, they also exhibit strong performance in other domains, such as vision. 
    This work contributes to the exploration of transformer-based models in synthetic data generation for diverse application domains. 
    In this paper, we present TabMT, a novel Masked Transformer design for generating synthetic tabular data. 
    TabMT effectively addresses the unique challenges posed by heterogeneous data fields and is natively able to handle missing data. 
    Our design leverages improved masking techniques to allow for generation and demonstrates state-of-the-art performance from extremely small to extremely large tabular datasets. 
    We evaluate TabMT for privacy-focused applications and find that it is able to generate high quality data with superior privacy tradeoffs.

TabMT: Generating Tabular data with Masked Transformers

Abstract

Autoregressive and Masked Transformers are incredibly effective as generative models and classifiers. While these models are most prevalent in NLP, they also exhibit strong performance in other domains, such as vision. This work contributes to the exploration of transformer-based models in synthetic data generation for diverse application domains. In this paper, we present TabMT, a novel Masked Transformer design for generating synthetic tabular data. TabMT effectively addresses the unique challenges posed by heterogeneous data fields and is natively able to handle missing data. Our design leverages improved masking techniques to allow for generation and demonstrates state-of-the-art performance from extremely small to extremely large tabular datasets. We evaluate TabMT for privacy-focused applications and find that it is able to generate high quality data with superior privacy tradeoffs.

1 Introduction

Generative models have attracted significant attention in the field of deep learning due to their ability to synthesize high-quality data and learn the underlying structure of complex datasets. Such models have been successfully applied to various data types including images [18], text[8], and tabular data [12]. This work concentrates on tabular data, which is prevalent in numerous fields like healthcare, finance, and social sciences. The heterogeneous nature of tabular data, characterized by its diverse data types, distributions, and relationships, presents distinct challenges not present in other domains.

The development of effective synthetic tabular data generators is crucial for numerous reasons including: privacy preservation, data augmentation, model interpretability, and anomaly detection. Prior work in this domain has produced a myriad of generative models, including Generative Adversarial Networks (GANs)[28][25], Variational Autoencoders (VAEs)[25], Autoregressive Transformer[22] [1], and Diffusion models[12]. Although these existing models strive to address the challenges associated with tabular data generation, there is still room for exploration and improvement. Specifically, we demonstrate improvements in robustness, scalability, privacy preservation, and handling of missing data.

Transformers [23], originally designed for natural language processing (NLP) tasks, have lead to significant advancements in a variety of applications. Their powerful capacity for modeling complex dependencies and

Paul F. Roysdon

AI Accelerator Leidos Inc

Categorical

Fields

Numerical

Fields

Quantizer

Embedding

Ordered

Embedding

Masking

𝑃 𝐹

!,#

= 𝑚

= 𝑝

!

𝑝

!

~𝑈(0,1)

Transformer

Dynamic

Linear

Predicted

Values

generalizing across applications has spurred researchers to extend transformers to other data types, such as images[11][4] and audio[17].

In this paper, we investigate transformers as synthetic tabular data generators, further expanding their cross-domain applicability. We specifically examine Masked Transformers (MT), originally developed to produce strong text embeddings [5], and successfully generalized across many domains[9][3][10]. We explore their utility as tabular data generators. We show Masked Transformers make robust and scalable tabular generators, achieving state-of-the-art performance across a wide array of datasets.

Our key contributions are as follows:

We propose TabMT, see Figure 1, a simple but effective Masked Transformer design for generating tabular data, that is general enough to work across many tasks and scenarios.

We provide a comprehensive evaluation of TabMT, demonstrating state-of-the-art performance when compared to existing generative model families, including GANs, VAEs, Autoregressive Transformers, and Diffusion models. We also showcase its scalability from very small to very large tabular datasets.

We highlight the applicability of our model in privacy-focused applications, illustrating TabMTs ability to arbitrarily trade-off privacy and quality through temperature scaling. Furthermore, our model's masking procedure enables it to effectively handle missing data, thereby increasing privacy and model applicability in real-world use cases.

2 Related Work

Language modeling: Transformer[23] based Language Modeling[5][8] learns Human language via token prediction. Given a context of either previous tokens in time, or a random subset tokens within a window, the model is tasked with predicting the remaining tokens. Using either an autoregressive model or masked model, respectively. Both these paradigms have shown success across a wide range of tasks and domains.[9][4][17][26][3] Masked Language models are traditionally used for their embeddings, although some papers have explored masked generation of content[3]. Our work builds on Masked Training and demonstrates its effectiveness for modeling and generating tabular data.

Deep Tabular Generators: Deep learning models are increasingly utilized for generating synthetic tabular data. Synthetic data is particularly important for tabular data as it is often subject to privacy requirements. Additionally, tabular datasets are often hard to acquire, and usually smaller than datasets in other domains. These conditions increase the importance and challenges associated with generating tabular data. Deep Tabular Generators have been constructed using essentially all Deep Generative model families Autoencoders[25], GANs[25][28][27][7], and AR transformers [22] [1] [16]. Our work uses Masked Transformers and demonstrates superior performance when compared to prior deep tabular generators.

Netflow Generators: Netflow data is a specific type of tabular data that captures network communication events and is commonly used for network traffic analysis, intrusion detection, and cybersecurity applications. Netflow datasets are typically extremely large, with complex rules between fields, and a high number of possible values per field. Generating realistic synthetic netflow data is crucial for developing and testing network monitoring tools and security algorithms. There have been several models developed specifically for netflow generation [21] [24] [15]. We demonstrate that our general tabular data generator handily outperforms domain specific models.

3 Method

TabMTs structure is particularly well suited for generating tabular data, for a number of reasons.

TabMT accounts for patterns bidirectionally between fields. Tabular data lacks ordering, meaning bidirectional learning likely produces better understanding and embeddings within the model.

A 'prompt' to a tabular generator is not likely sequential. TabMT's masking procedure allows for arbitrary prompts to the model during generation. This is unique as most other generators have very limited conditioning capabilities.

Missing data is far more common in tabular data than in other domains. TabMT is able to learn missing values by setting their masking probability to 1. Other generators require that we impute data separately before we can generate high-quality cleaned samples.

These structural advantages come from TabMT's novel masked generation procedure.

[mask]

[mask]

[mask]

[mask]

[mask]

Male

[mask]

[mask]

37

Male

2.3

[mask]

37

Male

2.3

-1

[mask]

Male

2.3

[mask]

sample

sample

sample

sample

Below we outline how we construct TabMT from the original masked training procedure outlined in BERT[5] and the justifications behind our design choices. We also outline the specific changes we make to allow for heterogeneous data types. A naive adaptation of BERT's masking procedure would look as follows. Given an n by l dataset F of categorical and numerical features, for each row F i , the transformer is provided with a set of unmasked fields F u i and a set of masked fields F m i . Each field in the masked set F m i has its value replaced with a mask token. The model is then tasked with predicting the original value for all masked tokens. The row F i is partitioned into the unmasked and masked sets by conducting a Bernoulli trial on each field, F i,j , such that P ( F i,j ∈ F m i ) = 0 . 15 .

The BERT masking procedure produces a strong embedding model, but not a strong generator. To create a strong generative model we make two key changes: sample our masking probability from a uniform distribution and predict masked values in a random order during generation. To understand why these changes are effective, we can look at the distribution of masked sets. As a result of the repeated Bernoulli trials during masking, the size of the masked set for each row | F m i | will follow a Binomial distribution. However, when generating data one field F i,j at a time, the model will inference on masked subset sizes (0 , . . . , l -1) , once each. We would like the training distribution of | F m i | to be uniform, matching the uniform distribution encountered when generating data. With a fixed masking probability P ( F i,j ∈ F m i ) = p m we will instead encounter a Binomial distribution centered around p m · l . However, if we sample our masking probability p m for each row F i such that P ( p m = p ) ∼ U (0 , 1) , we will train uniformly across subset sizes:

Fixing this train and inference mismatch is critical to forming a strong generator. A traditional autoregressive generator would generate fields ( F i, 0 , . . . , F i,l -1 ) , sequentially. However, tabular data, unlike language, does not have an inherent ordering. Generating fields in a fixed order introduces another mismatch between training and inference. During training F m i will take on the distribution

When generating in a fixed order, the model will infer across l distinct subsets and no others. However, if we instead infer in a random order, then at generation step 0 ≤ t < l , the distribution of F m i will be given by

Since we encounter each t exactly once, this overall distribution is identical to the masking distribution encountered during training, fixing the discrepancy caused by generating fields in a fixed order.

A transformer model will typically have an input embedding matrix E ∈ R k × d , where k is the number of unique input tokens and d is the transformer width. Because tabular data is heterogeneous, we instead construct l embedding matrices, one for each field. Each embedding matrix will have a different number of unique tokens k .

For categorical fields we use a standard embedding matrix initialized with a normal distribution. For each continuous field we construct an ordered embedding O ∈ R k × d from its unordered embedding matrix E and two endpoint vectors l , h ∈ R d .

To construct each ordered embedding matrix O , we first we quantize the values of the continuous field. Our default quantizer is K-Means. We consider the maximum number of clusters a hyper-parameter. Let v ∈ R k be the vector of ordered cluster centers. We construct an vector of ratios r ∈ R k using min max normalization such that

We use the ratio vector r to construct each ordered embedding in O :

This structure allows the transformer to both take advantage of the ordering of the properties and add unordered embedding information to each cluster. The unordered embeddings are useful in attention, multi-modal distributions, and encoding semantic separation between close values. We use this same structure to construct a dynamic linear layer at the output during prediction. This can be converted to a traditional linear layer once the model is trained.

Relying too heavily on the unordered embeddings might negate the benefit of our ordered embedding, as information isn't effectively shared between close values. To address this, we bias TabMT to rely on the ordering of embeddings. For continuous fields, we zero-init the unordered embedding matrix E . Whereas, the endpoint vectors l and h use a normal distribution with 0.05 standard deviation. Because entries in matrix O are not independent of each other, to sharpen the output distribution, the network must either rely on matrix E or increase magnitude of the endpoint vectors. This can reduce use of the priors encoded by O or cause instability. To combat this, we include a learned temperature which can sharpen the predicted distribution using a single parameter per field instead. Each field's predicted distribution ˆ y ∈ R k is given by

where z ∈ R k is a vector of logits, τ l is the learned temperature, and τ u is the user-defined temperature. See Figure 1 for an overall diagram of these components. Figure 2 shows the generation of a single sample. Detailed pseudocode is available in the Appendix.

4 Evaluation

In this section, we present a comprehensive evaluation of TabMT's effectiveness across an extensive range of tabular datasets. Our analysis involves a thorough comparison with state-of-the-art approaches, encompassing nearly all generative model families. To ensure a robust assessment, we evaluate across several dimensions and metrics.

Datasets: For our data quality and privacy experiments we use the same list of datasets and data splits as TabDDPM[12]. These 15 datasets range in size from ∼ 400 samples to ∼ 150 , 000 samples. They contain continuous, categorical, and integer features. The datasets range from 6 to 50 columns. For our scaling experiments we use the CIDDS-001[20] dataset, which consists of Netflow traffic from a simulated small business network. A Netflow consists of 12 attributes, which we post-process into 16 attributes; see the Appendix for more details. This dataset is extremely large with over 30 million rows and field cardinalities in the tens of thousands. Other datasets listed all have cardinalities below 50. Unlike our other benchmarks, we purposely do not quantize the continuous variables here to further test the scaling of our model. In other words, every unique value is treated as a separate category in our prediction process.

Prior Methods: We select four techniques to compare against, one from each each major family of deep generative models.

TVAE [25] is one of the first deep tabular generation papers introducing two models, a GAN and a VAE. We compare against their VAE because it is the strongest VAE tabular generator we are aware of.

CTABGAN+ [28], at the time of writing, is the state-of-the-art for GAN-based tabular synthesis.

TabDDPM [12] adapts diffusion models to tabular data, with the strongest results of all prior work.

RealTabFormer [22] is a recent work on adapting autoregressive transformers to tabular and relational data. This method is most similar to our technique, however, they use an autoregressive transformer which demonstrates worse results than our masked transformer.

4.1 Data Quality

We use the CatBoost variant of ML Efficiency (MLE)[27][12] for evaluating the quality of our synthetic data. This metric trains a CatBoost[6] model on the synthetic data instead of a weak ensemble. The CatBoost model is able to detect subtle patterns in the data, that weak classifiers cannot. This is a holistic metric that accounts for both diversity and quality of samples. For a fair comparison we use the the standard hyper-parameter tuning budget of 50 trials[28][25]. Our full search space is provided in the Appendix. Each data quality experiment was conducted using a single A10 GPU each. For evaluation, we generate scores and standard deviations 1 on the test set, training a CatBoost model 10 times on 5 samples of synthetic data.

MLE scores are presented in Table 1; note that we match or exceed state-of-the-art on 11 of 15 datasets. To gain a qualitative understanding of data quality we visualize the distribution of correlation errors; see Figure 3. To calculate the correlation errors, we first compute the correlation r i,j between each pair of fields ( i, j ) . To compute correlations involving categorical columns, we convert them to one-hot vectors. We then compute the correlation between columns on the synthetic data ˆ r i,j . The correlation error is the absolute difference between these two values | r i,j -ˆ r i,j | . These errors should approach zero because we expect the correlations between columns in the synthetic data to be the same as those in the real data. Erroneous and missing correlations appear as non-zero values in the histograms of Figure 3.

0.00

0.26

AD

TabDDPM

0.00

0.23

BU

TabDDPM

0.00

0.02

CA

TabDDPM

0.00

0.47

CAR

TabDDPM

0.00

0.42

DI

TabDDPM

0.00

0.55

KI

TabDDPM

0.00

0.16

CTabGAN+

0.00

0.24

CTabGAN+

0.00

0.19

CTabGAN+

0.00

0.23

CTabGAN+

0.00

0.06

CTabGAN+

0.00

0.22

CTabGAN+

4.2 Privacy and Sample Novelty

Maintaining privacy of the original data is a key application for synthetic tabular data. Machine learning is increasingly being used across a wide range of areas to produce valuable insights. Simultaneously, there is a rapid rise in both regulation and privacy concerns that need to be addressed. In Section 4.1 we demonstrated that data produced by TabMT is high enough quality to produce strong classifiers. Now we evaluate our model for privacy. This evaluation complements our quality evaluation and verifies that our model is generating novel data. Novelty means data is not substantially similar to samples encountered during training. A high-quality non-private model can trivially be formed by directly reproducing the training set exactly. None of this data is novel, but it is high quality. By ensuring our model is both private and high quality, we verify that our model has learned the distribution of the data, and not simply memorized the training set. Memorization is a larger issue in tabular data due to smaller dataset sizes and increased privacy concerns.

To evaluate privacy and novelty we adopt the median Distance to the Closest Record (DCR) score. To calculate the DCR of a synthetic sample, we find the nearest neighbor in the real training set by Euclidean distance. We report the median of this distance across our synthetic samples. Data with higher DCR scores will be more private and more novel. There is an inherent trade off between privacy and quality. Higher quality samples will tend to be closer to points in the training set and vice versa.

While models such as CTabGAN+[28] and TabDDPM[12] have a fixed trade-off between privacy and quality after training. TabMT can trade-off between quality and privacy using temperature scaling. By walking along the Pareto curve of our model, using temperature scaling, we can controllably tune the privacy and novelty of our generated data per application. By increasing a field's temperature, its generated values become more novel and private, but they are also less faithful to the underlying data distribution. The trade off between the quality and privacy here form a Pareto front for TabMT on each dataset.

We use a separate temperature for each field and perform a search to estimate the Pareto front. Each search was conducted using a single A10 GPU each. Search details are available in the Appendix. In Table 2, we compare TabMT's DCR and corresponding MLE scores to that of TabDDPM. We are always able to attain a higher DCR score, and in most cases a higher MLE score as well. This falls in line with recent results in other domains showing diffusion models are less private than other generative models[2]. A comparison with CTabGAN+ is available in the Appendix, compared to CTabGAN+ we obtain both higher privacy and MLE scores in all tested cases. Figure 4 shows the Pareto fronts of TabMT across several datasets.

4.3 Missing Data

Real world data is often missing many values that make training difficult. When a row has a missing value we must either drop the row, or find a method to impute the missing value. Other techniques such as the RealTabformer[22] or TabDDPM[12] cannot natively handle real world missing data, and must either use a different imputation technique or drop the corresponding rows. Our masking procedure allows TabMT to natively handle arbitrary missing data. To demonstrate this, we randomly drop 25% of values from the dataset, ensuring nearly every row is permanently missing data. Nevertheless, our

DS

TabDDPM

AB

0.050(0.550)

0.249

(0.533)

AD

0.104(0.795)

1.01

(0.811)

BU

0.143(0.906)

0.165

(0.908)

CA

0.041(0.836)

0.117

(0.832)

CAR

0.012(0.737)

0.041

(0.737)

CH

0.157(0.755)

0.281

(0.758)

DI

0.204(0.740)

0.243

(0.740)

FB

0.112(0.713)

0.252

(0.787)

DS

TabDDPM

GE

0.059(0.597)

0.234

(0.599)

HI

0.449(0.722)

0.483

(0.727)

HO

0.086(0.677)

0.151

(0.607)

IN

0.041(0.809)

0.061

(0.816)

KI

0.189(0.833)

0.335

(0.868)

MI

0.022(0.936)

0.026

(0.936)

WI

0.016(0.904)

0.063

(0.881)

0.77

0.78

0.79

0.80

0.81

0.82

Validation Score

1.0

1.1

1.2

1.3

1.4

1.5

Privacy Score

AD

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Validation Score

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

Privacy Score

CA

0.731

0.732

0.733

0.734

0.735

0.736

0.737

Validation Score

0.04

0.05

0.06

0.07

0.08

0.09

Privacy Score

CAR

0.72

0.73

0.74

0.75

0.76

0.77

0.78

0.79

Validation Score

0.32

0.34

0.36

0.38

0.40

Privacy Score

CH

0.55

0.60

0.65

0.70

0.75

Validation Score

0.18

0.20

0.22

0.24

0.26

0.28

0.30

Privacy Score

DI

0.860

0.865

0.870

0.875

0.880

0.885

0.890

0.895

0.900

Validation Score

0.4

0.5

0.6

0.7

Privacy Score

KI

model is still able to train, producing synthetic rows with no missing values in them. This facilitates training on real world data.Table 3 shows our accuracy when training with missing data.

Additionally, our model can be arbitrarily conditioned to produce any subset of the data distribution at no additional cost, allowing us to more effectively augment underrepresented portions of data. Prior art is largely incapable of conditioning when producing outputs.

4.4 Scaling

In this section, we examine the performance of our model when scaling to very large datasets. We use the CIDDS-001 dataset[20] as our benchmark dataset. We do not use anomalous traffic from the dataset, and randomly select 5% of the dataset as the validation set for reporting results. The results

in this section, together with those in Section 4.1, demonstrate that TabMT is both sample-efficient enough to learn with just a few hundred samples, while remaining general enough to scale to over thirty million samples. We train three model sizes on this dataset and compute metrics on the resulting samples. The model topologies are outlined in Table 4. Each model was trained on a single A10 GPU with the exception of TabMT-L which was trained using 4 V100s. We use the AdamW[14] optimizer with a learning rate of 0.002 and weight decay of 0.01, a batch size of 2048 and a cosine annealing learning rate schedule for 350,000 training steps and 10000 warm-up steps.

Because ML Efficiency and DCR are very costly to compute on a dataset of this scale, we instead adapt Precision and Recall[13] to the tabular domain. The original definitions of these metrics[13] rely on vision models to produce the embeddings used. We find our masking procedure produces strong embeddings for each sample, so we use the embeddings produced by TabMT-S. Specifically, we average the embeddings across the fields to produce 64 dimensional for each flow. We used a fixed neighborhood size of k = 3 . We include an additional diversity metric defined as the average set coverage across all properties of the generated data. Results in Table 5 demonstrate strong scaling and performance across model sizes. We can see Precision and Recall are both very close to that of the validation set. Sample diversity and quality both scale as model size increases.

We compare against the prior state-of-the-art NetflowGAN, or NFGAN[21]. This GAN was tuned specifically for this dataset. It is trained in two phases. First IP2Vec[19] is trained to produce Netflow embeddings. These embeddings are then used as targets for the generator during GAN training. Results from NFGAN are shown in Table 5. We can see that NFGAN obtains reasonably high precision, but poor recall and diversity. This is because the model suffers from mode collapse, producing samples in only a small portion of the full distribution.

Netflow has both correlations between the fields and complex invariants between fields. We can measure the violation rate of these invariants to understand how well our model is detecting patterns within the data. We measure against seven invariants proposed in [21]. As shown in Table 6 TabMT produces substantially more diverse data, while achieving a median 20x improvement in violation probability over NFGAN.

5 Limitations and Future Work

TabMT presents strong results even on large datasets, but our Transformer backbone means TabMT is slower than more lightweight methods built around small MLPs, or GANs which can produce a row in a single inference. Searching temperatures also adds time if optimal privacy is needed. Additionally, we must quantize continuous fields, while we outperform methods which do not quantize fields, this could pose issues in some applications. Future work might examine learning across tabular datasets, alternative masking procedures and networks to improve speed, or integration with diffusion models to better tackle continuous fields.

Broader Impact Synthetic data generation allows for privacy preservation, protecting sensitive data while still enabling data analysis. High Quality synthetic data may ease the pressure to resort to unethical methods of collection such as relying on underpaid labor. With this in mind, trading off data for additional compute does mean that the additional compute will contribute to increased CO2 emissions. Additionally, synthetic data carries the risk of misuse, such as the potential for manipulating results or research findings with fabricated data. All experiments were conducted using cloud A10 or V100 GPUs. For algorithm design and experiment result generation roughly 410 GPU days of compute were used.

6 Conclusion

In this paper, we outlined a novel Masked Transformer design and training procedure, TabMT, for generating synthetic tabular data. Through a comprehensive series of benchmarks we demonstrate that our model achieves state-of-the-art generation quality. This quality is verified at scales that are orders of magnitude larger than prior work and with missing data present. Our model achieves superior privacy and is able to easily trade off between privacy and quality. Our model is a substantial advancement compared to previous work, due to its scalability, missing data robustness, privacypreserving generation, and superior data quality.

References

Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280 , 2022.

Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188 , 2023.

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11315-11325, 2022.

Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning , pages 1691-1703. PMLR, 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Catboost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363 , 2018.

Justin Engelmann and Stefan Lessmann. Conditional wasserstein gan-based oversampling of tabular data for imbalanced learning. Expert Systems with Applications , 174:114582, 2021.

Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines , 30:681-694, 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems , 35:28708-28720, 2022.

Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems , 34:14745-14758, 2021.

Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. arXiv preprint arXiv:2209.15421 , 2022.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems , 32, 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

Liam Daly Manocchio, Siamak Layeghy, and Marius Portmann. Flowgan-synthetic network flow generation using generative adversarial networks. In 2021 IEEE 24th International Conference on Computational Science and Engineering (CSE) , pages 168-176. IEEE, 2021.

Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3565-3569. IEEE, 2021.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

Markus Ring, Alexander Dallmann, Dieter Landes, and Andreas Hotho. Ip2vec: Learning similarities between ip addresses. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW) , pages 657-666, 2017. doi: 10.1109/ICDMW.2017.93.

Markus Ring, Sarah Wunderlich, Dominik Grüdl, Dieter Landes, and Andreas Hotho. Flow-based benchmark data sets for intrusion detection. In Proceedings of the 16th European conference on cyber warfare and security. ACPI , pages 361-369, 2017.

Markus Ring, Daniel Schlör, Dieter Landes, and Andreas Hotho. Flow-based network traffic generation using generative adversarial networks. Computers & Security , 82:156-172, 2019.

Aivin V Solatorio and Olivier Dupriez. Realtabformer: Generating realistic relational and tabular data using transformers. arXiv preprint arXiv:2302.02041 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Maximilian Wolf, Markus Ring, and Dieter Landes. Impact of generative adversarial networks on netflowbased traffic classification. In 13th International Conference on Computational Intelligence in Security for Information Systems (CISIS 2020) 12 , pages 393-404. Springer, 2021.

Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in Neural Information Processing Systems , 32, 2019.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 , 2022.

Zilong Zhao, Aditya Kunar, Hiek Van der Scheer, Robert Birke, and Lydia Y. Chen. CTAB-GAN: effective table data synthesizing. CoRR , abs/2102.08369, 2021. URL https://arxiv.org/abs/2102.08369 .

Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y Chen. Ctab-gan+: Enhancing tabular data synthesis. arXiv preprint arXiv:2204.00401 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TabMT, a new Masked Transformer architecture designed for generating synthetic tabular data. While Transformers are predominantly used in natural language processing (NLP), TabMT demonstrates their effectiveness in dealing with heterogeneous data fields, such as images and tables, and efficiently manages missing data. The authors propose to sample the masking probability from a uniform distribution and predict masked values in random order during generation. By employing advanced masking techniques, TabMT can generate synthetic data with high performance across a wide range of dataset sizes. Moreover, the model proves valuable in privacy-sensitive applications, as it is capable of producing high-quality data while adhering to privacy restrictions.\n\nSTRENGTHS:\nThe motivation is easy to understand and the problem is important but less explored than common domains. It is not straightforward to apply existing techniques on images and texts to the tabular data.\nThe experimental designs are comprehensive, including several sections assessing data quality, Privacy and Sample Novelty, Missing Data, and Scaling, which are solid.\n\nWEAKNESSES:\nThe motivations for designs are not very clearly written. Please consider reorganizing the structure, adding highlighted paragraphs, and moving line 129-138 to the beginning of the method or merging them with the methodology introduction texts.\n\nThe introduction of temperature scaling is not well-motivated, is it for privacy purposes and why will it work?\n\nThe introduction of privacy tabular data generation is not detailed. The conclusion claims that \"the model is able to function under arbitrary privacy budgets.\" but I even didn't see how the budgets are computed and its defitions, motivations etc. Same for the definition of data \"novelty\". And the questions of \"why many designs in TabMT can help those aspects are not well explained\".\n\nQUESTIONS:\nThe work can benefit from discussing and comparing the work of causal inference using transformers, which is a particular kind of missing data generation problem where privacy protection is also pressing, e.g. [1], [2].\n\n[1] Exploring Transformer Backbones for Heterogeneous Treatment Effect Estimation\n[2] Differentially Private Synthetic Control\n\nLIMITATIONS:\nSee the weaknesses part", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

comments: SUMMARY:
The paper proposes a masked transformer model that can be used to generate tabular data. The authors propose modifications in the masking strategy in the transformer, in order to make it more effective for generating data. Empirically, the proposed generator improves performance in various datasets compared to other baselines. Moreover, the paper presents cases of privacy-focused applications, presence of missing values, and large datasets to further depict the usability of the proposed method in real-world situations.

STRENGTHS:
- The paper is well-written and very clear to its points. The authors define the problem with the masked transformer for generating tabular data, and presents an effective method to overcome the difficulty. The empirical results are well presented, and seems promising.

WEAKNESSES:
As noted in the paper, the major weakness would be the resource it takes to train the model. For general users, a pre-trained model with learning across multiple tables might be beneficial.

QUESTIONS:
- Is it possible for the model to generate new categories (or numbers) not present in the train set?

LIMITATIONS:
The presented work might not be accessible to regular users, given the amount of resource required to train the model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes a new generative model of table-type data based on the transformer.  And it can address the unique challenges posed by heterologous data fields and natively handle missing data.

STRENGTHS:
1. TabMT is a simple but effective Masked Transformer design for generating tabular data.
2. We highlight the applicability of our model in privacy-focused applications, illustrating TabMTs ability to arbitrarily trade-off privacy and quality through temperature scaling.
3. Experiments show the effectiveness of the proposed method.

WEAKNESSES:
1. The TabMT structure has only been modified in terms of the input layer, without improving the multi-layer transformer structure. In other words, this paper only made adaptive improvements to the data format and is not on par with NeurIPS.
2. The application method of the model structure proposed by TabMT is not clear. Categorical and Numerical use independent inputs, which makes it necessary to determine the type of data input into the model. How to accurately and automatically identify this type is challenging, which also limits the complexity of generating table data for the model, making it difficult to cope with complex scenarios. The effectiveness of the model should be evaluated for inaccurate category recognition in the experiment.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the effectiveness of masked transformers as generative models for synthetic tabular data generation. The proposed TabMT architecture effectively handles challenges related to heterogeneous data fields and missing data. The model shows promising experimental performance and demonstrates good performance even under privacy constraints.\n\nSTRENGTHS:\n- The paper proposes a promising masked transformer approach TabMT for generating tabular data. This is an elegant application of masked transformers.\n- The paper provides comprehensive experimental evaluation and compares to a diverse set of baselines on 15 datasets\n- The paper explores privacy-preserving capabilities of TabMT and shows promising results\n\nWEAKNESSES:\n- While the paper provides adequate experimental evaluation, it would be helpful to include recent state-of-the-art generative methods in evaluation such as STaSy [1]\n- In the privacy experiments, only TabDDPM was used as the baseline, it would be interesting to see more comparisons with other baselines\n\n[1] Kim, J., Lee, C. and Park, N., 2022. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018.\n\nQUESTIONS:\n1. Could you please clarify the training procedure? When training the masked transformer, does every feature have its own classification head for predicting the mask?\n2. Since you explore hyperparameter tuning for the downstream Catboost model, how was hyperparameter tuning performed for each of the generative models?\n3. In the privacy experiments, only TabDDPM was used as the baseline, would it be possible to add more comparisons with other baselines?\n4. It would be great to add recent state-of-the-art generative models into evaluation, such as STaSy [1]\n\n[1] Kim, J., Lee, C. and Park, N., 2022. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018.\n\nLIMITATIONS:\nAddressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}