TabMT: Generating tabular data with masked transformers
Authors: Manbir S Gulati
Paul F Roysdon
Venue: NeurIPS 2023 poster (2023)
Keywords: Tabular Data
Deep Learning
Generative Modeling
Transformers
Masked Transformers
Synthetic data
TL;DR: Masked Transformers generate high-quality tabular data, scale well, have better privacy, and can handle missing data/conditional generation

Abstract: Autoregressive and Masked Transformers are incredibly effective as generative models and classifiers.
    While these models are most prevalent in NLP, they also exhibit strong performance in other domains, such as vision. 
    This work contributes to the exploration of transformer-based models in synthetic data generation for diverse application domains. 
    In this paper, we present TabMT, a novel Masked Transformer design for generating synthetic tabular data. 
    TabMT effectively addresses the unique challenges posed by heterogeneous data fields and is natively able to handle missing data. 
    Our design leverages improved masking techniques to allow for generation and demonstrates state-of-the-art performance from extremely small to extremely large tabular datasets. 
    We evaluate TabMT for privacy-focused applications and find that it is able to generate high quality data with superior privacy tradeoffs.

TabMT: Generating Tabular data with Masked Transformers
Manbir Gulati AI Accelerator Leidos Inc
Paul F. Roysdon AI Accelerator Leidos Inc
Manbir. Sumbir.S.co. com
Paul .owssoon@leedos..comm
Abstract
Autoregressive and Masked Transformers are incredibly effective as generative models and classifiers. While these models most prevalent in NLP, they also exhibit strong performance in other domains, such as vision. This work contributes to the exploration of cansformer-based models in synthetic data generation for diverse application domains. this paper, we present TabMT, a novel Masked Transformer design for generating synthetic tabular data. TabMT effectively ad- dresses the unique challenges posed by heterogeneous data fields and natively able handle missing data. Our design leverages improved masking techniques allow for generation and demonstrates state-of-the-art performance from extremely small to extremely large tabular datasets. We evaluate TabMT for privacy-foeuse applications and find that it is able to generate high quality data with superior privacy tradeoff's.
Introduction
Generative models have attracted significant attention in the field of deep learning due their ability to synthesize high-quality data and learn the underlying structure of complex datasets. Such models have been successfully applied various data types including images [18], text[8] and tabular data [12] This work concentrates tabular data, which is prevalent in numerous fields like healthcare, finance, and social sciences. The heterogencous nature of tabular data, characterized by its diverse data types, distributions, and relationships, presents distinct challenges not present in other domains.
Categorical Fields
Numerical Fields
Quanttizr
Ordered Embedding
Embedding
Masking
P(F, m) n P)00((0.1)
The development of effective synthetic tabular data generators crucial for numerous reasons including: privacy preservation, data augmentation, model interpretability, and anomaly detection. Prior work in this domain has produced myriad of generative models, including Generative Adver- sarial Networks (GANs) (25] Variational Autoencoders (VAFss 25 Autoregressive Transformer 22 and Diffusion models [12) Although these existing models strive to address the challenges associated with tab- ular data generation, there is still room for exploration and improvement. Specifically, we demonstrate improvements in robustness, scalability, privacy preservation, and handling of missing data.
Transformer
Dynamic
Linear
Predicted Values
Figure 1: Diagram of the mask
TabMT.
Transformers [23]. originally designed for natural language processing (NLP) tasks, have lead to significant advancements in variety of appli- cations. Their powerful capacity for modeling complex dependencies and
token, p. is the masking probability of the 7th row
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

generalizing across applications has spurred researchers to extend transformers to other data types, such as images [T] 4 and a
In this paper, we investigate transformers as synthetic tabular data generators, further expanding their cross-domain applicability. We specifically examine Masked Transformers (MT), originally developed to produce strong text embeddings [5). and successfully generalized across many domains [3] We explore their utility as tabular data generators. We show Masked Transformers make robust and scalable tabular generators, achieving state-of-the-art performance across a wide array of datasets.
Our key contributions
follows:
1. We propose TabMT, see Figure 1 simple but effective Masked Transformer design for generating tabular data, that is general enough to work across many tasks and scenarios.
2. We provide comprchensive evaluation of TabMT, demonstrating saat--of-the-ar perfor- mance when compared to existing generative model families, including GANs, VAEs, Autoregressive Transformers, and Diffusion models. We also showease its scalability from very small to very large tabular datasets.
3. We highlight the applicability of model in trvacy-focuseee applications, illustrating TabM''s ability to arbitrarily trade-off privacy and quality through temperature scaling. Furthermore, our model's masking procedure enables it to effectively handle missing data, thereby increasing privacy and model applicability in real-world use cases.
Related Work
Transformer (23) based Language Modeling learns Human language via token prediction. Given a context of either previous tokens in time, or random subset tokens within a window, the model is tasked with predicting the remaining tokens. Using either an autoregressive model masked model, respectively. Both these paradigms have shown success across a wide range of tasks and domains. [ [77 26) 3 Masked Language models are traditionally used for their embeddings, although some papers have explored masked generation content(3]. Our work builds on Masked Training and demonstrates its effectiveness for modeling and generating tabular data.
Language modeling:
Deep Tabular Generators: Deep learning models increasingly utilized for generating synthetic tabular data. Synthetic data is particularly important for tabular data as it is often subject to privacy requirements. Additionally, tabular datasets often hard acquire, and usually smaller than datasets in other domains. These conditions increase the importance and challenges associated with generating tabular data. Deep Tabular Generators have been constructed using essentially all Deep Generative model families Autoencoders(25 (25), GANs 25 28 [
[7], and AR transformers [22] [ [16.. Our work uses Masked Transformers and demonstrates superior performance when compared to prior deep tabular generators.
Netflow Generators: Netflow data is specific type of tabular data that captures network communi- cation events and is commonly used for network traffic analysis, intrusion detection, and cybersecurity applications. Netflow datasets are typically extremely large, with complex rules between fields, and high number possible values per field. Generating realistic synthetic netflow data is crucial for developing and testing network monitoring tools and security algorithms. There have been several models developed specifically for netflow generation [21] [2] [15.. We demonstrate that our general tabular data generator handily outperforms domain specific models.
3 Method
TabMTs structure is particularly well suited for generating tabular data, for number of reasons.
1. TabMT accounts for patterns bidirectionally between fields. Tabular data lacks ordering, meaning bidirectional learning likely produces better understanding and embeddings within the model.
2. A "prompt" to a tabular generator is not likely sequential. TabMT's masking procedure allows for arbitrary prompts to the model during generation. This unique as most other generators have very limited conditioning capabilities.

---

3. Missing data is far more common in tabular data than in other domains. TabMT is able to learn missing values by setting their masking probability to Other generators require that we impute data separately before we can generate high-quality cleaned samples.
These structural vanaagee come from TabMT's novel masked generation procedure.
sample
[mask]
[mask]
[mask]]
sample
[
Male
Male
Male
Mi 1
sample
[
[mask]
2.3
2.3
2.3
sample
[moss]]
[mask]
[mask]
[massk]
Figure 2: A row of data being sampled from TabMT. Fields are sampled in random order and field values are sampled according to the predicted distributed.
Below we outline how we construct TabMT from the original masked training procedure outlined in BERT(5) and the justifications behind our design choices. We also outline the specific changes make to allow for heterogeneous data types. A naive adaptation of BERT's masking procedure would look follows. Given an n by / dataset F of categorical and numerical features, for each row F,, the transformer provided with a set of unmasked fields F and a set masked fields F.". Each field in the masked set F." has its value replaced with mask token. The model is then tasked with predicting the original value for all masked tokens. The row F; is partitioned into the unmasked and masked sets by conducting Bernoulli trial each field, Fa,j such that P(F.,) i. FT 0.15.
The BERT masking procedure produces strong embedding model, but not strong generator. To create strong generative model we make two key changes: sample our masking probability from uniform distribution and predict masked values in a random order during generation. To understand why these changes are effective, we can look at the distribution of masked sets. As a result of the repeated Bernoulli trials during masking, the size of the masked set for each row FF" will follow Binomial distribution. However, when generating data one field Fr, at time, the model will inference on masked subset sizes (0.....I-1) 1). once each. We would like the training distribution of F to be uniform, matching the uniform distribution encountered when generating data. With fixed masking probability P(F.,) E F = Pm we will instead encounter a Binomial distribution centered around Pm I. However, if we sample masking probability pm for each row F, such that P(pm p) ~ (0, 1), we will train uniformly across subset sizes:
P(F [[] ())'0)1-))-
1! kIl K) (I 1)! 1+1
A!(1 - K)!
k)
(I)
Fixing this train and inference mismatch is critical to forming strong generator. A traditional autoregressive generator would generate fields (Fr.o..... F1,- sequentially. However, tabular data, unlike language, does not have inherent ordering. Generating fields in a fixed order introduces another mismatch between training and inference. During training F." will take on the distribution
P(F 5) (4)-
When generating in fixed order, the model will infer across distinct subsets and no others. However, if we instead infer in a random order, then at generation step 0 < t < I, the distribution of F." will be given by
P(F =) t!
t)! 1!
1
Since we encounter each exactly once, this overall distribution is identical to the masking distribution encountered during training, fixing the discrepancy caused by generating fields in fixed order.
transformer model will typically have an input embedding matrix E R where k is the number of unique input tokens and d is the transformer width. Because tabular data is heterogeneous, we instead construct embedding matrices, one for each field. Each embedding matrix will have different number of unique tokens k.
3

---

For categorical fields we use standard embedding matrix initialized with a normal distribution. For each continuous field we construct an ordered embedding O R from its unordered embedding matrix E and two endpoint vectors 1, h E IR
To construct each ordered embedding matrix O, we first we quantize the values of the continuous field. Our default quantizer is K-Means. We consider the maximum number of clusters a Let E g be the vector of ordered cluster centers. We construct an vector of ratios r IR using min max normalization such that
min(v)
max v) min(v) (v)
We use the ratio vector to construct each ordered embedding in O:
(5) This structure allows the transformer to both take advantage of the ordering of the properties and add unordered embedding information to each cluster. The unordered embeddings are useful in attention, multi-modal distributions, and encoding semantic separation between close values. We use this same structure construct a dynamic linear layer at the output during prediction. This can be converted to a traditional linear layer once the model is trained.
0, = E, + 1 + (1 - r.)) h.
Relying too heavily the unordered embeddings might negate the benefit of our ordered embedding, as information isn't effectively shared between close values. To address this, we bias TabMT to rely on the ordering of embeddings. For continuous fields, we zero-init the unordered embedding matrix E. Whereas, the endpoint vectors and h use normal distribution with 0.05 standard deviation. Because entries in matrix O are independent of each other, to sharpen the output distribution, the network must either rely on matrix E or increase magnitude of the endpoint vectors. This can reduce use of the priors encoded by or cause instability. To combat this, we include learned temperature which can sharpen the predicted distribution using a single parameter per field instead. Each field's s predicted distribution y R is given by
(T((T)))))
(6)
E, (T)()))))))
where E * is a vector of logits, 7. is the learned temperature, and is the user-defined temperature. See Figure for an overall diagram of these components. |Figure 2|shows the generation of a single sample. Detailed pseudocode is available in the Appendix.
4 Evaluation
In this section, we present a comprchensive evaluation of TabMT's effectiveness across an exten- sive range of tabular datasets. Our analysis involves a thorough comparison with state-of-the-ar approaches, encompassing nearly all generative model families. To ensure a robust assessment, we evaluate across several dimensions and metrics.
Datasets: For our data quality and privacy experiments use the same list of datasets and data splits as [2] These 15 datasets range in size from ~ 400 samples to ~ 150, 000 samples. They contain continuous, categorical, and integer features. The datasets range from to 50 columns. For our scaling experiments we use the CIDDS-001 [20) dataset, which consists of Netflow traffic from a simulated small business network. A Netflow consists of 12 attributes, which we post-process into 16 attributes; see the Appendix for more details. This dataset is extremely large with over 30 million rows and field cardinalities in the tens of thousands. Other datasets listed all have cardinalities below 50. Unlike our other benchmarks, we purposely do not quantize the continuous variables here to further test the scaling of our model. In other words, every unique value is treated as a separate category in our prediction process.
Prior Methods: We select four techniques to compare against, one from each each major family of deep generative models.
TVAE 25 is one of the first deep tabular generation papers introducing two models, a GAN and a VAE. We compare against their VAE because the strongest VAE tabular generator we are aware of.

---

MLE scores presented in Table 1 note that match exceed state-of-the-ar 11 of 15 datasets. To gain a qualitative understanding of data quality we visualize the distribution of correlation errors; see Figure To calculate the correlation errors, first compute the correlation rty between each pair of fields (1,j). To compute correlations involving categorical columns, we convert them to one-hot vectors. We then compute the correlation between columns the synthetic data P..j. The correlation error is the absolute difference between these two values rijj Pi.j These errors should approach zero because we expect the correlations between columns in the synthetic data be the CTABGAN+22), at the time of writing, is the state-of oftte- for GAN-based tabular synthesis.
TabDDPM 12 adapts diffusion models to tabular data, with the strongest results of all prior work.
RearerbFormerr [2] is recent work on adapting autoregressive transformers to tabular and relational data. This method most similar to our technique, however, they use an autoregressive transformer which demonstrates worse results than our masked transformer.
4.1 Data Quality
We use the CatBoost variant of ML Efficiency (M [2] for evaluating the quality of our synthetic data. This metric trains CatBoost[6] model on the synthetic data instead of weak ensemble. The CatBoost model is able to detect subtle patterns in the data, that weak classifiers cannot. This is holistic metric that accounts for both diversity and quality of samples. For fair comparison we use the the standard hyper- parameter tuning budget of 50 trials trials[28) [8][22), Our full search space is provided in the Appendix. Each data quality experiment was conducted using single A10 GPU each. For evaluation, we generate scores and standard deviations" on the test set, training a CatBoost model 10 times samples of synthetic data.
Table MLE and standard deviations across techniques. The highest MLE score for each dataset is highlighted in bold. Fj is used for classification. /22 is used for regression tasks. &; the source paper [22] cites a ~ 3% higher test accuracy using the real train set over what other papers achieve, likely because they used different split or version of this dataset.
DS
TVAE
CTabbGA+
Realfaab.
TabDDPM
TabMT
Real
AB
0.403
0.467+10004
0.504:10.01
0.550-10.010
0.535+10.004
0.556:10004
AD
0.701200002
0.702+00003
0.811=0.002
0.795110.001
0.814
0.815+0.002
BU
0.804
0.88410.000
0.928=0.003"
0.906110.003
0.908=1.002
0.906=00.002
CA
0.752-10.001 0.525=10004
0.808
0.836110.002
0.83811.002
0.857=0.001
CAR
0.717110000
0.7331100001
0.73711.001
0.73811.000
0.73810.001
CH
0.732+00006
0.702+00012
0.755+-00006
0.741+00005
0.740+0.009
DI
0.714
0.734+0.020
0.732++0027
0.740++0.020
0.769
0.785+00013
FB
0.685+000003
0.509+00011
0.771=0.004
0.713+
0.798+
0.837+0.001
GE
0.434.10.006
0.40610.009
0.597-10.006
0.605
0.636110.007
III
0.000
0.66410.002
0.722-10.001
0.727
0.724
HO
0.49310.006
0.50410.005
0.677
0.6191100004
0.002
IN
0.784
0.79710.005
0.8002
0.811
0.014
KI
0.804
0.444+00014
0.833+0.014
0.876=10.011
0.907
MI
0.912+00.011
0.89240.002
0.936+0.001
0.938
0.934100000
WI
0.50120.012
0.79810.021
0.904100009
0.881=000009
0.898+0.006
same as those in the real data. Erroncous and missing correlations appear as non-zero values in the histograms of Figure 3
For comparison, we use the same enror reporting method as prior work, however, we plan to present thorough coverage of more accurate error estimation for generative models in future work.
5

---

TA0OPN
TaceeT
TackeT
220
CREGAN-
CREGAMA
CRRGAN
BAMI
Figure 3: A comparison of Correlation Error Histograms between TabMT (Green) vs TP (Purple) and TabMT (Green) vs CTabGAN+ (Purple). A good generator should have correlation errors distributed close to zero (the left of the plot). We can see TabMT's correlation errors are consistently distributed closer to zero than either TabDDPM or CTabGAN+.
4.2 Privacy and Sample Novelty
Maintaining privacy of the original data is key application for synthetic tabular data. Machine learning is increasingly being used across wide range of areas to produce valuable insights. Si- multaneously, there is a rapid rise in both regulation and privacy concerns that need to be addressed. In Section 4.1 we demonstrated that data produced by TabMT is high enough quality to produce strong classifiers. Now we evaluate our model for privacy. This evaluation complements our quality evaluation and verifies that our model is generating novel data. Novelty means data is not substantially similar to samples encountered during training. A high-quality non-private model can trivially be formed by directly reproducing the training set exactly. None of this data is novel, but it is high quality. By ensuring our model is both private and high quality, we verify that our model has learned the distribution of the data, and not simply memorized the training set. Memorization is larger issue in tabular data due to smaller dataset sizes and increased privacy concerns.
To evaluate privacy and novelty adopt the median Distance to the Closest Record (DCR) score. To calculate the DCR of synthetic sample, we find the nearest neighbor in the real training set by Euclidean distance. We report the median of this distance across our synthetic samples. Data with higher DCR scores will be more private and more novel. There is an inherent trade off between privacy and quality. Higher quality samples will tend to be closer points in the training set and vice versa.
While models such CTabGAN+ (28) and T2 have fixed trade-off between privacy and quality after training. TabMT can trade-off between quality and privacy using temperature scaling. By walking along the Pareto curve of our model, using temperature scaling, we can controllably tune the privacy and novelty of our generated data per application. By increasing a field's temperature, its generated values become more novel and private, but they also less faithful to the underlying data distribution. The trade off between the quality and privacy here form a Pareto front for TabMT on each dataset.
We use a separate temperature for each field and perform a search to estimate the Pareto front. Each search was conducted using single A10 GPU each. Search details are available in the Appendix. In Table 2, we compare TabMT's DCR and corresponding MLE scores to that of TabDDPM. We are always able to attain higher DCR score, and in most cases a higher MLE score as well. This falls in line with recent results in other domains showing diffusion models are less private than other generative models 2 A comparison with CTabGAN+ is available in the Appendix, compared to CTabGAN+ we obtain both higher privacy and MLE scores in all tested cases. Figure 4/soww the Pareto fronts of TabMT across several datasets.
4.3 Missing Data
Real world data often missing many values that make training difficult. When a row has a missing value we must either drop the row, or find method to impute the missing value. Other techniques such as the RealTabforrmer22 [2 or TabDDPM [2] cannot natively handle real world missing data, and must either use different imputation technique or drop the corresponding rows. Our masking procedure allows TabMT to natively handle arbitrary missing data. To demonstrate this, we randomly drop 25% of values from the dataset, ensuring nearly every row is permanently missing data. Nevertheless, our

---

Table DCR score comparison between TabDDPM and TabMT. Corresponding MLE scores are in parentheses.
DS
TabDDPM
TabMT
DS
TabDDPM
TabMT
AB
0.050(0.550)
0.249(0.533)
GE
0.059(0.597)
0.234(0.599)
AD
0.104(0.795)
1.01(0.811)
HI
0.449(0.722)
(.483)0.727)
BU
0.143(0.906)
0.165(0.908)
HO
0.086(0.677)
0.151(0.607)
CA
(.041(0.836)
0.117(0.832)
IN
0.041(0.809)
(.861)
CAR
0.012(0.737)
0.041(0.737)
KT
0.189(0.833)
(.335
CH
0.157(0.755)
0.281(0.758)
MI
0.022(0.936)
0.026(0.936)
DI
0.204(0.740)
0.243(0.740)
WI
0.016(0.904)
0.063(0.881)
FB
0.112(0.713)
0.252(0.787)
0.1
CA
0.9
C.7
304
33
006
035
8.77
8:18
1.11
8.3
0.53
0.50
MM
0.05
0.78
035
C.55
K78
6032
3.71
X77
0.55
0,77
CH
D1
3.43
9.6
3.24
2.33
4.22
9.9
313
3.29
3150 EX05 0JTE 6375 0.00 6.05 #:39 C.55 0.00 Soare
Wilkedkn Soore
Weloodtonn
Figure 4: Pareto Fronts of TabMT balancing the tradeoff between privacy (DCR) and data quality (Validation Score). While some datasets have a smooth transition the temperature changes, others have a sharp drop-off.
model is still able to train, producing synthetic rows with no missing values in them. This facilitates training on real world data Table 3 shows our accuracy when training with missing data.
Table 3: MLE of TabMT when training with 25% of values missing. Delta represents the difference in MLE from training with no missing values
DS
MLE
Delta
AD
0.813
-0.001
KI
0.868
-0.008
Additionally, our model can be arbitrarily conditioned to produce any subset of the data distribution at additional cost, allowing us to more effectively augment underrepresented portions of data. Prior art is largely incapable of conditioning when producing outputs.
4.4 Scaling
In this section, we examine the performance of our model when scaling to very large datasets. We use the CIDDS-001 [2t as our benchmark dataset. We do not use anomalous traffic from the dataset, and randomly select 5% of the dataset as the validation set for reporting results. The results
7

---

in this section, together with those in Section 4.1, demonstrate that TabMT is both sample-efficee efficien enough to learn with just few hundred samples, while remaining general enough to scale over thirty million samples. We train three model sizes on this dataset and compute metrics the resulting samples. The model topologies are outlined Table 4] Each model was trained on a single A10 GPÚ with the exception of TabMTTLL which was trained using 4 V100s. We use the Adam W [[14] optimizer with a learning rate of 0.002 and weight decay of 0.01, a batch size 2048 and a cosine annealing learning rate schedule for 350,000 training steps and 10000 warm-up steps.
Table 4: Model topologies used in scaling experiments. The large model sizes here demonstrate we can scale well terms of model size and dataset size.
Model
Width
Depth
Heads
TabMT-S
64
12
TabMT-M
384
12
8
TabMT-L
576
24
12
Because ML Efficiency and DCR are very costly compute on dataset of this scale, we instead adapt Precision and Recall [3] to the tabular domain. The original definitions of these metrics [3 rely on vision models to produce the embeddings used. We find our masking procedure produces strong embeddings for each sample, so we use the embeddings produced by TabMT-S. Specifically, we average the embeddings across the fields to produce 64 dimensional for each flow, We used a fixed neighborhood size of k = 3. We include an additional diversity metric defined the average set coverage across all properties of the generated data. Results Table 3 demonstrate strong scaling and performance across model sizes. We can see Precision and Recall both very close to that of the validation set. Sample diversity and quality both scale as model size increases.
Table 5: Precision, Recall, and Diversity metries for all tested models. TabMT outperforms
NFGAN 21 on all metrics
Source
Validation Set
TabMT-S NEGAN
TabMT-M
TabMT--L
Precision(?)) %)
90.42
77.64
82.58
85.77
88.10
Recall(%) Diversity($)) 100.0
90.58
63.32
91.88
91.82
91.12
46.97
89.81
90.56
99.43
We compare against the prior state-of-the-art NetflowGAN, NFGAN This GAN was tuned specifically for this dataset. trained in two phases. First IP2 19 is trained to produce Netflow embeddings. These embeddings are then used targets for the generator during GAN training. Results from NEGAN are shown in Table 5) We can see that NEGAN obtains reasonably high precision, but poor recall and diversity. This is because the model suffers from mode collapse, producing samples in only a small portion of the full distribution.
Netflow has both correlations between the fields and complex invariants between fields. We can measure the violation rate of these invariants to understand how well our model is detecting patterns within the data. We measure against seven invariants proposed in (2I). As shown in Table 6 TabMT produces substantially more diverse data, while achieving a median 20x improvement in violation probability over NFGAN.
5 Limitations and Future Work
TabMT presents strong results even on large datasets, but our Transformer backbone means TabMT is slower than more lightweight methods built around small MLPs, or GANs which can produce a row in a single inference. Searching temperatures also adds time if optimal privacy is needed. Additionally, we must quantize continuous fields, while we outperform methods which do not quantize fields, this could pose issues in some applications. Future work might examine learning across tabular datasets, alternative masking procedures and networks to improve speed, or integration with di diffusion models to better tackle continuous fields.
8

---

Table 6: Error rates on netflow invariant tests. *: because we construct embeddings per field, our model cannot violate check 5. These tests check structural rules reflected in Netflow, such as the fact that two public ip addresses cannot communicate to each other.
NFGAN
TabMT-S
TabMT-M
TabMT-L
TCP Flags
2.33e-03
4.63e-04
2.16e-04
6.54e-06
Private IPs
2.00e-04
7.25c-05
2.67c-05
1.14e-05
TCP Port
3.00e-04
9.32e-05
3,94e-05
1.47e-05
DNS
1.60e-03
2.34e-03
1.56e-03
2.05e-04
Valid Values*
2.00e-03
0.00e-00
0.00e-00
0.00e-00
NetBios
7.43e-01
4.73e-03
4.64e-03
1.27e-03
Packet Ratios 5.10e-03
2.28e-03
1.79e-03
1.16e-03
Broader Impact Synthetic data generation allows for privacy preservation, protecting sensitive data while still enabling data analysis. High Quality synthetic data may ease the pressure to resort unethical methods of collection such relying on underpaid labor. With this in mind, trading off data for additional compute does mean that the additional compute will contribute to increased CO2 emissions. Additionally, synthetic data carries the risk of misuse, such as the potential for manipulating results or research findings with fabricated data. All experiments were conducted using cloud A10 or V100 GPUs. For algorithm design and experiment result generation roughly 410 GPU days of compute were used.
6 Conclusion
In this paper, we outlined a novel Masked Transformer design and training procedure, TabMT, for generating synthetic tabular data. Through comprehensive series of benchmarks we demonstrate that our model achieves state-of-the-arr generation quality. This quality is verified at scales that are orders of magnitude larger than prior work and with missing data present. Our model achieves superior privacy and is able to easily trade off between privacy and quality. Our model is substantial advancement compared to previous work, due its scalability, missing data robustness, privacy- preserving generation, and superior data quality.
References
[1] Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280, 2022.
[2] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprint 20X.::2301.13188. 2023.
[3] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325 2022.
[4] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 11X1v:1810.004805 2018.
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Catboost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.1 13363, 2018.
[7] Justin Engelmann and Stefan Lessmann. Conditional wasserstein gan-based oversampling of tabular data for imbalanced lcarning. Expert Systems with Applications, 174:1 14582, 2021.
[8] Luciano Floridi and Massimo Chiriatti, Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-699 2020.

---

[9] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16000-16009, 2022.
[10] Po- Yao Huang, Hu Xu, Juncheng Ii, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35:28708- 287200. 2022.
[11] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers make one strong gan, and that scale up. Advances in Neural Information Processing Systems, 34:14745-14758, 2021.
[12] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models, arXiv preprint arXiv:2209 15421, 2022.
[13] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinea, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
[14] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[15] Liam Daly Manocchio, Siamak Layeghy, and Marius Portmann. Flowgan-synthetic network flow generation using generative adversarial networks. 202/ IEEE 24th International Conference on Computational Science and Engineering (CSE), pages 168-176. IEFE, 2021.
[16] Inkit Padhi, Yair Schiff, Igor Meinyk, Mattia Rigotti, Youssef Mrouch, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series, In ICASSP 2021-202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3565-3569. IEEE, 2021.
[17] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356 2022.
[18] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss. Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[19] Markus Ring. Alexander Dallmann. Dieter Landes, and Andreas Hotho. Ip2vee: Learning similarities between ip addresses. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 657-666, 2017. doi: 10.1199/CDMW.2017..99
[20] Markus Ring, Sarah Wunderlich, Dominik Grüdl, Dieter Landes, and Andreas Hotho. Flow-based benchmark data sets for intrusion detection. In Proceedings the 16th European conference on cyber warfare and security. ACPI, pages 361-369, 2017.
[21] Markus Ring. Daniel Schlör, Dieter Landes, and Andreas Hotho. Flow-based network traffic generation using generative adversarial networks. Computers & Security, 82:156-172, 2019.
[22] Aivin Solatorio and Olivier Dupriez. Realtabformer: Generating realistic relational and tabular data using transformers. arXiv preprint arXiv:2302.02044 2023.
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[24] Maximilian Wolf, Markus Ring, and Dieter Landes. Impact of generative adversarial networks netflow- based traffic classification. 13th International Conference on Computational Intelligence in Security for Information Systems (CISIS 2020) 12, pages 393-404. Springer, 2021.
[25] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances Neural Information Processing Systems, 32, 2019.
[26] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[27] Zilong Zhao, Aditya Kunar, Hiek Van der Scheer, Robert Birke, and Lydia Y. Chen. CTAB-GAN: effective table data synthesizing. CoRR, abs/2102.08 2021. wtt.ttttts.//eeetett wwwp:/o.g///////// 2102, 08369
[28] Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y Chen. Ctab-gant: Enhancing tabular data synthesis. arXiv preprint arXiv:2204.00 00401, 2022.
10


Reviews:
[{"review_id": "fGQgGyVf4G", "reviewer": "Reviewer_sX8C", "summary": "This paper explores the effectiveness of masked transformers as generative models for synthetic tabular data generation. The proposed TabMT architecture effectively handles challenges related to heterogeneous data fields and missing data. The model shows promising experimental performance and demonstrates good performance even under privacy constraints.", "questions": "1. Could you please clarify the training procedure? When training the masked transformer, does every feature have its own classification head for predicting the mask?\n2. Since you explore hyperparameter tuning for the downstream Catboost model, how was hyperparameter tuning performed for each of the generative models?\n3. In the privacy experiments, only TabDDPM was used as the baseline, would it be possible to add more comparisons with other baselines?\n4. It would be great to add recent state-of-the-art generative models into evaluation, such as STaSy [1]\n\n[1] Kim, J., Lee, C. and Park, N., 2022. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018.", "limitations": "Addressed", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes a promising masked transformer approach TabMT for generating tabular data. This is an elegant application of masked transformers.\n- The paper provides comprehensive experimental evaluation and compares to a diverse set of baselines on 15 datasets\n- The paper explores privacy-preserving capabilities of TabMT and shows promising results", "weaknesses": "- While the paper provides adequate experimental evaluation, it would be helpful to include recent state-of-the-art generative methods in evaluation such as STaSy [1]\n- In the privacy experiments, only TabDDPM was used as the baseline, it would be interesting to see more comparisons with other baselines\n\n[1] Kim, J., Lee, C. and Park, N., 2022. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer sX8C, thank you for your careful consideration and review of our paper.\n\n**Q:** Could you please clarify the training procedure? When training the masked transformer, does every feature have its own classification head for predicting the mask?\n\n**A:** When predicting we use a separate linear layer for each feature, but there are no extra transformer blocks or any other separate layers. It is equivalent to using a single linear layer for all features and masking out impossible values before prediction. The parameters are shared with the embedding layers.\n\n\n**Q:** Since you explore hyperparameter tuning for the downstream Catboost model, how was hyperparameter tuning performed for each of the generative models?\n\n**A:** Each dataset has a fixed set of catboost hyperparameters which are used for all evaluations and all methods on that dataset. The hyperparameters are found by tuning them on the real dataset to maximize the validation score. We use the same tuned values as other works to ensure a fair comparison. They were found using 100 tuning trials with 5 hyperparameters of Catboost.\n\n**Q:** In the privacy experiments, only TabDDPM was used as the baseline, would it be possible to add more comparisons with other baselines?\n\n**A:** We chose to evaluate against TabDDPM because its quality scores were closest to ours, creating a stronger comparison. Privacy and quality tradeoff with each other, so comparing privacies at vastly different qualities is less useful. However, here are some privacy scores from CTABGAN+:\n| Method       | AD           | CA           | CAR          | CH           | DI           | KI           |\n|--------------|--------------|--------------|--------------|--------------|--------------|--------------|\n| **TabMT (ours)** | **1.01(0.811)**  | **0.117(0.832)** | **0.041(0.737)** | **0.281(0.758)** | **0.243(0.740)** | **0.335(0.868)** |\n| CtabGAN+     | 0.119(0.772) | 0.056(0.525) | 0.012(0.733) | 0.212(0.702) | 0.196(0.734) | 0.226(0.444) |\n\nWe obtain both higher privacy and quality on all datasets here. \nWe can add the full results for this model to the paper.\n\n**Q:** It would be great to add recent state-of-the-art generative models into evaluation, such as STaSy\n\n**A:** We are happy to mention StaSy and score-based modeling in our related work section. We have looked over StaSy and it is certainly a strong paper. Unfortunately, none of the datasets the paper tests against overlap with the ones we test with. Implementing StaSy and training it across our fifteen datasets would be quite time-consuming. Additionally, training our model on StaSy’s datasets would also take a fair bit of effort and would be disconnected from the other evaluations we perform. Due to the considerable effort involved, we likely won’t be able to include it in the evaluations at this time.\n\n\nAgain, thank you very much for your response to our paper and the time you took to review it."}, {"review_id": "Ghc2ZzVUj5", "reviewer": "Reviewer_gJJU", "summary": "This paper proposes a new generative model of table-type data based on the transformer.  And it can address the unique challenges posed by heterologous data fields and natively handle missing data. \n", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TabMT is a simple but effective Masked Transformer design for generating tabular data.\n2. We highlight the applicability of our model in privacy-focused applications, illustrating TabMTs ability to arbitrarily trade-off privacy and quality through temperature scaling.\n3. Experiments show the effectiveness of the proposed method.", "weaknesses": "1. The TabMT structure has only been modified in terms of the input layer, without improving the multi-layer transformer structure. In other words, this paper only made adaptive improvements to the data format and is not on par with NeurIPS.\n2. The application method of the model structure proposed by TabMT is not clear. Categorical and Numerical use independent inputs, which makes it necessary to determine the type of data input into the model. How to accurately and automatically identify this type is challenging, which also limits the complexity of generating table data for the model, making it difficult to cope with complex scenarios. The effectiveness of the model should be evaluated for inaccurate category recognition in the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer gJJU, Thank your for taking the time to read and review our paper.\n\n**W:** The TabMT structure has only been modified in terms of the input layer, without improving the multi-layer transformer structure. In other words, this paper only made adaptive improvements to the data format and is not on par with NeurIPS.\n\n**R:** We have contributed more than changing the input layer within our paper. Our paper’s goal is not to alter the transformer structure as a whole. We construct a Tabular Data Generator which achieves **state of the art quality** using a model design **fundamentally different from existing models**. Specifically we do so using Masked Transformers, which are very understudied for generation purposes and we show they suit the problem of tabular data generation very well.\n\nWe contribute the following:\n\n- We construct a Masked Transformer architecture and training task for the tabular domain.\n- We improve upon the traditional masking task itself, and outline a procedure to generate data from this new model. (Masked Transformers are typically poor data generators)\n- We introduce ordered embeddings to help our model deal with numerical values better.\n- We thoroughly justify our architecture, intuitively, mathematically, and empirically.\n- We achieve State of the Art Generation Quality on over a dozen benchmarks.\n- We achieve State of the Art Privacy at these generation qualities, while being able to tradeoff privacy and quality.  Most other models can't tradeoff between these and are fixed .\n- We show how our architecture changes allow it to deal with missing data natively, something existing models can’t do.\n- We explain how our model can be used to condition on arbitrary subsets of data, in a way that other generators cannot.\n- We scale our model to tens of millions of rows and show it still performs well, a much larger scale than previous works.\n- To our knowledge, this is the most thoroughly evaluated and highest performing transformer for generating tabular data. It is also the highest performing model across all existing generative tabular model families including GANs, VAEs, Diffusion Models, and Autoregressive Transformers while using a completely different and novel generation scheme.\n\nThese points are all outlined in the paper. While we have contributed much more than just changing the input layer, if this change alone achieved state of the art, while having the unique privacy tradeoff, conditioning, and missing data capabilities our model has, we believe this would be notable and something to pay attention to. **We are not aware of another model which even has two of these capabilities**, certainly not while also achieving SoTA generation quality.\n\n**W:** The application method of the model structure proposed by TabMT is not clear. Categorical and Numerical use independent inputs, which makes it necessary to determine the type of data input into the model. How to accurately and automatically identify this type is challenging, which also limits the complexity of generating table data for the model, making it difficult to cope with complex scenarios. The effectiveness of the model should be evaluated for inaccurate category recognition in the experiment.\n\n**R:** We do not aim to automatically learn or identify which columns are categorical and which columns are numerical. We assume that this is known ahead of time, and that the model is allowed to treat categorical and numerical variables differently within the model. In the majority of cases, the data type (float, integer, string) is sufficient for this purpose, but it should be known metadata by the practitioner. As far as we are aware, this assumption is used by essentially all tabular generators.\n\n**References:** \\\nTVAE and CTGAN: https://arxiv.org/pdf/1907.00503.pdf\n\nTabDDPM: https://arxiv.org/pdf/2209.15421v1.pdf\n\nCTabGAN+: https://arxiv.org/pdf/2204.00401.pdf\n\nAgain, Thank you very much for reading and reviewing our paper. We hope our responses can help clarify some things and explain why we believe our work is an impactful and novel contribution to the area of Tabular Data Synthesis."}, {"review_id": "HcYBAXOXkV", "reviewer": "Reviewer_aH7d", "summary": "The paper proposes a masked transformer model that can be used to generate tabular data. The authors propose modifications in the masking strategy in the transformer, in order to make it more effective for generating data. Empirically, the proposed generator improves performance in various datasets compared to other baselines. Moreover, the paper presents cases of privacy-focused applications, presence of missing values, and large datasets to further depict the usability of the proposed method in real-world situations. ", "questions": "- Is it possible for the model to generate new categories (or numbers) not present in the train set?\n", "limitations": "The presented work might not be accessible to regular users, given the amount of resource required to train the model.  ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and very clear to its points. The authors define the problem with the masked transformer for generating tabular data, and presents an effective method to overcome the difficulty. The empirical results are well presented, and seems promising.", "weaknesses": "As noted in the paper, the major weakness would be the resource it takes to train the model. For general users, a pre-trained model with learning across multiple tables might be beneficial. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer aH7d, thank you for your rating and the time you took to carefully read and review our paper.\n\n**W:**  ”As noted in the paper, the major weakness would be the resource it takes to train the model. For general users, a pre-trained model with learning across multiple tables might be beneficial.”\n\n**R:** You’re right, a pretrained model which can learn across datasets is definitely something we are interested in and plan to explore in the future. We allude to this in our future work section.\n\n**L:** ”The presented work might not be accessible to regular users, given the amount of resource required to train the model.”\n\n**R:** The compute used by our model is more than some of the other models we compared against, but we do still believe it should be accessible for most users to train for most use cases. Training on a single GPU takes around 0.5-2.5 hours on most datasets. A strong model for our scaling dataset with tens of millions of rows can be trained within a GPU day. These models as a whole use much less compute than ones used in NLP or Vision, but we still felt it important to mention the compute usage as a limitation, since it is often ignored.\n\n**Q:** ”Is it possible for the model to generate new categories (or numbers) not present in the train set?”\n\n**A:** Generating new categories is very difficult to do, at the time of writing, we are not aware of any model which can do this well in general. However for ordered or continuous variables our quantization means most values generated are not present in the training set. Additionally, altering the quantizer to be a GMM instead of the default K-Means, or interpolating between the support allows us to generate arbitrary values. Our default model does not do this, since we found the simpler finite support method still gave us strong results overall. The ability of our model to condition on any subset of fields during generation means it can also be seamlessly combined with other methods of generating data, for example, to create a joint diffusion-masked model. Prior work is unable to do this.\n\nAgain, thank you very much for your time and review. We are glad you liked the paper, and we hope it can help researchers and practitioners to examine new research directions and solve more problems."}, {"review_id": "Nlu3n3KEAw", "reviewer": "Reviewer_EYXT", "summary": "This paper introduces TabMT, a new Masked Transformer architecture designed for generating synthetic tabular data. While Transformers are predominantly used in natural language processing (NLP), TabMT demonstrates their effectiveness in dealing with heterogeneous data fields, such as images and tables, and efficiently manages missing data. The authors propose to sample the masking probability from a uniform distribution and predict masked values in random order during generation. By employing advanced masking techniques, TabMT can generate synthetic data with high performance across a wide range of dataset sizes. Moreover, the model proves valuable in privacy-sensitive applications, as it is capable of producing high-quality data while adhering to privacy restrictions. ", "questions": "The work can benefit from discussing and comparing the work of causal inference using transformers, which is a particular kind of missing data generation problem where privacy protection is also pressing, e.g. [1], [2].\n\n[1] Exploring Transformer Backbones for Heterogeneous Treatment Effect Estimation\n[2] Differentially Private Synthetic Control", "limitations": "See the weaknesses part", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation is easy to understand and the problem is important but less explored than common domains. It is not straightforward to apply existing techniques on images and texts to the tabular data.\nThe experimental designs are comprehensive, including several sections assessing data quality, Privacy and Sample Novelty, Missing Data, and Scaling, which are solid.", "weaknesses": "The motivations for designs are not very clearly written. Please consider reorganizing the structure, adding highlighted paragraphs, and moving line 129-138 to the beginning of the method or merging them with the methodology introduction texts.\n\nThe introduction of temperature scaling is not well-motivated, is it for privacy purposes and why will it work?\n\nThe introduction of privacy tabular data generation is not detailed. The conclusion claims that \"the model is able to function under arbitrary privacy budgets.\" but I even didn't see how the budgets are computed and its defitions, motivations etc. Same for the definition of data \"novelty\". And the questions of \"why many designs in TabMT can help those aspects are not well explained\".\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi Reviewer EYXT, Thank you for the time you took to read our paper and write your review.\n\n**W:** The motivations for designs are not very clearly written. Please consider reorganizing the structure, adding highlighted paragraphs, and moving line 129-138 to the beginning of the method or merging them with the methodology introduction texts.\n\n**R:** We can reorganize these things and add bolding to make things clearer. 129-138 would likely work better at the top of this section.\n\n**W:** The introduction of temperature scaling is not well-motivated, is it for privacy purposes and why will it work?\n\n**R:** Temperature Scaling is often used when generating data from transformers to increase diversity or improve generation quality (see the references below). We can add a few more sentences explaining our learned temperature. Our learned temperature allows the network to sharpen the output logits across the embeddings. Allowing sharpening through temperature gives the model a more effective method of doing so without relying on the unordered embeddings or magnitude changes. This is important in our ordered embeddings since they are interpolations, instead of each embedding being independent of each other. We have confirmed the learned temperature acts in this way as all learned temperatures have averages well below 1 (sharpening), and nearly all end up below their initialized value.\n\n**References:**\n\nTemperature use during generation to improve generation quality or diversity:\n\nhttps://arxiv.org/pdf/1904.09751.pdf\n\nhttps://transformer.huggingface.co/doc/gpt2-large (text generation demo with temperature)\n\nTemperature during Learning:\n\nhttps://arxiv.org/pdf/2002.05709.pdf (hyperparameter which helps sharpen the logits)\n\nhttps://arxiv.org/pdf/2103.00020.pdf (learned temperature instead of hyperparamter)\n\n**W:** The introduction of privacy tabular data generation is not detailed. The conclusion claims that \"the model is able to function under arbitrary privacy budgets.\" but I even didn't see how the budgets are computed and its defitions, motivations etc. Same for the definition of data \"novelty\". And the questions of \"why many designs in TabMT can help those aspects are not well explained\".\n\n**R:** Lines 189-203 help address this within the paper. Here we define the Distance to Closest Record metric (DCR), which has also been used in prior work, but allow us to explain things further. Perhaps instead of saying “arbitrary privacy budgets”, we should say “produce data with arbitrary privacy scores”. By budget we mean a privacy score (DCR) threshold. The model is able to operate under arbitrary privacy budgets since we are able to walk the Privacy v. Quality Pareto curves of our model. **Most previous works do not have the ability to change the privacy scores of their generated data after training**. Figure 4 is a good illustration of this concept. Imagine we would like our DCR to be above some number for our synthetic data (the threshold is our budget), we can change the temperatures of our model to achieve this DCR, no matter what the DCR is, up to the point of generating random data. Other models will have a fixed DCR after training and if that falls below what you desire, then that model becomes unusable for your application. Our model is tunable in this respect in a way most others aren’t and can function across all reasonable thresholds, because the temperature can be tuned in order to achieve it. We also show that at high quality scores our model also achieves higher privacy than models of similar quality. This demonstrates temperature works well to control this tradeoff in our model.\n\nDCR is also a good measure of novelty in addition to privacy because as samples get further away in terms of distance from all points seen during training, we can expect them to generally be more novel. Here’s another quote from the paper related to this.\n\n*By ensuring our model is both private and high quality, we verify that our model has learned the intrinsic structure of the data, and not simply memorized it.*\n\nThis is what we mean by novelty. If our synthetic dataset is far in terms of distance from the training set, the model could not have simply memorized it. Our model has SoTA privacy and quality scores, meaning it produces novel data which is also high quality.\n\nPrevious Work in this field using DCR for these purposes:\n\nRealTabFormer: https://arxiv.org/abs/2302.02041\n\nTabDDPM: https://arxiv.org/pdf/2209.15421v1.pdf\n\nCTabGAN: https://arxiv.org/pdf/2102.08369.pdf\n\nNearest neighbors are commonly examined from the training set, even in other domains, to see if generative models are producing truly novel data and are not memorizing or overfitting (e.g. BigGAN: https://arxiv.org/pdf/1809.11096.pdf)\n\n**Q:** The work can benefit from discussing and comparing the work of causal inference using transformers, which is a particular kind of missing data generation problem where privacy protection is also pressing, e.g. [1], [2].\n\n**A:** We can add a discussion of this work within the related work section to talk about the challenges these techniques solve and the different approach they take for it.\n\nOur model achieves state of the art generation quality and privacy while having the ability to tradeoff between these arbitrarily, natively learn with missing data present, and condition on arbitrary subsets of the distribution during generation. We are not aware of any prior art which is able to do these things within a singular model, certainly not while also achieving state of the art generation quality. We achieve these results while using an innovative masked generator architecture, and thoroughly validate our architecture up to tens of millions of rows.\n\nThank you very much for the time you spent reviewing and reading our paper. We hope these comments can help clarify some of the details of our paper, and why we believe it to be a worthwhile and novel contribution to Tabular Data Synthesis."}]

Author rebuttal:

Meta-review: The paper received accept, weak accept, and two borderline rejects post rebuttal. However, 3 out of the 4 reviewers (except the weak accept one sX8C) did not update their recommendation. The AC checked all the materials, including the author response to each of the reviewers, and the supplementary materials, and decides to accept the paper. Overall, the paper extends the research of tabular data generation with masked Transformers, and show extensive comparisons with prior works justifying the proposed design. Please incorporate the feedbacks from the reviews for the final version.
Decision: Accept (poster)