SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: MLE of TabMT when training with  $25\%$  of values missing. Delta represents the difference in MLE from training with no missing values.
DS	MLE	Delta
AD	0.813	-0.001
KI	0.868	-0.008
Additionally, our model can be arbitrarily conditioned to produce any subset of the data distribution at no additional cost, allowing us to more effectively augment underrepresented portions of data. Prior art is largely incapable of conditioning when producing outputs.
In this section, we examine the performance of our model when scaling to very large datasets. We use the CIDDS-001 dataset[20] as our benchmark dataset. We do not use anomalous traffic from the dataset, and randomly select  $5\%$  of the dataset as the validation set for reporting results. The results
in this section, together with those in Section 4.1, demonstrate that TabMT is both sample-efficient enough to learn with just a few hundred samples, while remaining general enough to scale to over thirty million samples. We train three model sizes on this dataset and compute metrics on the resulting samples. The model topologies are outlined in Table 4. Each model was trained on a single A10 GPU with the exception of TabMT-L which was trained using 4 V100s. We use the AdamW[14] optimizer with a learning rate of 0.002 and weight decay of 0.01, a batch size of 2048 and a cosine annealing learning rate schedule for 350,000 training steps and 10000 warm-up steps.
Table 4: Model topologies used in scaling experiments. The large model sizes here demonstrate we can scale well in terms of model size and dataset size.
Model	Width	Depth	Heads
TabMT-S	64	12	4
TabMT-M	384	12	8
TabMT-L	576	24	12
Because ML Efficiency and DCR are very costly to compute on a dataset of this scale, we instead adapt Precision and Recall[13] to the tabular domain. The original definitions of these metrics[13] rely on vision models to produce the embeddings used. We find our masking procedure produces strong embeddings for each sample, so we use the embeddings produced by TabMT-S. Specifically, we average the embeddings across the fields to produce 64 dimensional for each flow. We used a fixed neighborhood size of  $k = 3$ . We include an additional diversity metric defined as the average set coverage across all properties of the generated data. Results in Table 5 demonstrate strong scaling and performance across model sizes. We can see Precision and Recall are both very close to that of the validation set. Sample diversity and quality both scale as model size increases.
Table 5: Precision, Recall, and Diversity metrics for all tested models. TabMT outperforms NFGAN[21] on all metrics
Source	Validation Set	NFGAN	TabMT-S	TabMT-M	TabMT-L
Precision(%)	90.42	77.64	82.58	85.77	88.10
Recall(%)	90.58	63.32	91.88	91.82	91.12
Diversity(%)	100.0	46.97	89.81	90.56	99.43
We compare against the prior state-of-the-art NetflowGAN, or NFGAN[21]. This GAN was tuned specifically for this dataset. It is trained in two phases. First IP2Vec[19] is trained to produce Netflow embeddings. These embeddings are then used as targets for the generator during GAN training. Results from NFGAN are shown in Table 5. We can see that NFGAN obtains reasonably high precision, but poor recall and diversity. This is because the model suffers from mode collapse, producing samples in only a small portion of the full distribution.
Netflow has both correlations between the fields and complex invariants between fields. We can measure the violation rate of these invariants to understand how well our model is detecting patterns within the data. We measure against seven invariants proposed in [21]. As shown in Table 6 TabMT produces substantially more diverse data, while achieving a median  $20\mathrm{x}$  improvement in violation probability over NFGAN.
TabMT presents strong results even on large datasets, but our Transformer backbone means TabMT is slower than more lightweight methods built around small MLPs, or GANs which can produce a row in a single inference. Searching temperatures also adds time if optimal privacy is needed. Additionally, we must quantize continuous fields, while we outperform methods which do not quantize fields, this could pose issues in some applications. Future work might examine learning across tabular datasets, alternative masking procedures and networks to improve speed, or integration with diffusion models to better tackle continuous fields.
Table 6: Error rates on netflow invariant tests. *: because we construct embeddings per field, our model cannot violate check 5. These tests check structural rules reflected in Netflow, such as the fact that two public ip addresses cannot communicate to each other.
NFGAN	TabMT-S	TabMT-M	TabMT-L
TCP Flags	2.33e-03	4.63e-04	2.16e-04	6.54e-06
Private IPs	2.00e-04	7.25e-05	2.67e-05	1.14e-05
TCP Port	3.00e-04	9.32e-05	3.94e-05	1.47e-05
DNS	1.60e-03	2.34e-03	1.56e-03	2.05e-04
Valid Values*	2.00e-03	0.00e-00	0.00e-00	0.00e-00
NetBios	7.43e-01	4.73e-03	4.64e-03	1.27e-03
Packet Ratios	5.10e-03	2.28e-03	1.79e-03	1.16e-03
Broader Impact Synthetic data generation allows for privacy preservation, protecting sensitive data while still enabling data analysis. High Quality synthetic data may ease the pressure to resort to unethical methods of collection such as relying on underpaid labor. With this in mind, trading off data for additional compute does mean that the additional compute will contribute to increased CO2 emissions. Additionally, synthetic data carries the risk of misuse, such as the potential for manipulating results or research findings with fabricated data. All experiments were conducted using cloud A10 or V100 GPUs. For algorithm design and experiment result generation roughly 410 GPU days of compute were used.
In this paper, we outlined a novel Masked Transformer design and training procedure, TabMT, for generating synthetic tabular data. Through a comprehensive series of benchmarks we demonstrate that our model achieves state-of-the-art generation quality. This quality is verified at scales that are orders of magnitude larger than prior work and with missing data present. Our model achieves superior privacy and is able to easily trade off between privacy and quality. Our model is a substantial advancement compared to previous work, due to its scalability, missing data robustness, privacy-preserving generation, and superior data quality.
[1] Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280, 2022.
[2] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188, 2023.
[3] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[4] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[6] Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Catboost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363, 2018.
[7] Justin Engelmann and Stefan Lessmann. Conditional Wasserstein gan-based oversampling of tabular data for imbalanced learning. Expert Systems with Applications, 174:114582, 2021.
[8] Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. *Minds and Machines*, 30:681–694, 2020.