Tabby: Tabular Data Synthesis with Language Models
Authors: name: Sonia Cromp, name: Satya Sai Srinath Namburi GNVV, name: Mohammed Alkhudhayri, name: Catherine Cao, name: Samuel Guo, name: Nicholas Roberts, name: Frederic Sala

While advances in large language models (LLMs) have greatly improved the
quality of synthetic text data in recent years, synthesizing tabular data has
received relatively less attention. We address this disparity with Tabby, a
simple but powerful post-training modification to the standard Transformer
language model architecture, enabling its use for tabular dataset synthesis.
Tabby enables the representation of differences across columns using Gated
Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby
results in data quality near or equal to that of real data. By pairing our
novel LLM table training technique, Plain, with Tabby, we observe up to a 44%
improvement in quality over previous methods. We also show that Tabby extends
beyond tables to more general structured data, reaching parity with real data
on a nested JSON dataset as well.