Tabby: Tabular Adaptation for Language Models

Authors: Sonia Cromp, Satya Sai Srinath Namburi GNVV, Catherine Cao, Mohammed Alkhudhayri, Samuel Guo, Nicholas Roberts, Frederic Sala
Primary area: foundation or frontier models, including LLMs
Keywords: tabular, generative, llm, mixture-of-experts, synthesis, transformer

Abstract:
While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received far less attention. Many of the top-performing approaches to this problem rely on techniques that adapt models originally developed for other modalities, potentially leaving generative performance on the table. We address these disparities in attention and performance for tabular data by introducing Tabby, a simple but powerful post-training modification to the standard Transformer-based language model architecture that enables its use for tabular dataset synthesis. Tabby relies on Gated Mixture-of-Experts layers, allowing each data column to be modeled by a dedicated set of parameters within the transformer multi-layer perceptrons or language modeling heads.  Applying Tabby to Distilled-GPT2 improves synthetic data quality up to 7% compared to previous tabular dataset synthesis methods, achieving performance near or equal to that of real data.

Decision: reject
Reviewer scores: 1, 3, 5

Reviews and discussion:
<thread>
creation date:
1730357960326

thread_type:
official_review

<thread_object>
creator: reviewer_rywz

<content>
summary:
In this paper, the authors present a tabular data synthesis approach, Tabby. The novelty of Tabby lies in two main aspects: (1) modifying the original transformer model by applying MoE-like techniques to better model tabular data, and (2) designing a specialized data format for tabular data. Experimental results show that Tabby achieves comparable performance to the previous state-of-the-art, Tab-DDPM, and outperforms GTT NT.

strengths:
- The model modifications and data organization are well-motivated and intuitive.
- The distribution of the synthesized data is very close to the natural data.
- The experimental results looks good.

weaknesses:
- Tabby seems achieve comparable results to Tab-DDPM with marginal performance gain in Table 2.
- The method is quite simple and not much effective in final performance.

questions:
Q1: Have you computed the FLOPs for training on different datasets? It seems that Tabby uses a fixed pattern to organize tabular data, which may require more tokens for computation.

Q2: Regarding Claim 2, could you provide a scaling curve showing performance relative to model size or data quantity? It would be interesting to see how Tabby impacts different models and how the amount of Tabby data influences the learning process. Additionally, a comparison of the scaling curve between Tabby data and natural data would serve as evidence of Tabby data being a scalable alternative to natural data.

Q3: I'm not sure if the modification to original network is necessary. Is there an ablation study?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565777523

thread_type:
official_review

<thread_object>
creator: reviewer_mewm

<content>
summary:
This work introduces a new model called Tabby for tabular data. Tabby is an architecture modification that enables transformer-based language models to synthesize more realistic tabular data. It introduces Gated Mixture-of-Experts layers to better model the complex interdependencies and diverse data types found in tabular datasets. Tabby outperforms previous tabular data synthesis methods, achieving outstanding performance on multiple benchmarks.

strengths:
1. Tabby achieves strong performance in benchmark evaluation. It generates high-quality synthethic tabular data in comparison with the baseline methods.
2. The introduction of MoE shows effectiveness in helping the model understand tabular data structure and generate higher-quality tabular data.

weaknesses:
1. The design of MoE layer is complex. For a table with V columns, this article should design an MoE model with V experts to adapt to the table. This is not generalizable to data of diverse formats. It is suggested to modify the model design to be more compatible and more generalizable.
2. Scalable experiments are advised to be conducted. This study needs to provide experimental results on datasets of larger scales and also more commonly used datasets.
3. The experiments are advised to be conducted on contemporary large language models, including Llama, Qwen, Mistral, instead of Distilled-GPT2.

questions:
1. Have you conducted experiments on the recently released large language models? If yes, which model sizes did you choose?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730740535302

thread_type:
official_review

<thread_object>
creator: reviewer_mxt7

<content>
summary:
The paper proposes to use a MOE LLM fine tuned on table data for synthetic table data synthesis.  The authors find that their method outperforms previous methods on table synthesis benchmarks.

strengths:
The proposed method outperforms other methods on most datasets and metrics presented.

weaknesses:
* There is no significant difference between tabby and the non-tabby (I assume no MOE?) baseline.  Given that MOE has a lot more parameters, this is a negative finding.
* The papers contributions are very minor - applying MOE to a narrow problem (table generation).  And the results are not all that strong.
* It's not easy from the presentation what exactly do the tasks require, what exactly are the baselines and model variations.

questions:
Can you please detail the various architectures MMLP, MH and MMLP+MH? 
Why does MMLP+MH underperform, even though it is more complex?
Do you replace every layer with MOE?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732128788783

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the attentive reviewers for their feedback and comments on our paper. We have received much valuable feedback that allows us to better explain the advantages of Tabby as compared to prior works, such as its freedom from prior assumptions on data column modalities. We reiterate the contributions and strengths of Tabby here:

- **Tabby achieves state of the art performance on the standard tabular evaluation metrics, with fewer limitations than the prior state of the art method** (Tab-DDPM). Tab-DDPM is unable to generate realistic datasets with real-valued numerical labels and is unable to synthesize strings to the same degree as LLMs, resulting in limitations to the privacy of Tab-DDPM’s synthetic data and the ability of Tab-DDPM to model areas of the data distribution that do not occur in its training data (for further details on these limitations, refer to our comment to Reviewer RYWz). Tabby wholly circumvents these limitations, while achieving comparable state of the art performance and even outperforming Tab-DDPM on some datasets.
- **Tabby is the first architecture modification that allows LLMs to directly generate tabular data**. Because the prior LLM tabular synthesis approaches (e.g. GReaT, TapTap and Tabula) are training techniques, they can be used in concert with Tabby. Further, Tabby is applicable to any transformer-based LLM, as we demonstrate in Section 5.2.
- **Tabby’s Mixture of Experts (MoE) layer design is flexible and may serve as a basis for future work in other structured modalities**, such as JSON objects and geospatial data. While our paper demonstrates the benefits of replacing language modeling heads and transformer block MLPs with MoE layers, there are many other possible combinations of MoE layers. For instance, future work can replace attention heads with MoEs or nest MoEs within each other to generate nested data structures such as JSONs. Tabby’s high performance for tabular synthesis demonstrates that these architecture variations bring large improvements in our ability to generate many varieties of structured data using pre-existing, pretrained architectures such as LLMs.

We again thank our reviewers for the careful attention that they have devoted to our paper, as well as the PC, AC and others who assist in the conducting of ICLR’s rigorous reviewing process.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Tabby is a method for transformer-based Large Language Models (LLMs) to synthesize high-fidelity tabular data. Tabby produces higher-quality synthetic data for 4 out of 6 datasets compared to previous methods. An architecture modification is proposed for the task. The reviewers argue that the additional complexity of MoE design is undesirable, and the paper needs more scalable experiments to show generalizability. The paper needs further improvements to make it more solid.