Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis
Authors: Sonia Cromp
Satya Sai Srinath Namburi GNVV
Mohammed Alkhudhayri
Catherine Cao
Samuel Guo
Nicholas Roberts
Frederic Sala
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have greatly improved the quality of synthetic text data. We aim to extend these advances to tabular data with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby represents differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. Pairing Tabby with Plain, our novel tabular training technique, we observe up to a $7\%$ improvement in quality (measured by MLE) over previous methods. Additionally, our approach is more flexible than prior strategies and extends beyond tables, to more general structured data. In a structured JSON setting, Tabby outperforms all other methods by $2$-$3$ points and is the only approach with MLE equal to the upper bound of non-synthetic data.

Published in Transactions on Machine Learning Research (01/2026)
Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis
Sonia Cromp
sonic wis.sssssscceuu
Deportment of Computer Sciences University of Saceonsin-MMdiine
Satya Sai Srinath Namburi GNVV GE CeaethCare
salg..nambur@@gghccahccrr...mm
Catherine Cao
cono@nettlia.cco com
Netflir
Mohammed Alkhudhayri
malkhuudhaggriShusse..t
Department of Computer Sciences University of Wisconsin-Meedt Madison
Samuel Guo
sguo258@wisc.com edu
Department of Computer Sciences University of Wisconsin-Madison
Nicholas Roberts
nickl www.::c...w..c...........
Deportment of Computer Sciences
University of Wiscomsin-Madisonn
Frederic Sala
fredsala &cs. wisc. edu
Department of Computer Sciences
University of Aitonsin-MMaiion
Reviewed on OpenReview: https: // openreview. net/ forum? id= 69FPVnbOBm
Abstract
Large language models (LLMs) have greatly improved the quality synthetic text data. We aim to extend these advances to tabular data with Tabby, a simple but powerful post-training modification the standard Transformer language model architecture, enabling its for tabular dataset synthesis. Tabby represents differences across columns using Gated fixturr--f-Exxertts with column-speeific sets of parameters. Empirically, Tabby results in data quality near equal to that of real dała. Pairing Tabby with Plain, our novel tabular training technique, we observe up to 7% improvement in quality (measured by MLE) over previous methods. Additionally, our approach is more flerible than prior strategies and extends beyond tables, to more general structured data. a structured JSON setting, Tabby performs all other methods by 2-3 points and the only approach with MLE equal to the upper bound of non-synthetic data.
1 Introduction
Modern life is built tabular data: sirplane black boxes, website analytics and hospital patient records are just few examples of this versatile modality. Despite widespread use of tables and repeated calls for improved table modeling approaches (Fang et al. 2024 Davila et al. 2024), the tabular modality has received less attention in recent deep learning rescarch than images or text (van Brougel & van dor Schaar 2024)
Progress towards realistic tabular data synthesis encounters several key challenges. Firsi, table columns often exhibit complex interdependencies. Second, many tabular datasets mix multiple datatypes, A single table

---

Published in Transactions on Machine Learning Research (01/2026)
might contain free-text fields, numerical features, and even nested JSON or dictionary columns. Third, although the order of tokens within one column is important, the order of columns with respect to each other is usually not. meaningful and is potential source of spurious correlations during training. How best to design model architectures and training tech- niques that address these issues is open question.
Tabby Multi-Head Version
Base LLM Without Tabby
Transformer Block
Transformer Block
Transformer Block
There have been notable efforts to adapt several model archilectures to tabular dala, recently locusing generative adversarial networks (GANs) (Xu et al. 2019), LLMs (Borisov ct al. 2022 and diffusion models (Kotelnikov et al.) 2022). However, because these architectures were each designed with images or text in mind, significant preprocessing must be made to tabular datasets in order to allow their use.
Transformer Block 2
Transformer Block B
Transformer Block B
For these reasons, works including van Breugel & van der Schaar (2024) have called for the develop- ment of pretrained Large Tabular Models (LTMs) to fill a similar role to text and image foundation models, such as GPT (Achiam et al. 2023) or Stable Diffusion [Blattmann et al. (2023). Unfortunately, the creation of an LTM requires (1) large, diverse
Hond
Head
Head
Head
2
V
Mixture-of-Experts LM Head
Figure 1: Tabby Multi-Head modifications (right) com-
pared to an original, Non-Tabby LLM on left.
tabular pretraining sets which have not yet been rated, (2) specialized tabular model architecture which has yet to be designed, and (3) substantial compute resources for pretraining. These challenges are even more pronounced in the development of foundation models for structured non-tabular modalities, such as JSON and geospatial data.
This work takes an initial step towards LTMs with Tabby, a post-training modification to the standord transformer LLM architecture to enable tabular and other structured data synthesis. After training on text data but before finctuning on structured data data-Tabby replaces select LLM blocks with deterministic Mizture-of Mfature-of-Experts (MoE) layers (Shazeer et al. 2017). allowing each tabular column, JSON attribute other structured feature to be modeled by dedicated set of parameters in the LLM. The greater expressivity afforded by this change results in higher-fidelity synthetic data. Fine-tuning with our novel Plain technique results in still higher performance. We show that even small Tabby models are capable of outstripping large non-Tabby LMs with parameter counts orders of magnitude greater.
To our knowledge, Tabby is the first architecture modification to make LLMs belter-suited to table generation. Using a pretrained LLM as a starting point allows Tabby to take advantage its text pretraining, avoiding the logistical challenges of training a LTM fully from scratch. We find that, according to standard metrics, Tabby produces synthetic data near- or at- at-parity with real data on 5 out of 8 table datasets. Additionally, Tabby is not limited to tables and can be easily extended to other structured data. We validate this by synthesizing nested JSON data at at-parity with real data as well. Our contributions are:
We introduce Tabby, H simple architecture modification that allows transformer-based LLMs to synthesize more realistic tabular and structured non-tabular data.
We demonstrate that Tabby produces higher-quality synthetic data for 4 out of 6 popular tabular datasets, is compatible with more tabular data than prior high-quality synthesis methods, and can be extended beyond tables to a broader class of structured data modolities.
We introduce our novel tabular LLM training method, Plain. Named after its surprising simplicity compared to prior table training approaches for LLMs, Plain increases data quality in all datasets when used together with independently of Tabby, compared to the prior SOTA LLM approach.
Cooeeaase: hetps://githubs http::::/ggttbb..c////cceeeeeeee con/soCronp/tabyy
2

---

Published in Transactions on Machine Learning Research (01/2026)
2 Tabby Architecture & Plain Train Method
We now formally introduce our two novel contributions for LLM table synthesis: Tabby is an architecture modification that may be applied to any transformer-basce language model (LM) (Vaswani 2017). and Plain is a training technique for training any (Tabby or non- (an-Taby) LM on tabular data. Tabby and Plain are especially powerful together: Tabby increases model expressivity in a way that is specially-suiter to tabular data, while Plain allows the model to more effectively fit to the key features of tabular dataset during training, yielding more realistic data synthesis.
In Section 2.1 we describe Tabby for tabular or other structured data. In Section 2.2 we outline the process for training an LM tabular data using our Plain training technique. Then, in Section 2.3 we provide additional insight into how Tabby models are trained (using Plain, or pre-existing LLM table training techniques such as GReaT (Borisov et al. 2022)) by comparing the training process's 's forward pass and loss calculation for a Tabby model with a non- non-Tabby model.
2.1 Architecture of Tabby Models
The intuition behind the Tabby modification is simple: we want to allow the model to learn individual columns as distinct but interdependent tasks. The right side of Figure | depicts our best-performing Tabby model for tabular data, where Tabby modifies only the language modeling head.
To provide a general definition of a Tabby model, consider tabular datasct with V columns. Let the order of blocks within an arbitrary transformer-based LM be represented as [L1 L.,..., Ln]. We apply the MoE technique by replacing LM block La with vector Aa = L, La....... L. of V blocks. Thus, Tabby model with one MoE block Aa is represented
[L, L2. La-1, [L,11 La,,..... La,y], Lat1. Lu).
The datasct's i-th column is modeled by La,i within Ag. In this way, Tabby's MOE layers use a deterministic, schema-based routing function where the chosen expert is determined by the current column index. The gating function can be expressed mathematically as y = Σド li = (x)), where x; denotes a position within the i-th column, and f, is the j-th expert.
This technique may be applied to any set of layers within the model. While focus the language modeling (LM) head in Figure [1 and Section 3 evaluations, also conduct experiments applying Tabby to the transformer multi-layer perceptrons and attention blocks in Appendix E We refer to Tabby models with MoE LM Heads as Tabby Hulti-Head (MH) models.
2.2 The Plain Technique for Fine- Tuning LLMs Tabular Data
Suppose our trainset contains N rows and column names denoted by 11 ......, uv, such that of represents the value of the j-th row in the i-th column. To provide the LM with its expected text modality input, we convert the j-th row as follows, where <EDS> is the end-of-sequence token and <EOC> is specialized end-of-column token which we introduce to divide the text between columns:
"<BOS> U1 is of <EDC> U2 is vz <EDC> ... By is uy <EOS>"
Converting the tabular dataset in this fashion allows an LM to fine-tune on the dataset in a normal sequence- to-sequence style. Because Plain encodes data the same way prior LLM table training techniques, GRea'T and Tabula (Borisov et al. 2022 Zhao et al. 2023 Plain does not require more FLOPs than prior methods.
During inference, the prompt for each row is the begimning-ofsscquenee token <BOS>. During generation, the LM will output text in a similar format to the training data, which can then be parsed into tabular data as desired. We note that the simplicity of Plain is particularly impressive given its favorable performance compared to prior LLM table trainêng methods, as we show in Section 3
2.3 Tabby Training
Now that we have introduced Tabby and the Plain training method, we are able to provide further insight into aspects of the training process unique to Tabby. Suppose that we construct a Tabby model from a base
2 head" refers here the language model output layer, distinct from attention heads in the MLP blocks.
3

---

Published in Transactions on Machine Learning Research (01/2026)
LM by replacing one of its blocks La with an MoE set Ag- At the beginning of fine-tuning the Tabby model, weights for each block in A, are initialized to equal the weights of La- These column experts then diverge from each other over the course of fine-tuning.
The Tabby training process requires only slight modifications compared to other LMs for tabular data. Instead of representing each training row as one string, we convert each row into list of V strings:
["er is of <EOC>", "02 is tz <EOC>",
"vv is of <EOS>"]
Internally, the Tabby model begins by training on column 1 with prompt <BOS>; attending to tokens through -- 1 when predicting the k-th token. Afler computing the loss on column 1, this column's tokens are appended to the prompt used to train column 2. Backpropagation is performed at the end of batch of rows. The prompt when training on column i is
"<BOS>01 is of <EOC> U2 is o
<BOC> 2-11 is di1 <EOS>"
Because calculate losses for each column separately, are able to monitor the performance of each column individually during training. This favorable side-effect is demonstrated in Section 3.4
2.4 Extensions
We address two additional aspects of Tabby and Plain: (1) generalizations that go beyond tabular data and (2) optimizations for datasets with large numbers of columns.
Synthesis for general structured modalities: The flexibility in Tabby MoE layer design enables extensions to a variety of structured datatypes, such as hiererchical data. For example, we create a model for nested JSON data by applying Tabby recursively in Figure 5 The JSON structure is preserved inherently in the model, so that Plain's method of representing data features does not need to be modified to indicate nested features. As we show in Scction 3.4, the combination of Plain and Tabby is the only synthesis approach to reach equal performance to real, non-synthetie data.
Because Tabby MoEs contain one block per dataset column, model parameter count is proportionate to the number of data features. In practice, however, techniques such as parameter sharing (Ravanbakhsh et al. 2017) can drastically reduce the number of parameters to represent a Tabby model. Additionally, Tabby may be implemented so that only one block in the MOE layer in memory at a time, a potential strategy to achieve memory requirements identical to a non-Tabby model.
High-dimensional data:
3 Experimental Results
Our evaluations seek to assess the following claims:
Claim 1: Plain-trained Tabby models generate higher-quality tabular data than prior approaches.
Claim 2: The Tabby architecture modification allows smaller LLMs to achieve similar or better synthetic data fidelity than LLMs with higher parameter counts.
Claim 3: Tabby architecture modifications may also be applied to other structured data beyond tabular data, resulting in higher-quality synthetic data for these modalities as well.
Claim 4: Tabby's loss formulation allows for convenient tracking of per-column performance at training time, leading to better understanding of model behavior.
Claim 5: Tabby's language modeling capabilities enable it to capture the underlying semantic structure of column values unseen during pretraining, allowing it generate novel yet realistic values beyond the pretraining distribution.
After providing key evaluation setup details in Section 3.0 we compare Tabby to a broad array of prior works on diverse labular datasets in Section 3.1 to evaluate Claim 1. As Tabby may be applied to any transformer- based LM, we explore Claim 2 for LMs of varying sizes in Section 3.2 To demonstrate Claim 3, we apply Tabby a nested (JSON) dataset Scction 3.3. In Scction 3.4 we investigate how Tabby adapts to individual columns within a dataset during finetuning as demonstration of Claim 4. Lastly, we demonstrate Claim 5 in A Appendix E.5
4

---

Published in Transactions on Machine Learning Research (01/2026)
Table I: Summary statistics of datasets. The first three columns list the number of rows each data split, while the next two columns display the number of categorical versus numerical features, respectively. The rightmost column details whether the dataset considered classification (C) or regression (R) task in downstream evaluations.
N Train
N Validation
N Test
# Cat.
# Num.
Task
Diabetes (Kahn 1994
576
57
8
Travel TTeaashii
2023
715
71
168
4
2
C
Adult (Becker Konavi
1996
36631
3663
8548
Magic [ock 200
17117
1711
1902
O
10
C
Shoppers (Sakar & Kastro
2018
11097
1109
1233
10
Abalone Nash al. 1994
3132
313
732
7
R
Rainfall Zamann 2018
12566
1256
2933
2
1
R.
House Pacc Barry
1997
15480
1548
3612
8
R
3.0 Setup
We detail here our experiments' essential information, including baselines, evaluation datasets and metries. Additional details are located in Appendix D
Baselines and Comparisons: We evaluate variety of recent tabular synthesis techniques.
LLM Approuches: Prior LLM table synthesis approaches are limited to the development of training techniques. We compare Tabby and Non-Tabby LLMs trained under three different paradigms:
1. Our lightweight and simple Plain training paradigm, detailed in Section 2.2
2. GReaT (Borisov et al. 2022), which encodes tabular data similarly to Plain, but permutes the orders in which columns are presented in training and imposes some conditional restrictions at sample lime. For more details, see Section
3. GRea'T combined with TapTap Zhang et al. 2023 and Tabula Zhao et al. 2023 this combination ion as GTT. Tap pretrains the LLM on tabular data, while Tabula encodes each categorical column into an ordinal format by replacing each unique column value with an integer.
We abbreviate
To align with the prior works [Borisov et al. 2022 Zhang et al. 2023 Zhao et al. 2023 LLM methods use Dist (DGPT2) (Radford et al. 2019) as base model unless ot stated.
Non- LLM Approuches: To represent Lon tabular synthesis techniques, we include CTGAN+ (Zhao et al. 2022). CTGAN (uu et al. 2019) and TVAE (uu al. 2019), the leading GAN and VAF. approaches, as well diffusion models TabSyn (Zhang al. 2024), TabDiff (Shi al. 2025). Tab-DDPM (Kotelnikov et al. 2024). Although diffusion models SOTA
et al. 2022 and Forest Diffusion Jolicoeur-Martinee
approach to achieving high MLE scores, they do so under strong assumptions and are incompatible with many tabular datascts-sce Figure and Section 4
Additional details on how models are trained and sampled are available in Appendix D
Datasets: We evaluate Tabby eight common tabular datasets, which are summarized in Table D The majority of these datasets are standard for the evaluation of tabular synthesis techniques, allowing for easy comparison with prior approaches. For more information on these datasets, see Appendix B
Metrics: We focus on machine learning efficacy (MLE) (Dankar et al., 2022), the standard metric for quantitative evaluation of synthetic tabular data. brief, MLE compares the performance of downstream classifiers that were trained using either real or synthetic data.
Our MLE results in the following sections are interpreted as follows: the downstream classifier that is trained using non-synthetic, real data is considered the upper bound and any MLE score higher than this "Non- Synthetic" classifier's score considered the best. If no score surpasses the "Non-Synthetic" score, then any highest score is considered the best. Figure summarizes the MLE calculation
5

---

Published in Transactions on Machine Learning Research (01/2026)
Train Data
Generative
Synth
Model
Data
Test
Data
Downstream
Downstream
model
Eval
MLE
Eval
model
Figure 2: Process for calculating our primary metric, Machine Learning Efficacy (MLE). We train a generative model, which produces synthetic dataset. Two downstream classifiers trained: one on the generative model's training data and the other the synthetic data. Each downstream model is evaluated on real test data. MLE is the difference in downstream models' test-time performance. Higher scores indicate belter-quality synthetic data.
process. In Appendix we provide a formal definition of MLE, as well as several more metrics including Shape, Trend (Shi ct al. 2025). Distance to Closest Record (DCR) and Membership Inference Attack (Yao et al. 2025 performance. These metrics additionally compare the synthetics abilities to preserve trainset. distributions without memorizing the trainset.
Our evaluation involves a comparison between 14 synthesis methods (including Tabby) across 8 datasets. So, while we do report final scores on each task individually, we would also like to understand which method performs the best across all of the tasks in our evaluation.
Aggrogation of results:
To do so, we aggregate MLE scores using performance profile curves (Dolan & Moré 2002), a robust way to visually compare scores across noisy evaluations in large number of environments. Performance profiles improve over simpler aggregation techniques, such averaging scores or computing the average rank of methods across tasks. Specifically, performance profiles are useful when scores for different tasks might be different scales (which can be an issue with averaging scores), and can take into account methods that are extremely close to the best-performing method on task without dropping them a full rank (which can be problematic when averaging ranks).
To summarize these curves, we also calculate the area under the performance profile (AUP) scores (Roberts et al., 2022), which serve as final ranking of methods. In short, the performance of synthesis method across all eight datasets may be represented just one performance profile curve. Methods with better performance will have higher curves and, therefore, higher AUP scores. As such, the method with highest AUP score is considered the best overall method. Details on performance profiles are in Appendix D)
3.1
Tabby versus Baseline Synthesis Methods
We begin by validating our first claim.
Claim 1: Plain-trained Tabby models generate higher-quality tabular data than prior approaches.
Setup: Table 2 lists MLE for each dataset. For classification datasets (Diabetes, Travel, Adult), the reported metric is the accuracy of the downstream random forest classifier, while for regression datasets (Abalone, Rainfall, House), we report the cocfficient of determination R. of the downstream random forest regressor.
The "Non-Synthetie" row corresponds to the performance achieved by training the downst classifier or regressor on real instead of synthetic data. We consider this row be a performance ceiling for synthetic approaches. Any model and training technique that achieves MLE equal to or better than "Non-Synthetic" is considered to be top-performing approach and is presented in bold.
We find that Plain-trained Tabby models achieve the highest MLE in 5/8 datasets. Further, Tabby reaches upper-bound performance on Diabetes, Travel, Adult, Magic and Shoppers, indicating that Tabby synthetic data is a capable stand-in for real data similar scenarios for these datasets.
Results:
We also find that Plain is the best- berf-perfo technique for training tabular LLMs in almost all cases: for all eight datasets, the highest-scoring LLM is trained using Plain. Plain-trained Tabby MH models demonstrate the highest MLE among all LLM archilectures and training styles.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2:
Machine Learning Efficacy (MLE, 1). The "Non-Syuthelie row is upper-bound
performance given by real, non-syuthetic data. Top results (or any higher than upper-bound) are bolded. The number of datasets that model achieves top performance on are counted in the "# Best" column. An asterisk indicates that at least one of three runs did not produce valid samples. Tabby models are presented in italic. The best-performing Tabby model, Plain Tabby MH DGPT2 is presented in purple and achieves best performance on 5/8 datasets. Terminology glossary in Appendix A
Diabetes
Travel
Adult
Magic
Shoppers
Abalone
Rainfall
House
Best
Non- -on-Synthetic
0.73
0.87
0.85
0.82
0.88
0.45
0.54
0.61
CIGAN 0.39 1 0.00
0.43 1 0.33
0.76 1 0.00
0,58 1 0.07
0.85 1 0.00
0.01 1 0.01 0.00 1 0.00
0.00 1 0.00
0
CTGAN+ 0.62 = 0.00
0.81
0.00
0.76
0.00
0.64 + 0.00
0.85 0.00
0.24 + 0.03
0.22
+ 0.15
0.55
+ 0.00
0
TVAE 0.62 + 0.00
0.81
0.01
0.71 + 0.04
0.85 0.00
0.07
+ 0.03
0.00
0.00
0.05 + 0.09
0
CLLM 0.74 0.02
0.00
0.02
0.17
0.00
0.89 + 0.01
0.00 : 0.00
0.00
N/A*
2
TabSyn 0.65 1 0.01 0.13
0.80
0.81 + 0.03
0.88 1 0.01
0.23
0
0.60
1 0.01
1
TabDiff 0.75 1 0.02 0.86
16610000
0.01
0.82 1 0.03
0.89 0.01
0.41
0.43
0.61 1 0.00
4
Forest Diffusion 0.76 1 0.00 0.86 1.001
0.00
0.64 0.00
0.85
0.00
0.35
0.45
0.56 0.01
1
Tab-DDPM 0.75 1: 0.02 0.87
0.04
0.81 1.000
0.86
0.41
0.54
0.43
3
0.00 Plain Base 0.75 : 0.02 0.86
0.80
0.89
0.44
0.01
0.52 + 0.03
0.53 + 0.08
Plain Tabby MH 0.74 = 0.00 0.8
0.82 + 0.02
0.89
0.43
0.01
0.49 = 0.00
0.60 = 0.00
GReaT Base 0.62 1 0.01
0.85 0.00
0.01
0.80 0.01
0.87 0.00
0.41
0.01
N/A*
0.56 i 0.01
0
GReaT Tabby MH 0.64 1 0.01
1.001
0.00
0.81 0.01
0.89
0.01
0.40
0.01 0.00 1 0.00 0.56 1 0.01
GTT Base DGPT2 0.72 1 0.06
0.87
0.02
0.83 0.001
0.79
0.01
0.87 0.00
0.40
1 0.01
0.05 1 0.01
0.55 1 0.02
1
GTT Tabby MH 0.62 + 0.00
0.85
+
0.01 0.76 0.07
0.81 + 0.02
0.88 +
0.00 0.37 1 0.02 0.26 + 0.37
0.53 + 0.00 1
For the Rainfall dataset, pre-existing LLM tabular training techinques introduce undesirable effects. Entries marked by an asterisk (*) for this dataset indicate that at least one of three runs were unsuccessful in synthesizing any valid samples. Particularly, the Non-Tabby GReaT model is unable to produce valid samples in any of the runs, Meanwhile, each Plain-trained model is successfully sampled and outperforms all GReaT or GTT-trained models in all three runs, indicating that Plain-trained Tabby models are capable of modeling complexities within the Rainfall dataset that pre-existing LLM-based tabular synthesis works are unable to capture.
The performance profile curves in Figure 3 support our findings. In particular, Plain-trained Tabby MH achieves the highest. AUP score. This indicates that Plain-trained Tabby MH performs the best among all methods when comparing across all datasets.
Performance Profile Analysis:
Further, we see that the top two synthesis approaches are the two Plain-trained models, which surpass the prior SOTA method of TabDiff. Given that these models rely on fewer assumptions than diffusion approaches, and are simpler to train than the GTT or GReaT LLMs, we find that both Tabby MH and Plain training are powerful advancements for the task of tabular data synthesis.
Comparing Multivariate Modeling Capabilities: We further compare the multivariate modeling capabilities of Tab-DDPM, Plain-trained Tabby MH and the prior top- pop-performing based approach of GReaT-trained Non-Tabby with TapTap and Tabula in Figure We plot the House dataset's target column (Median House Value) as a function of its most predictive feature in the dataset (Median Income), for (left to right) real data, Plain Tabby MH, non- Ton-Taby GTT and Tab-DDPM.
Tab-DDPM plot (left) differs the most from the real data (right) because the Tab-DDPM model only supports integer-valued regression targets. Accordingly, both LLM-based approaches more accurately capture the target column's distribution than Tab-DDPM.
Meanwhile, GReaT sampling (center left) constrains that the target column distribution in the training dataset is replicated in synthetic data, by prompting the model with target values selected randomly based on their frequency in the training data. Accordingly, GReaT models will not generate target values outside those in the training data, which can be undesirable for datasets with few rows or limited target column coverage. In contrast, Plain training (center right) allows the model to generate previously unseen target.
7

---

Published in Transactions on Machine Learning Research (01/2026)
1.0
Fraction of problems palt)
0.8
AUP 0.75, Non-Synthetic
0.6
AUPm: 0.75, Plain Tabby MH
AUP,: 0.72, Plain Base
AUP 0.71. TabDiff
0.4
AUP 0.67, Tab-DDPM
AUP, 0.58, GTT Base
AUPm GReaT Tabby
0.2
AUP 0.53, GReaT Base
AUP 0.53, GTT Tabby MH
AUPm TabSyn
0.0
0.0
0.1
0.4
0.5
0.6
0.7
log(t)
Performance profile curves and AUP scores across computed using the MLE scores on our evaluation lasks. The top performing method is Tabby MH DGPT2 with Plain training.
Figure 3:
Tob.DDPM
GTT Non-Tabby
Plain-Trained Tabby MH (Ours)
Real Data
Medlan Houuee House vatue
0.0
2.5
5.0
7.5
10.0
12.5 15.00.0
2.5
5.0
7.5
10.0
12.5
15.00.0
255
5.0
7.5
10.0 125
15.00.0
2.5
5.0
7.5
10.0
12.5
15.0
Modiaa
Figure 4: The House dataset's target Median House Value column function of its most-predictive feature, Median Income. Left to right: synthetic data from Tab-DDPM, the prior best LLM-based method and Plain Tabby MH, followed by the original data distribution. Tabby demonstrates a similar distribution to the real data and to GTT, but without GTT's assumptions (listed in Section 3.1
values. The improved modeling capacity of Tabby over the to effectively capture the overall target column distribution.
model allows Plain's 's sampling approach
3.2 Investigating the Choice of Base Model
We now lurn to our second claim.
Claim 2: The Tabby architecture modification allows smaller LLMs to achieve similar or better synthetic data fidelity than LLMs with higher parameter counts.
Comparisons: We compare synthesis quality across LLMs of varying sizes. We consider 7 LLMs, listed in Table 3 evaluating Non-Tabby and MH versions of each. Each model is Plain-trained under conditions provided in Section 3.0 then sampled 500 times. Results averaged across two runs. Llama models use
LoRA (u et al. 2021 all linear transformer layers, with the LM head fully fine-tuned.
Results: Table 3 and Figure display results for the Travel dataset, with results for Diabetes and House (plus additional metries and results for GReaT training) in Appendix E.3
We find that Tabby improves MLE or maintains upper-bound MLE for 6/7 models, without nccossarily increasing the cost of inference (Section 2.4). Although higher- parammeer models are generally correlated with greater generative abilitics, Figure demonstrates that this not always the case: Interestingly, find that the Llama models (1.2B and 8B parameters each), have lower average MLE than smaller models.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: MLE for Base and MH versions of 7 LLMs with varying parameter counts, for the Travel dataset. Results higher than Non-Syuthetic are presented bold. Tabby improves or maintains upper-bound MLE for 6/7 models. The parameter count of Tabby MH model for V-column dataset equals the parameter count of the base model without its head, plus V times the parameter count of the base model's head. Results for Diabetes and House (and additional metries) in Appendix E.3
MLE (t)
Params
Non-Synthetic (Upper Bound)
0.87
Base Pythia 14M
0.86 0.01
14M
Tabby MH Pyliia 14M
0.82 1 0.02
53M
Base Distilled-GPTY
0.88 = 0.00
82M
Tabby MH DistilledGiPT
0.89 + 0.02
310M
Base GPT2
0.89 + 0.01
120M
Tabby MH GPT2
0.87 0.01
360M
Base Pythia 160M
0.87 0.01
160M
Tabby MH Pythia 160M
0.86 + 0.00
390M
Base Pythia 410M
0.86 + 0.02
410M
Tabby MH Pythia 410M
0.88 + 0.03
710M
Base Llama 3.2 1B
0.82 = 0.01
1.2B
Tabby MH Llama 3.2 1B
0.84 0.02
2.8B
Base Llama 3.1 8B
0.84 = 0.01
8.0B
Tabby MH Llama 3.1 8B
0.86 + 0.03
11B
Tabby offers favorable performance improvements relative to the scaling curve and allows even small models to better outperform large, resource-intensive models.
3.3 Extending Tabby Beyond Tabular Data to General Structured Modalities
While tabular data is frequently overlooked in contemporary machine learning research, related structured modalities such nested data receive even less attention. While GReaTT, TapTap, Tabula, CTGAN and TVAE are focused solely on tabular data and do not clearly extend beyond tables, we demonstrate that Tabby can be generalized to address our third claim.
Claim 3: Tabby architecture modifications may also be applied to other structured data beyond tabular data, resulting in higher-quality synthetic data for these modalities as well.
We plain-train non-Tabe non-Tabby and Tabby MH models on a JSON dataset of patients being evaluated for Glaucoma (Manoj 2024). Each datapoint has 10 fcatures, organized in groups: a group of columns representing qualitative aspects of the optic nerve, group of 2 columns corresponding to measuremenss between the optic nerve and eye, and standalone feature for the diagnosis (examples in Box D The binary classification target is inside the first group and assesses whether the optic nerve is thinning. As with tabular datasets in Section 3.1 and 3.2 we train downstream classifiers to predict the target variable and then present the resulting ML.E.
Comparisons:
We also consider the discrimination metric: Given equal numbers of real and synthetic samples, we mcasure the accuracy of a discrimination classifier that is trained to distinguish real versus synthetic datapoints. Because 50% accuracy would indicate that the classifier fully unable to distinguish real from synthetic, we report the accuracy's distance from 50% in Table 4 that lower scores indicate higher-quolle synthesis.
Results: Table demonstrates that Tabby MH improves MLE parity with real data. Tabby MH's lower discrimination signifies this model's samples more realistic than non-Tabby samples.
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table MLE and Discrimination scores for Plain-trainee Base and MH models on a dataset of JSON records. Each record contains diagnostic information of glaucoma sufferer or healthy patient.
MLE (t)
Discrim. (L)
Non-Synthetic (Upper Bound)
0.97
CTGAN
0.52
0.46
Forest Diffusson
0.95
0.31
Tab-DDPM
0.94
0.45
Base DGP12
0.93
0.06
Tobby MH DGPT2
0.97
0.01
3.4 Tracking the Adaptation to Individual Columns
We address our fourth claim by examining Tabby's progress while fine-tuning on tabular data. Claim Tabby's loss formulation allows for convenient tracking of per-column performance at training time, leading to better understanding of model behavior.
For three runs, we train a Tabby MH model on a subset of the House dataset containing 5160 rows and six columns. We log the individual columns' losses on the evaluation dataset every 2500 steps while training for 10 epochs, then average across the runs.
Setup:
Results: Individual column losses shown in Figure This information be vital to understanding model behavior and training progress, as elaborated in Section E.4
3.5 Discussion
We find that Tabby models synthesize high-quality data in variety of settings. In particular, Plain-trained Tabby MH consistently outperforms all prior LLM-based approaches and is comparable to better than diffusion-based approaches in most settings, despite Tabby enjoying greater flexibility under fewer assumptions than those made for diffusion models. The Tabby architecture modification allows LLMs to better model both univariate column distributions and multivariate relationships across columns.
Unusually, we find that the baseline Plain training technique with Distilled-GPIT performs quite well on several standard evaluation dalasets. The high performance of the Plain training technique compared to prior LLM works on tabular synthesis, which are more complex, is surprising. Notably, GPT-2 and related models can even outperform much larger models such as Llama on these benchmarks. A plansible explanation that tabular generative ability may depend on the proportion of parameters allocated to the LM head relative to the rest of the model about 33% in GPT-2 versus roughly 7% in Llama-8B suggesting that heavier output heads may better capture the structured dependencies characteristic of tabular data.
As of this writing, the Adult, House and Diabetes datasets have become quite prevalent for tabular synthesis evaluation. We hope that future research will build off of our evaluation setup by continuing to include more diverse and challenging tabular datasets, along with extensions to other structured modalities.
Limitations: We highlight two additional priorities for future work:
Privacy preservation is important for trainscts that contain sensitive data, such as patient modical information. While Tabby's privacy preservation is similar to prior works (Appendix E a method with strong formal privacy guarantees an important next step for privacy-critccal applications.
Comppuatiionalll-onnttttaaai environments tasks with particularly large datasets may require deep learning approaches that are specifically designed with efficiency in mind. The Plain training method and insights from Section 3.2 may be useful towards this priority, but we leave further experimentation and the efficient implementations detailed in Section 2.4 to future work.
10

---

Published in Transactions on Machine Learning Research (01/2026)
4 Related Work
Tabular data has played central role in machine learning since the field's early days. In particular, decision trees and relatives (ong & Lu 2015) are well-adapted to table classification or regression. Table synthesis is a growing area, though frequently overlooked in favor of image and text synthesis.
Classical synthesis: Classic machine learning models, such as random forest models or Bayesian networks, may be used to synthesize tables (Reiter 2005 Zhang et al. 2017), but are limited in the data types and distributions that may be represented.
Generative Adversarial Networks (GANs): Many tabular synthesis methods rely GANs (Goodfellow 2014 Xu et al., 2019), but have encountered several limitations. In particular, distributions of ordinal columns are frequently imbalanced, leading GANs to undesirable phenomena such as mode collapse. Continuous columns may possess multiple modes and complex distributions, which GANs also struggle to capture (Xu et al. 2019).
ct al.
Diffusion Models:
Forest Diffusion (Joliceeu--Martinaau et al. 2024) and Tab-DDPM (Kotelnikov et al,, 2022) are statc-of-tho-art table synthesis approaches based on the diffusion model. Both show top performance many standard tabular metrics and are reliable for certain applications. Unfortunately, this performance is achieved by strong assumptions on the nature of the data space-for instance, numeric target variables may only assume integer values (see Figure 4 and diffuusio models are unable to model non-categorical string columns such as addresses or telephone numbers. The ability to reach table synthesis performance comparable to that of diffusion models, but with fewer assumptions, as an area of active research.
LLMs: A small, but growing, body of work has applied LLMs' flexible modeling abilities to tables. GReaTT Borisov et al. 2022) is method to convert tabular data into sentence format compatible with LLMs, then "shufffing" the order in which columns occur for each row to improve the modeling of inter-column dependencies. TapTap (Zhang et al. 2023) pretrains LMs on a variety of tabular data before fine-tuming on a downstream table synthesis lask, while Tabula (Zhao et al., (023) explores methods of preprocessing the training data to decrease sequence length. Other LLM-based works have adapted these advances to relational tables (Solatorio & Dupriez, 2023). or used the emergent abilitics of very large models such as GPT-4 to generate synthetic data using In-Context Learning in place of fine-tuning (Seedat al. 2024). Many of these works can be used in concert with Tabby, as demonstrated in Section B and they offer the additional advantage over other architectures of enabling pretrained tabular models to adapt effectively to new datasets.
MoE Architectures: The key innovation of Tabby is the application of Gated Mixture of Expert (MoE) layers Shazeer et al. 2017 Masoudnia & Ebrahimpour 2014) for LLM table synthesis. MoE layers have
enjoyed utility in multitask (Ma et al. 2018 Gupta et al. 2022) and multimodal learning (Zhao et al. 2024
Park et al. 2018), by creating sets of model parameters dedicated to specific task.
5 Conclusion
Tabby is an MOE-based architecture modification that allows LLMs to generate realistic tabular data. Tabby reaches MLE parity with real data in 5/8 datasets. We hope this promising performance spurs future work on architecture modifications that allow LLMs to represent structured data.
Broader Impact Statement
Tabby is an improvement in the realism of synthetic tabular data, with extensions to non-tabular structured data. As such, Tabby (and its downstream applications) will have positive impacls on tasks that require synthetic structured data, such as low-data downstream tasks or privacy-critical tasks. However, Tabby may also be uscful for negative downstream applications, such as the falsification of data-a drawback common many advancements in the realism of generative modeling.
11

---

Published in Transactions on Machine Learning Research (01/2026)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 4.X7:7333.08774, 2023.
Barry Becker and Ronny Kohavi. https://doi.org. 10. 24432/C5XW22
Adult. UCI
Machine Learning Repository, 1996.
DOL:
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable
Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, November 2023. URL http:
15127 arXiv:2311.15127 [cs].
R. Bock. MAGIC Gamma Telescope. https:://do..org/10.. 111.1100222425555 C52CCB.
UCI
Machine
Learning Repository, 2004.
DOL:
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelozyk, and Gjergji Kasneci. Language Models are Realistic Tabular Data Generators. September 2022. URL https://openreriew.nees
http:/pparrrriiee..eeee/rrrrrtct net/forummid
cEygmQNOeI
Fida K Dankar, Mahmoud K Ibrahim, and Leila Ismail. A multi-dimennsional evaluation of synthetic data generalors. IEEE Access, 10:11147-11158, 2022.
Maria F. Davila, Sven Groen, Fabian Panse, and Wolfram Wingerath. Navigating Tabular Data Synthesis Research: Understanding User Needs and Tool Capabilities, May 2024. URL http://arxiv.orr wrw. / 2405. 20959 MeXiiv2445.20953 [es].
Elizabeth D. Dolan and Jorge J. Moré. Benchmarking optimization software with performance profiles. Mathemalical Programmingg (2):20011 213, January 2002. TSSN 1436-4646. doi: 10. 10.1007/<101070100266
URL httpp://arxii..orr/aa/////0/0110
Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large Language Models(LLMss) on Tabular Data: Prediction, Generation, and Understanding Survey, February 2024. URL http://arxiv.. http://wwivv....raaaeeeo att.//bb//42..2994// arXiv:2402.17944 [cs].
Eyal German. Tab-mia: A benchmark for membership inference attacks on tabular data, 2025. URL https://githul: heg.st/bt.co//rergggeaaaaaaaaaaa
Ian Goodfellow, Jean Couget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Shashank Gupta, Subhabrata Mukherjee, Krishan Subudhi, Eduardo Gonzalez, Damien Jose, Ahmed H. Awadallah, and Jianseng Gao. Sparsely activated mixture-of-experts are robust multi-lask learners. ar preprint 2Xix:2204.07688 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXie preprint ar arXiv:2106 09685, 2021.
Alexia Jolicoeur-Martineau, Kilian Fatras, and Tal Kachman. Generating ing and Imputing Tabular Data via Diffusion and Flow-based Gradient-Booste Trees, February 2024. URL http://arxiv, org/ absaabas///00 09968 ar arXiv:2309.0948 0999999999988 [cs].
Michael Kahn. Diabetes. UCI Machine Learning Repository, 1994. URL http:/tttss::.oooeeeeeeeee wwt.:oopo......ssrrec..e.. org/search?
byyerdatakorrtmmmaaaill//sssssse DOL: https://doi..rrr//00 10t.:/.0rr110..422/// C5T59G.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Akim Kotelniov,, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling Tabular 15421 arXiv:2209154221
Data with Diffusion Models, September 2022.
[es].
Anton D. Lautrup, Tobias Hyrup, Arthur Zimek, and Peter Schneider-Kamp. Syntheval: a framework for detailed utility and privacy evaluation tabular synthetic data. Data Mining and Knowledge Discovery, URL hattps://doi.org//
39(1):6, December 2024. ISSN 1573-756X. doi: 1007/s10118002-01111
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference knowledge discovery & data mining, pp. 1930 1939,1999,0000 2018
Aswanth Manoj. Glaucoma Diagnosis JSON Analysis Dataset Hugging Face, 2024. URL https:// jatprii__sooo_aaaaiiiiis analysis.
huggingface buggingface.co/daaasssee http.////ttttttttttttttt//aaaana diagnosiidasiiiissemeomao baagmosis__jso______
Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: literature survey, Artificial Intelligence Review, 42:275 293,, 2014.
Warwick J Nash, Tracy L Sellers, Simon R Talbot, Andrew J Cawthorn, and Wes B Ford. The population biology of abalone (haliotis species) in lasmania. i. blacklip abalone (h. rubra) from the north coast and islands of bass strait. Sea Fisheries Division, Technical Report, 48:p41,, 1994.
R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3): 291-297, 1997.
David Keetae Park, Seungjoo Yoo, Hyojin Bahng, Jacgul Choo, and Noscong Park. Megan: Mixture of experts of generative adversarial networks for multimodal image generation. arXiv preprint arXiv:1805. 024881, 2018.
Zhaozhi Qian, Dogdan-Constaniie Cebere, and Mihaela van der Schaar. Synthcity: facilitating innovative use cases of synthetic data in different data modalities, 2023. URL
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Tlya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188):9, 2019.
Siamak Ravanbakhsh, Jeff Schncider, and Barnabas Poczos. Equivariance through parameter-sharing. In International conference on machine learning, pp. 2892 2901. PMLR, 2017.
Jerome P Reiter. Using cart to generate partially synthetic public use microdata. Journal of Official Statistics Stockholm, 21(3),441, 2005.
Nicholas Roberts, Samuel Guo, Cong Xu, Ameet Talwalkar, David Lander, Lvfang Tao, Linhang Cai, Shuaicheng Niu, Jianyu Heng, Hongyang Qin, Minwen Deng, Johannes Hog, Alexander Pfefferle, Sushil Am- managhatta Shivakumar, Arjun Krishnakumar, Yubo Wang, Rhea Sukthanker, Frank Hutter, Euxhen Hasanaj, Tien-Dung Le, Mikhail Khodak, Yuriy Nevmyvaka, Kashif Rasul, Frederic Sala, Anderson Schneider, Junhong Shen, and Evan Sparks. Automl decathlon: Diverse tasks, modern methods, and efficiency at scale. In Marco Ciccone, Gustavo Stolovitzky, and Jacob Albrecht (eds.), Proceedings of the NewrlPS 2022 Competitions Track, volume 220 of Proceedings of Machine Learning Research, pp. 151 170. PMLR, 28 Nov-09 Dec 2022. URL
mlr h.t..prrss/v2200 wttp//////rrrbttttaaa.aa html
C. Sakar and Yomi Kastro. Online Shoppers Purchasing Intention Dataset. UCI Machine Learning Repository, 2018. DOL: https://doi.org//0.244/22. S888.
Naboel Scedat, Nicolas Huynh, Boris van Breugel, and Mihacla van der Schaar. Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes, June 2024. URL http:
12112 arXiv:2312.12112 [os).
13

---

Published in Transactions on Machine Learning Research (01/2026)
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th annual meeting of the associalion for compulational linguisties (volume 1: long popers), pp. 1715 1725, 2016.
Noam Shazeer, Azalia. Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. orXic preprint arXiv:1701.0633 2017.
Juntong Shi, Minkai Xu, Harper Hua, Hengrui Zhang, Stefano Ermon, and Jure Leskovec. TabDiff: Mixed-type Diffusion Model for Tabular Data Generation, February 2025. URL 2410. 20626 arXiv:2410.2066 [cs].
org/abs/
Aivin V. Solatorio and Olivier Dupriez. RFal.TabFormer: Generating Realistic Relational and Tabular Data
using Transformers, February 2023. URL httsp://arxiv.org//bbb//// 02041 arXiv:2302.00411 [cs].
Yan-yan Song and Ying Lu. Decision tree methods: applications for classification and prediction. Shanghai
Archives of Psychiatry, (732):130 135, April 2015. ISSN 1002-0829. doi: 10.19919)
URL https: ncbi nlm. mih htt.g/////cc/aaccceee////o//////
Tejashvi. Tour & Travels Customer Churn Prediction, 2023. URL https///wwwwkgggee.../ataaataee
Boris van Breugel and Mihaela der Schaar. Why Tabular Foundation Models Should Be Research Priority, May 2024. URL. htttss://axxi: wwwpaawwwvv.oooaaaaaaaea abs 2405 01147v2
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, and Hao Wang. HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection, August 2024. URL
0292277770000777 arXiv:2408. 0009277 [cs] version:
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante and Kalyan Veeramachanenii Modeling Tabular data using Conditional GAN, October 2019. URL stat].
arXiv:1907.00503 [cs,
Zexi Yao, Nataša Krčo, Georgi Ganev, and Yves-Alexandre de Montjoye. The der delusion: Measuring the privacy risk of synthetic data, 2025. URL httt::://axxiv.or//ab//////
Yousuf Zaman. Machine Learning Model on Rainfall A Predicted Approach for Bangladesh. 2018.
Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-Type Tabular Data Synthesis with Score-based Diffusion Latent Space, May 2024. URL
23110.9666666. arXiv:2310.0965) [cs].
Jun Zhang, Graham Cormode, Cecilia M Procopiue, Divesh Srivastava, and Xiaokui Xiao. Privbayes: Private data release via bayesian networks. ACM Transactions on Database Systems (TODS), 42(4):1-41 2017.
Tianping Zhang, Shaowen Wang, Shuicheng Yan, Jian Li, and Qian Liu. Generative Table Pre-training Empow- ers Models for Tabular Prediction, May 2023. URL httt:://pprpiiiii wwwpxivvvorg///o httpg/ab/23305.0000 09696 arXiv:2305.09696 [cs].
Xueliang Zhao, Mingyang Wang, Yingchun Tan, and Xianjic Wang. Tgmoc: A text guided mixturo-of- experts model for multimodal sentiment analysis. International Journal of Advanced Compuler Science & Applications, 15(8), 2024.
Zilong Zhao, Aditya Kunar, Robert Birke, and Lydia Y. Chen. CTAB-GAN+: Enhancing Tabular Data Synthesis, April 2022. URL http:://axxiii org/ abs 2204. 00401 ar XrXiv:2204.04401 [cs].
Zilong Zhao, Robert Birke, and Lydia Chen. TabuLa: Harnessing Language Models for Tabular Data Synthesis, October 2023. URL http://arxiv. http://aardv.oorgaa///211.11 12746
12746
arXiv:2310.12 12746
[cs] version: 1.
14

---

Published in Transactions on Machine Learning Research (01/2026)
A Terminology Glossary
For convenience, in addition to definitions within the main text, list and define the most frequently-used terms and abbreviations in our paper here:
DGPT2: Distilled-GP12 (Radford al. 2019
Distance to Closest Record (DCR): Metric for synthelic data quality and privacy, defined in
Appendix D
Discrimination: Metric for synthetic data quality, defined in Appendix D
GReaT [Borisov et al. 2022: The landmark work fine-tuning pre-existing LLMs to synthesize tabular data by encoding datapoints as text. Similar to Plain training, but includes train-time complications such as shuffling the order in which columns are encoded and sample-time complications such as the inability to generate label values that do not occur in the training dataset. Discussed in-detail in Section 4
+aeaT+Tabula (GT): The combination of GReaT training plus Tabula (Zhao et al. 2023 data
encoding; Section
The combination of GReaT training plus Tabula encoding and TapTapp (Zhang et al. 2023 pre-training on tabular data (which is performed after the LLM is pre-trained on text data).
TapaT+TapTap Tabula (GTT):
Tabula
Low Rank Adapters (LoRA): Parameter-efficient training method from (Hu et al. 2021
Mixturo-of-Experta (MoE): Architecture technique which replaces one block with set of specialized blocks; see Section !
Multi-Head (MH): The best-performing variant of Tabby, which replaces the LLM's language model
output layer with an MoB layer.
Machine Learning Efficacy (MLE): Our primary evaluation metric, introduced in Section 3.0 and
discussed in-detail in Appendix D
Multi-MMLP (MMLP): Tabby modification that applies MoE the transformer blocks' MLPs.
Multi-MLP and LM Head (MMLP+MH); Tabby modification that applies MoEs to both the transformer blocks' MT.Ps and to the language model output layers.
Used for the MLE metric, this score represents the performance of a downstream classifier trained on real, instead of synthetic, data. See Appendix D for details.
Non-Synthetic (Upper Bound):
Non-Tabby (NT): An LLM without the Tabby modification, also referred to Base LLM.
Plain: Our simple but high-performing technique for training LLMs on tabular data; introduced in Section 2
Tab-DDPM (TDDPM): A state-of-the-arr tabular synthesis technique based on the diffusion model architecture, which relics several important assumptions; Scction
B
Additional dataset information
We select variety of tabular datasets for our evaluations, with two goals in mind. First, the inclusion of the most standard tabular
Diabetes, Adult and House-alloo for easy comparison with prior works. Second, we include classification and regression datasets from a variety of domains, such as Earth science (Rainfall), business (Travel) and medicine (Diabeles). This diversity allows us to demonstrate that Tabby models achieve high performance across a variety of real-world data types and distributions. Refer to Table for download links to cach datasct.
Diabetes Kahn 1994] contains medical information on female hospital patients, including age, mumber of pregnancics and skin thickness. Downstream models learn to predict whether given paticnt suffers from diabetes. Apart from the label, all dataset columns are numerical, with some columns taking only integer values, while others floats.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Download links for each dataset.
Dataset
Link
Diabetes
Travel
Adult
Magic
Shoppers
Rainfall
Abalone
House
https:/ hattps:/sccikee
Glaucoma
https://huggingfaree http://ww...///cctteeeeeeme datasets /aae///aaaaaaaa/aaaaaaaaeee
Travel (Tejashvi 2023) was collected by a travel agency wishing predict customer churn. With the binary variable churn the target, features include whether the traveler booked hotel, frequent flyer status and traveler age. While most features are categorical, there are two numerical columns: traveler age and the number of times that the customer has used the travel agency in the past.
Adult (Becker & Kohavi 1996) is dataset commonly used to benchmark tabular classification algorithms. Each row contains basic information one American adult, such as their age, years of education and marital status. For each adult, the downstream task is to predict whether their annual income is above or below 850, 000. The features are a mix of categorical and numerical columns, with each numerical column taking only integer values.
The Magic (Bock 2004) dataset consists of Monte Carll-simulated observations from the Cherenkov gamma telescope, designed to differentiate high-energy gamma-ray events from background hadronic showers. Each row represents a single event characterized by 10 continuous features derived from Cherenkov image paramcters, such as length, width, and asymmetry. The target variable binary, indicating whether the event originated from gamma signal or a hadronic background, making the dataset suitable for evaluating classification performance on sayyics-based data.
The Shoppers (Sakar & Kastro 2018) dataset contains session-level data from an e-commerce website over a one-year period. Each record describes single visit using 18 behavioral and contextual features, including page visit counts and durations, bounce and exit rates, visitor type, month, and weekend indicator. The target variable is binary, indicating whether the session resulted in purchase, making the dataset suitable for evaluating classification methods imbalanced user-behavior data.
Our first regression datasct Abalone (Nash et al. 1994). which records the basic measurements of abalones, such weight and height. The larget variable is the abalone's age.
The Rainfall [Zaaan 2018) dataset, while challenging to many LLM-based synthesis methods, contains only four columns which record historical weather data in Bangladesh. Its target variable is the amount of rainfall recorded, and the features are the year, month and weather station location.
House Pace & Barry, 1997) a standard regression dataset. Each row represents a block of houses in California during the 1990 census. The dataset records the number of households residing in the block, the block's median building age, average number of bedrooms, and other basic information. The dataset's target column is the block's median house value, which is numerical and allows to assess Tabby's synthetic data in regression task.
Glaucoma (Manoj 2024) dataset consists of JSON records describing ophthalmic patients under consideration for glaucoma diagnosis. Each record contains various qualitative and quantitative information about the eye, demonstrated by the examples in Box D
C Tabby for Nested Data
Figure provides visualization of the Tabby architecture used in Section E.4 to generate nested JSON data.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Tabby Multi-Head For Nested Data
Transformer Block
Transformer Block
Transformer Block B
Nested
Nested
Head
Head
Head
Head
Head
Head
2
3
10
Figure An overview of the Tabby MH modifications for the nested Glaucoma dataset.
D Details on Experimental Setup
Calculation of results: The reported result. for each model and training setup is the average across three training runs, where not otherwise stated. For each of the three trained models, we sample 10, 000 datapoints, compute evaluation metrics separately for the three resulting synthetic datasets, then calculate the average metric value across all runs. For LLM approaches, each model is trained for up to 50 epochs, using early stopping when the validation loss (assessed every 5000 steps) fails to improve twice in row. We perform grid scarch to select the learning rate with lowest validation loss for each model and training setup, with selected learning rates reported in Appendix E.6. For non-LLM works, we follow the procedures detailed in each these works.
More detailed definition of Machine Learning Efficacy (MLE): Given a synthetic dataset produced by a generative model, we begin to calculate MLE by training one downstream classifier using the synthetic datasct. Then, we evaluate the performance of this downstream classificr on a real test set, drawn from the same distribution as the generative model's train set. We compare this classifier's performance a second classifier, which is trained classifier performs worse than the classifier trained on real data, then (intuitively) the synthetic data is of lower quality than the real data: for instance, the distributions of features in the real data are not well-reflected in the synthetic data.
the same training data the generative model. If the synthetically-traant trrandd
Put another way: given a real datasct, we form disjoint training and test sets, donoted R and respectivoly. A generative model is trained on R then generates synthetic dataset.
To calculate MLE, a downstream classifier or regressor KR trainod using R to prodict a predetermined label column, using all other columns as features. An additional classifier or regressor Kşs is similarly trained on S. Then, the performance of Kg and K & the real test dataset. D is evaluated: high-fidelity synthetic dataset.
17

---

Published in Transactions on Machine Learning Research (01/2026)
S will allow Ks to exhibit similar performance to Kg despite never encountering real datapoints before test-time. We report both Kn and Kg in our results, considering MLE to be the difference in performance between KR and Ks.
We random forest. classifier regressor as our downstream model K. For classification datasets, we compare the accuracy of KR and Ks, while for regression datasets, compare the coefficient of determination R². We define the coefficient of determination R² as max(1 0), where r and are the residual sum of squares and total sum of squares, respectively. This formulation means that if a model performs worse than random guessing, its R value will be represented as 0. For both the accuracy and R² coefficient metrics, a higher score indicates higher-quality data.
Information on Performance Profiles: For given method E M, its performance profile curve is defined as
Pm (T) =
T
8t.m
$T)
[7]
milmm M (t
for set of tasks and scores 8t. : E T, where lower values indicate better performance on each task. order to satisfy the requirement that lower scores are better for the MLE metric, we set 8t,m = ML.,, Then for each method, obtain final score by taking the area under the curve Pr Por(T) to obtain the AUP score
AUP = -f""em( (m(()))
with т= being the smallest 7 such that (m( = for all methods m M, and where a higher AUP score indicates better performance.
Discrimination: Discrimination (Qian et al. 2023) quantifics the degree to which the generative model introduces spurious correlations other patterns that differentiate synthetic from real data. Given the real training dataset R and synthetic dataset S, we sample the same number of rows from each. Next, we train a random forest classifier C to discriminate between real and synthetic examples. Highest-quality synthetic data will result in 50% discrimination accuracy, indicating that is unable to distinguish between R and S. For this reason, our reported discrimination scores are calculated as the absolute difference between 50% and the accuracy of discriminator C. Accordingly, lower discrimination scores represent better performance.
Distance to Closest Record: (DCR) Distance Closest Record (DCR) (Lautrup et al. 2024 quantifies
the distance between each synthetic datapoint and its most-similar example in the training set R. In addition to synthesis quality, this metric is indication of the degree to which the model memorizes samples during training. Specifically, for each synthetic example E S, we compute its distance to every training point R (using LO distance for categorical columns and L1 distance for numerical columns) and take the smallest. of these distances. The overall DCR then reported as the average of these minimum distances across all synthetic examples in SS. Lower DCR associated with high-quality synthesis, but a DCR score implics that most synthetic examples are merely copies of training dataset points memorized during training. As such, we consider the best. DCR to be the lowest nonzero score.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Box D: Representative examples from Glaucoma Manoj 2024
diaguosis's : "glaucoma', disc
disc_info's info {
'disc_ 'disc_size's size': "large',
coup_disc_tt ratio : 0.8
rim info {
'rim _al_ppalo'" : true
simm_colo's 'pale"
bayonetiug': : true,
Sharp_edge" true.
'laminar_ dot sign':
_iigaa true,
'notching's true,
"rim_thine thinninnnn" true
}.
"diagnosis'. : "normal', disc info 'disc_ 'disc_size's size': 'normal', "cup_dise ratio 0.4
"iim innof 'imm palllaalo's false,
'pink"
bayoneting'. : false
sharp_edgee false,
laminar_ dot siuussssu: ;; false, 'notching' false
'rim tim_ntinnnngg: false
}.
"diagnosis" "normal', dise info : 'ddis__izee' size': "normal', 'cup_dise ratio': dist___atiio'' 0.4
rim info
'rim _al_pallo'ss : false,
"pink".
bayoneting' : false,
Sharp_edger false
Haminar_doe doin_sinn::: false,
notching's false
rim thinning false
19

---

Published in Transactions on Machine Learning Research (01/2026)
E Further Experimental Results
E.1 Additional Metrics for Main Results Tables
For the experiment presented in Section 3.1 we include five additional metries:
1. Discrimination (Table () works similarly to a GAN's discriminator (Goodiellow et al. 2014p: the downstream discriminator model receives equal numbers of real and of synthetic datapoints, then learns to distinguish between them. The more difficulty that the discriminator model encounters in distinguishing between real and synthetic, the more we can say that the real datasct's patterns are preserved within the synthetic examples.
2. Distance to Closest Record (DCR, Table 7 measures how far the average synthetic datapoint lies from its nearest non-synthetic datapoint. If DCR equals zero, it indicates that the model has memorized its trainset, while a very large DCR indicates that the model is not preserving the trainset's patterns very well: small, non-zero, DCR scores are ideal.
Shape (Sh et al. 2025 (Table measures, for each individual column, how well the synthetic column matches the real column's distribution. We follow the SDMetrics library's implementation, which uses the Kolmogorov-Smirnoy statistic for categorical and the complement of Total Variation Distance for numerical columns, then averages the distances across all columns to report final summary number.
Trend (hi et al. 2025 (Table -which is often used in conjunction with Shape-meat measures relationships belween columns, by quantifying the degree to which these relationships the real data are preserved by the synthetic data. We again use the SDMetrics library's implementation, which measures Pearson correlation for relationships between numerical columns and complement of Total Variation Distance for relationships between categorical columns or one categorical and one numerical column (which is first binned to discretize the values).
5. Wasserstein Distance (Table 10 computes the distance between the real and synthetic datasets' numeric columns. This metric allows us to capture the similarity of relationships among all numeric columns, as opposed to the pairwise interactions measured by the Trend metric.
6. Membership Inference Attacks (Yao et al. 2025) (MIA, Table 11 assess the likelihood of synthesis method to leak the contents of its training data. We use the implementation from German (2025), which evaluates the performance of 50 different attack strategies on each synthesis method. We focus on the Abalone datasct and aggregate results by providing the maximum AUROC, mean AUROC and maximum accuracy across the 50 attacks for Tabby and other SOTA methods. For each metric, a lower number indicates a lower rate of trainset leakage and a better degree of privacy.
Runtime (Table 12 is provided for Tabby and other SOTA methods. We report the time in minutes to train for one epoch and to produce 100 samples with one NVIDIA RTX A6000. Each method is implementation available online.
run using the
These metrics largely corroborate our findings in Section 3.1 In particular, Plain Tabby MH's low DCR and discrimination scores indicate that this model's synthetic data closely resembles that of real data. Additionally, the DCR scores are small but nonzero, which indicates that the model is generating novel datapoints rather than simply repeating datapoints memorized during training.
Additionally, we provide results for the HARMONIC method (Wang et al. 2024 in Table
20

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: Discrimination metric (4), defined in Appendix D for approaches compared in Section 3.1 Tabby
produces data with better MLE without worsening the synthetic data's 's discrimination score, performing competitively with Tab-DDPM.
Diabetes
Travel
Adult
Magic
Shoppers
Abalone
Rainfall
House
OTGAN 0.42 = 0.00 0.27 = 0.01 0.48 0.00 0.34 0.03 0.50 + 0.00 0.46 E 0.00 0.18 4 0.05 0.32 E 0.06
CTGAN+ 0.34 = 0.01 0.01 = 0.52 0.52
0.22 0.02
0.50 0.50::000 0.00 0.02 + 0.47
0.47 1 0.02
0.02 + 0.00
TVAE 0.45 1 0.02 0.50 0.00 0.46 1 0.01
0.50 0.00
0.45 1 0.02
0.41 1 0.01
0.39 1 0.03
CLLM 0.28 = 0.03 0.03
0 0.39
0.30
0.03
0.37 0.37
0.02
N/A*
TabSyn 0.10 - 0.00 0.00
0.44
0.01 0.33
0.50 1..00
0.01
1.000
0.00 1 0.00
TabDiff 0.33 = 0.19 0.19 0.47 0.47
0.15 0.01
0.45
0.430.122
0.12 + 0.00
Forest Diffusion 0.27 = 0.00 0.28 0.00 0.50
0.00 0
0.04
0.00
0.09
0.00
0.16 + 0.00
Tab-DDPM 0.11 = 0.00 0.05 = 0.03 0.01 0.01 0.01 +
0.50
0.00
0.01 + 0.02 0.33 + 0.04
Plain Base 0.04=
0.03 = 0.02 0.09
0.13
0.05
0.08 0.06
0.07 = 0.06
Plain Tabby MH 0.06 = 0.02
0.02 - 0.01 0.01
0.12
0.06
0.06
0.08
0.00 0.03 1 0.01
GReaTT Base 0.28 = 0.01
0.06 0.01 0.20
0,22
0.20
0.08
N/A*
0,16 0.01
GReaTT Tabby MH 0.29 = 0.02
0.08 = 0.03 0.20 0.01
0.18 0.01
0.17 0.00
0.11 0.03
0.03
0.45 = 0.09* 0.19 + 0.01
GTT Base 0.27 = 0.02
0.07 = 0.01 0.20
0.21
0
0.20 0.01
0.05 0.01
0.39 # 0.11
0.18 # 0.03
GTT Tabby MH 0.28 = 0.02
0.07 = 0.02
0.13 0.05
0.18
+ 0.01
0.18 + 0.00
0.16 + 0.01
0.31 + 0.21
0.20 + 0.01
Table 7: Distance to Closest Record (DCR, t20), defined in Appendix D for approaches compared in
Section 3.1 Tabby MH exhibits low, nonzero scores, indicating that its synthetic examples closely resemble real data without simply copying the training data points.
Diabetes
Travel
Adult
Magic
Shoppers
Abalone
Rainfall
House
CTGAN 0.82 1 0.00 0.59 1 0.03 1.70 : 0.09 0.75 =:2000000000 0.07
1.52 0.10
0.76 0.02
0.03 0.01
0.13 I 0.02
CTGAN+ 0.52 0.03
0.52
0.04
3.24
0.66
0.3510.13
1.30
0.26
0.03 1 0.01
0.07 1 0.00
TVAE 0.27 0.001 0.00
0.16
0.21
0,41 0.05
0.03 1 0.00
0.07 1 0.00
CLLM 0.4 0.01
0.39
1.6
0.32
0.00
0.04 0.14
N/A*
TabSyn 0.47
0.4
1.79
0.28
1.79
0.38
0.26
0.01
0.06 + 0.00
TabDiff 0.43 1133:0000
0.07
0.35
0.03
0.25
0.62
0,11
0.01
0.06 + 0.00
Forest Diffusion 0.29 0.06
0.35
0.20
0.06
0.09
0.01
0.06 + 0.00
Tab-DDPM 0.63 1 0.04 0.00
0.00
0.31
1 0.03
0.19
0.01
3.31
0.12
0.1210000
0.01 0.01 1 0.00
0.08 1 0.00
Plain Base 0.01 # 0.00 0.00
0.33
0.15 0.08
0..0000000000000111000000.00000.
0.00
0.00 + 0.00 0.03 + 0.01
Plain Tabby MH 0.02 # 0.00 0.01
0.25
0.03
0.10
0.03
0.08
0.03
0.00
0.00
0.04 0.00
GReaT Base 0.33 0.00 0.02
0.12
0.03
0.18
0.00
0.10
0.00
N/A*
0.06 + 0.00
GReaT Tabby MH 0.36 0.00 0.01
0.17
0.08 0.19
0.01
0.10
0.01
0.06 + 0.00
GIT Base 0.31 0.01
0.02
0.04
0.18
0.01
0.10
0.02 1 0.00
0.06:0.000
GTT Tabby MH 0.37 + 0.01 0.02 + 0.00
0.16 = 0.07
0.19 = 0.00
0.39
0.03
0.10 + 0.00
0.00 + 0.01
0.05 1 0.00
Table 8: Shape Shi et al. 2025) (t), for diffusion and LLM approaches compared in Section 3.1 Tabby Plain is best-performing method on 3 datasets similar to the performance of diffusion- techniques, but without the same limiting assumptions the nature of the dataset.
Diabetes
Travel
Adult
Magic
Shoppers
Abalone
Rainfall
House
TabSyn 0.79 = 0.14 0.82 0.13 0.81 0.15 0.99 = 0.00 0.98 0.00 0.84 = 0.12 0.98 # 0.00 0.99 0.00
TabDiff 0.96 0.00
1.00 0.99 1.000000000
0.00
0.98 1 0.00 0.98
0.99
Forest Diffusion 0.91 0.00
0.95
0.00 0.89 0.00
0.00
0.68 0.00
0.96 4 0.02 0.95 0.00
0.00
Tab-DDPM 0.89 0.00 0.99 0.00
0.00
0.63
0.99 +
0.98 0.00 0.00
Plain Base 0.96 0.96 0.00 0.99 0.90 0.00 0.00
0.00
0.00
0.00
0.95 + 0.03
Plain Tabby MH 0.98 = 0.000
0.01
0.85
0.98
0.93 0.93
0.00
0.97 = 0.00
GRea'T Base 0.81 1.01 0.000
0.00
0.85
1.00
0.95
0.001
N/A+
0.92 0.00
GReaT Tabby MH 0.85 = 0.00 0.93
0.00
0.01
0.80
0.91
0.08
0.23
= 0.40x 0.89 + 0.00
GTT Base 0.80 0.00 0.00
0.00
0.8.00022.885110000
0.85
1.00
0.00
0.000 0.50
0.43+
0.91 0.02
GTT Tabby MH 0.83 0.01 0.04
0.09
0.86
0.00
0.00
0.00
0.45
0.09
21

---

Published in Transactions on Machine Learning Research (01/2026)
Table Trend (Shi al. 2025) (t), for diffusion and LLM approaches compared in Section 3.1
Tabby Plain
is best-performing method 3 datasets, which is similar to other SOTA approaches.
Diabetes
Travel
Adult
Magic
Shopping
Abalone
Rainfall
House
TabSyn 0.84 0.12 0.68 0.22 0.61 0.33 0.99
0.00 0.99
0.69 0.25 0.95 0.00 0.99 0.00
TabDiff 0.97 0.00
0.92 0.001 0.97
0.97
0.04 0.99 0.58
Forest Diffusion 0.87 0.00
0.68
0.00 0.60
0.00
0.38
0.99 0.00
Tab-DDPM 0.88 0.00 0.97 0.000 0.00
0.97
0.58
0.95
0.00 0.99 0.00
Plain Base 0.95 0.95 0.04 0.06 0.04
0.68
0.96
0.01
0.89
0.96
Plain Tabby MH 0.97 0.00 0.98 0.00 0.02
0.86
0.00
0.87 0.94
0.99 0.00
GRea'T Base 0.86 + 0.01 0.84
0.77
0.91
0.00
0.00
0.00
N/A*
0.95 0.95
GReaT Tabby MH 0.85 + 0.02 0.89 +:0 0.77 0.06 0.90
0.90
0.92
0.47 0.06- 0.95 0.01
GTT Base 0.88 + 0.01 0.84 0.10
0.79 0.03
0.91 0.01
0.88 0.01
0.93
0.52 = 0.17+
0.95 t 0.02
GTT Tabby MH 0.86 1: 0.01 0.85 1 0.10 0.56 + 0.22 0.88 0.01
0.91 + 0.00 0.91 0.01 0.48 i 0.33
0.96 1 0.01
Table 10: Wasserstein distance (4), for diffusion and LLM approaches compared in Section 3.1 is the best-performing method on Diabeles, House and Shopping.
Plain Tabby
Diabetes
Travel
Adult
Magic
Shopping
Abalone
Rainfall
House
TabSyn 75.27
1.44+1.07
4.164=3224 00.51
278.13265.51
L69+1.81
25.05+8.58 132.33+66.19
TabDiff 26.83=2.01
0.30
1.154=46633 38.80+1.93
24..57=74.74
19.71 0.41
115.99+
Forest Diffusion 19.08=#048
0.35
9.9E3=3.6E3
38.20+0.72
274.67
0.43
28.69=5.6
110.40+20.50
Tab-DDPM 80.54-8.099
0.30
8.3E3=1.9E3
36.4511.00
3.1E411.7E3
0.37-0002
18.47
104.4733144
Plain Rase 21.03
0.20
1.3F4=766B3
52.71+2.33
352.944115.00
0.61
26.56
116.17
Plain Tabby MH 14.75:11.32
0.29:1000
1.2F4-2.93
36.64:1.63
180.40187.97
0.52:10.22
32.95
91.48 9.08
GReaT Base 82.751:3.86
0.50
26541185B3
37.8812.78
688.17159.05
0.33
N/A*
387.831
GReaT T Tabby MH 82.69.12.09
0.46110.04
3.2E411.744
55.8014.16
0.34 644.95116.52
76616E6-
304.11163.08
GTT Base 81.94
0.4810..03
2.23415.93
34.8710061
155.93
0.30
3E616E6,
309.51 78.00
GTT Tabby MH 78.70 0.3710.01
4.9411884 56.8916.76
562.43
0.45 3E616E6 375.05.120.07
Table 11: Membership Inference Attack (MTA) performance, for SOTA approaches compared in Section 3.1
Scores are aggregated across 50 attack techniques by reporting maximum AUROC (4), mean AUROC (1) and maximum accuracy (L). We find that Tabby performs similarly to most other methods.
Plain
Plain Tabby
GTT
TabDiff
CTGAN+
CLLM
HARMONIC
Base
Base
Max AUROC (L)
53.40
100.00
55.50
55.40
54.00
56.00
58.10
Mean AUROC (L)
51.03
52.66
52.43
52.45
51.64
53.42
51.45
Max Acc. (L)
52.60
100.00
54.20
54.20
52.90
54.80
62.50
Table 12: Runtime on the Abalone dataset, for SOTA approaches compared in Section 3.1 "Training" reports
the minutes to train one epoch, while "Synthosis" reports the minutes to produce 10 samples.
Plain Tabby
GReaT
GTT
HARMONIC
CTGAN
CGAN+
CLLM
TabDiff
TabSyn
Base
Base
Training
2:05
2:01
2:03
24:25
0:01
0:01
N/A
0:02
0:02
Synthesis
0:10
0:10
0:11
7:30
0:01
0:01
0:04
0:00
0:00
Table 13: Performance of the HARMONIC synthesis method
Wang et al.
2024 by various metrics.
Diabetes
Travel
Adult
Abalone
Rainfall
House
MLE
0.64
0.83
0.76
0.32
0.54
0.61
Discrimination
0.00
0.00
0.00
0.00
0.00
0.00
DCR
0.39
0.03
0.59
0.12
0.03
0.08
Shape
0.71
0.95
0.80
0.85
0.53
0.68
Trend
0.85
0.74
0.63
0.91
0.41
0.97
Wasserstein
49.81
0.99
61194.57
0.84
113.87
559.72
22

---

Published in Transactions on Machine Learning Research (01/2026)
Tabby Multi-MLP
Tabby Multi-Attention
Tabby Multi-Head
Original Base LLM
Transformer Block
Transformer Block
Transformer Block i
Transformer Block
Transformer Block 2
Norm
Norm
Norm
Transformer Block i
Transformer Block
Transformer Block
Attention
Attention
Att'n
Att'n
Att'n
Norm
Norm
Norm
MLP
MLP
MLP
MLP
MLP
Transformer Block B
Transformer Block
Transformer Block
Transformer Block
Head
Head
Head V
2
An overview of the Tabby MH modifications that can occur inside the LLM transformer blocks. Left to right: an original, non-Tabby LLM, a Tabby LLM with MoE MLP block, a Tabby LLM with MoE attention block, and a Tabby LLM with both MoE MLP and attention blocks. Tabby is very flexible, so to accommodate a wide variety of tabular datasets.
Figure 6:
E.2 Applying Tabby to Transformer MLPs Attention Blocks
We examine in-detail the performance of Tabby models with MoE applied to the transformer MLPs attention blocks. We use the following terminology to refer to these architectures, visualized in Figure i
Multi-MLP when each transformer's MLP block is replaced with an MoE layer,
Multi-MLP and Multi-Head (MMLP+MH) when each transformer's MLP block is replaced with an MoE layer and the LM head replaced with an MoE layer,
Multi-Attentinn (MA) when each transformer's attention block is replaced with MoF. layer.
We focus Tabby MH in Sections 3.1 3.4 because it demonstrates top performance in most settings. We display results for the MMLP and MMLP+MM M architectures across six datasets for MLE, discrimination and DCR in Tables 14 15 and 16 respectively. All three metrics are displayed for the MA architecture on two
datasets in Table 17
E.3 Additional Metrics for Experiment Applying Tabby MH to Models of Varying Sizes
The results in Section 3.2 compare the MLE scores of Plain-trained models of varying sizes on the Travel dataset. Table 18 incorporates the results for the Diabetes and House datasets as well. Similarly, Table 19 presents results for models trained using GReaT and Tabula (TapTap is not included here, because TapTap--retrrined checkpoints are available only for Distill-GPT2 and GPT2).
23

---

Published in Transactions on Machine Learning Research (01/2026)
Table 14: Machine Learning Efficacy (MT.E, t); defined in Section 3.0 for Base non-Tabby GPT2 models,
well as Tabby models with MoB layers applied to the transformer MLPs, language modeling head, or both (notated as MMLP, MH, and MMLP+MH respectively) on select datasets.
Diabetes
Travel
Adult
Abalne
Rainfall
House
Synthyythetic
0.73
0.87
0.85
0.45
0.54
0.61
Plain Non-TTbbb Tabby 0.75 = 0.02 0.86 + 0.01
0.85 0.00 0.44 =: 0.01 0.52 = 0.03 0.55 : 0.08
Plain Tabby MMLP 0.75 = 0.03 0.83 + 0.02
0.77
= 0.32 = 0.03
0.35 = 0.04
0.00 + 0.00
Plain Tabby MH 0.74 +: 0.00 0.88 0.01 0.85 0.00 0.43
0.01 0.49 + 0.00 0.60 + 0.00
Pluin Tabby MMLP 0.68 + 0.02 0.83
0.76
0.33 1 0.03
0.36 1 0.19 0.02 1 0.03
GReaT Non- Non-Tabby 0.62 1 0.01
0.83 1 0.01 0.41
0.01
N/A*
0.56 1 0.01
GReaTT Tabby MMLP 0.74 =: 0.01 0.85
0.84 - 0.01
0.38
- 0.01
0.24 -: 0.25 0.56 + 0.02
GReaT Tabby MH 0.64 + 0.01 0.00
0.83 = 0.00
0.40 + 0.01 0.00 + 0.00*
0.56 + 0.01
GReaT Tabby MMLP++MP 0.69 + 0.04 0.02 0.83
0.01
0.38
0.03
0.17
0.57 + 0.01
GTT Non-Tabby 0.72 + 0.06 0.87 0.02 0.83 0.01
0.40
= 0.01
0
0.55 +: 0.02
GTT Tabby MMLP 0.69 + 0.04 0.87 :: 0.01 0.84
0.00
0.36 + 0.01 0.03 0.00 0.56 +: 0.01
GTT Tabby MH 0.62 + 0.00 0.85
0.76 + 0.07
0.37
+ 0.02 0.26 + 0.37
0.55 + 0.00
GTT Tabby MMLP+MH 0.70 = 0.04
0.85 = 0.02
0.84 = 0.00
0.38
= 0.02
0.09
= 0.13
0.57 + 0.00
Table 15: Discrimination metric (4), defined in Appendix D. for Base non-Tabby GPT2 models, as well as Tabby models with MoF. layers applied to the transformer ML.Ps, language modeling head, or both (notated as MMLP, MH, and MMLP+MH respectively) on select datasets.
Diabetes
Travel
Adult
Abalone
Rainfall
House
Plain Non- TonbTabby 0.04 + 0.01 0.03 = 0.02 0.09 + 0.01 0.06 0.01 0.03 0.01 0.07 + 0.06
Plain Tabby MMLP 0.22 + 0.08 0.02 0.02 0.22 0.06 0.19
0.12
0.09
Plain Tabby MH 0.06 +: 0.02 0.02 = 0.01 0.10 0.01 0.06 0.00
0.08 0.00
0.03
Plain Tabby MMLP+ME MH 0.19 + 0.02 0.03 0.02 0.25 0.11
0.12
0.23
GReaTT Non-TTbby 0.28
0.06 0.01 0.00
N/A*
0.16
GRea'T Tabby MMLP 0.23
0.10
0.10:1..0000000000000
0.27
0.16
GReaT Tabby MH 0.29
0.08
0.00
GReaT Tabby MMLP+1M 4ML 0.24 1: 0.01
0.09
0.01 0.24 0.24
0.16 0.000
GTT Non- Con-Tabyy 0.27 0.02
0.07 -.07=0001 0.01 0.20 0.39 0.39
0.03
GTT Tabby MMLP 0.28 0.01
0.09
= 0.01 01810000000001000000000000011110
0.01
GTT Tabby MH 0.28 0.02
0.07
= 0.02 0.0
0.31
0.20
GTT Tabby MMLP MMH 0.21 + 0.01
0.08 + 0.01 0.18 +
0.00 0.21 + 0.24
0.16 0.01
Table 16: Distance to Closest Record (DCR, 420), defined in Appendix D for Base non-Tabby GPT2 models, as well as Tabby models with MoE layers applicd to the transformer MLPs, language modeling head, or both (notated as MMLP, MH, and MMLP+MH respectively) on select datasets.
Diabetes
Travel
Adult Abalone
Rainfall
House
Plain Tan-Tabby0.00 + 0.000.01 + 0.00 0.33 + 0.15 0.01 + 0.00 0.00 0.00 0.03 0.01
Plain Tabby MMLP 0.35 + 0.03 0.08 + 0.00 0.55 + 0.09 0.21 + 0.02 0.03 + 0.00
1.7c12 + 2.7e12
Plain Tabby MH 0.02 + 0.00 0.01 +: 0.00 0.25 0.03 0.01 1 0.010.01 + 0.00
0.01 + 0.00
Plain Tabby MMLP+MH 0.34 = 0.02 0.07 0.00 0.39
0.20 + 0.08 0.03 = 0.01 2.3e12 + 4.1e12
GReaT Non- Tan-Tabby 0.33 1 0.00 0.02 0.00 0.02 0.10
N/A*
0.06 1 0.00
GReaT Tabby MMLP 0.34 + 0.01 0.02
0.02
0.00 0.00 0.01
0.06 + 0.00
GReaT Tabby MH 0.36 + 0.00
0.10
0.01*
0.06 + 0.00
GReaT Tabby MMLP MH 0.33 = 0.02
00000000011100000000000000000000
0.06 - 0.00
GTT Non- Tabbyabby 0.31 + 0.01 0.0 0.10
0.02
0.06 + 0.00
GTT Tabby MMLP 0.31 = 0.02 0.02 0.14 0.10
0.01*
0.06
GTT Tabby MH 0.37 = 0.01
01010000000000000000000000000000
0.05 0.00
GTT Tabby MMLP 0.31 1 0.00 0.02 00000000001000000000000000000000
0.06 1 0.00
24

---

Published in Transactions on Machine Learning Research (01/2026)
Table 17: All evaluation metrics, for non-Tabby modcls and Tabby models with MoE applicd to the transformer allention blocks (abbreviated as Tabby MA) select datasets. Base LLM is DGPT2.
MLE (t)
Discrimination (L)
DCR (Usa)
Diabetes
House
Diabetes
House
Diabetes
House
Non-Synthetic (Upper Bound)
0.73
0.61
Plain Non-Tabbe Tabby
0.75
0.55
0.04
0.07
0.01
0.03
Plain Tabby MA
0.62
0.08
0.23
0.28
0.41
0.08
GTT Non- Tabby 0.72
0.55
0.27
0.18
0.31
0.06
GTT Tabby MA
0.62
0.56
0.31
0.17
0.36
0.06
Table 18: Results using Plain training for all three datasets of the experiment in Section 3.2 non-Tabby and Tabby MH models across base LLMs of varying sizes.
which compares
Travel
Diabetes
House
MLE (t)
Params
MLE (1)
Params
MLE (t)
Params
Non-Synthetic (Upper Bound)
0.87
0.73
0.61
Base Pythia 14m
0.86 + 0.01
14M
0.76 0.02
14M
0.52 + 0.07
14M
Tabby MH Pythia 14m
0.82 = 0.02
53M
0.77
66M
0.54 0.01
66M
Base Distilled-GPT2
0.88 = 0.00
82M
0.73
82M
0.53 0.53
82M
Tabby MH Distilled-GP11
0.89 0.02
310M
0.73
390M
0.01
390M
Base GPT2
0.89 = 0.01
120M
0.76
120M
0.60
120M
Tabby MH GPT2
0.87 = 0.01
360M
0.73 0.03
430M
0.53 + 0.11
430M
Base Pythia 160M
0.87 0.01
160M
0.75 0.04
160M
0.52 + 0.11
160M
Tabby MH Pythia 160M
0.86 + 0.00
390M
0.73 + 0.02
470M
+
470M
Base Pythia 110M
0.86 + 0.02
410M
0.74 0.03
410M
0.28 0.28
410M
Tabby MH Pythia 110M
0.88 + 0.03
710M
0.72 0.05
820M
0.54
820M
Base Llama 3.2 1B
0.82 1 0.01
1.2B
0.73
1.2B
0.29 0.001
1.2B
Tabby MH Llama 3.2 1B
0.84 1 0.02
2.8B
0.68
3.3B
0.18
3.3B
Base Llana 3.1 8B
0.84 + 0.01
8.0B
0.75
8.0B
0.35 0.001
8.0B
Tabby MH Llama 3.1
0.86 0.03
11B
0.72 + 0.01
12B
0.30
12B
Table 19: Results using GReaT and Tabula training for all three datasets of the experiment in Section 3.2
which compares non- non-Tabby and Tabby MH models across base LLMs of varying sizes.
Travel
Diabetes
House
MLE (t)
Params
MLE (t)
Params
MLE (t) Params
Non-Synthetic (Upper Bound)
0.87
0.73
0.61
Base Pythia 14m
0.81 0.00
14M
0.60
0.04
14M
0.46 + 0.06
14M
Tabby MH Pythia 14m
0.81 0.00
53M
0.67
0.01
66M
0.51 + 0.03
66M
Base Distilled-GPTy
0.86 + 0.00
82M
0.62 0.00
82M
0.57 1 0.00
82M
Tabby MH Distilled-GPT2
0.84
310M
0.70 0.06
390M
1 0.01
390M
Base GPT2
0.85 0.02
120M
0.64 0.02
120M
0.55 0.00
120M
Tabby MH GPT2
0.87 + 0.03
360M
0.74 0.03
430M
0.58 1 0.01
430M
Base Pythia 160M
0.81 0.01
160M
0.70 0.01
160M
0.00 0.00
160M
Tabby MH Pythia 160M
0.82 + 0.02
390M
0.73 0.03
470M
0.54 + 0.02
470M
Base Pythia 410M
0.85 0.01
410M
0.73 0.03
410M
0.53 0.53 0.02
410M
Tabby MH Pythia 410M
0.83 -
710M
0.74 0.04
820M
0.58 0.01
820M
Base Llama 3.2 1B
0.82 0.01
1.2B
0.70 0.08
1.2B
0.53 0.01
1.2B
Tabby MH Llama 3.2 1B
0.78 0.03
2.8B
0.71 0.03
3.3B
0.43 + 0.08
3.3B
Base Llama 3.1 8B
0.78 + 0.04
8.0B
0.67 0.01
8.0B
0.53 + 0.01
8.0B
Tabby MH Llama 3.1 8B
0.83 0.03
0.73 0.02
12B
0.45 + 0.00
12B
25

---

Published in Transactions on Machine Learning Research (01/2026)
MLE ( (1)
as Function of Model Parameter Count
0.92
Architecture
Machine Learning Efficacy (MLE)
Non-Synthetic
0.90
Pythia 14M
DGPT2
0.88
GPT2
Pythia 160M
Pythia 410M
0.86
Llama 1B
Llama 8B
0.84
Model Type
0.82
Tabby Base
0.80
107
10s
10
1010
10
Model Parameter Count
Figure 7: Machine Learning Efficacy (MLE) as a function of parameter count for 7 base LLMs, using Non-Tabe Tobyyabby or Tabby MH architectures. Non- Tabby points displayed in blue; MH points in purple. Red line represents Non-Synthetic, upper-bound performance.
Per-Column Validation Loss Over 10 Training Epochs
2.0
Average
Median Income
1.8
Median Age
Rooms
1.6
Bedrooms
Validation Loss
Occupancy
1.4
Median House Value
1.2
1.0
0.8
0.6
0
10000
20000
30000
40000
50000
Training Step
Figure 8: Per-column validation loss across 10 epochs of training Tabby MH Distilled-GPT2 on a subset of House, with average validation loss (black line). While the Occupancy column initially displays the highest loss, Median Income improves little throughout training and becomes the highest-loss column by step 32000.
E.4 Analysis from Tracking the Adaptation to Individual Columns
Individual column losses are shown in Figure S We observe that Occupancy is the largest contributor to the model's loss until step 32000. While Modian Income's loss is initially the sccond-lowest, it improves little throughout the training process and exhibits the highest loss of all columns the end of training. Additionally, convergence occurs across most columns around step 40000.
26

---

Published in Transactions on Machine Learning Research (01/2026)
Table 20: Five example rows of the synthetic training data used to investigate Claim 5 in Section E.5
Code
ID
upi_ _2i__4007272221 2001
427 027.424.6044 604-d
sku_ [au__355434_20
60. 20.272.46 433-i
sku 8712978211111
727. 884.985-0
upi_712558_ 2003
460.401,m 237-E
sku_ (al__91173_2201 2008
856.31 239-c
These insights are useful in cases where the model struggles learn some columns more than others. Such information may indicate need for better preprocessing for difficult column, or gathering more datapoints to remonstrate the column's distribution. Additionally, the ability to track each column's ' loss individually and to determine that the losses are roughly balanced across columns, rather than very low in some columns and very high in others, may improve trust in the model we can understand that there is a low, aleatoric error in each column as opposed to a sizeable epistemic error in a few columns.
E.5 Investigating Tabby's Generalization to Unseen and High-Cardinality Categorical Features
We now seek to demonstrate our fifth claim.
Claim 5: Tabby's s language modeling capabilities enable it to capture the underlying semantic structure of column values unseen during pretraining, allowing it to generate novel yet realistic values beyond the pretraining distribution.
Setup: We train a Llama 3-8B Tabby MH model for one epoch (LR = le - 4) on a 5000-row synthetic dataset, designed to imitate a common business scenario such as a product database. The dataset includes two columns: "Code" contains a 6-digit random number, prepended by "upi or "sku " and followed by a year (2000,2026). "ID" contains three numbers, where the first number is [1,999] and other numbers are [0,999], followed by hyphen and single lowercase or uppercase letter. We ensure that each trainset Code and each ID are unique. Some examples are in Table 20
Results: We find that the model produces examples of the same structure as the training data. In particular, out of 1000 generated samples:
1000/1000 rows meet all of the rules for the Code column value,
999/1000 rows meet all of the rules for the ID column value (one row omitted the final hyphen and letter),
Only one Code value in the trainset reappears in the synthetic set,
Only one synthetic Code value is reused across the synthetic set,
All synthetic IDs are unique and
No trainset IDs appear in the synthetic set.
Tabby's generalization capabilities are particularly valuable in real-world settings where datasets include non-categorical string features such as names, product IDs, addresses, telephone numbers. While these capabilities exlend Tabby's 's applicability across diverse data domains, the model's behavior in such cases depends on the implementation details of its underlying language model and tokenizer.
Discussion:
For example, when aut-of-vocabulle (OOV) tokens appear, their handling is determined by the specific tokenizer used. In the Distilled-GPT2 backbone employed in our main experiments, the tokenizer applies a byte-level variant of Byte Pair Encoding (BPE) (Radford et al. 2019), following the subword tokenization approach Sennrich et al. (2016). This ensures that every possible UTF-8 string can be represented without resorting to dedicated OOV tokens. Instead, previously unseen strings are decomposed into byte-level subtokens, which the model can interpret and adapt based on surrounding context. By contrast, other tokenizers may rely on explicit OOV symbols restricted vocabularies, potentially affecting Tabby's ability to generalize to novel strings and impacting overall performance in such settings.
27

---

Published in Transactions on Machine Learning Research (01/2026)
Table 21: Learning rates for LLM results presented in Section 3.1
Diabetes
Travel
Adult
Magic
Shoppers
Abalone
Rainfall
House
Plain Non- Jon-Tabby
le
le
le
1e-4 4
le
Plain Tably MMLP
le
le
le
le
le
le
le
Plain Tabby MH
le
le
le
le-
le 4
le 4
le
Plain Tabby MMLP MM
le
le 4
le 4
le 4
le 4
le 4
le 4
le 4
GReaT Nou- Tabby
le 4
le
le
le
le 4
le 4
le 4
le 4
Cltaa Tabby MMLP
le 4
le
le
le
le 4
le 4
le
lc 4
GReaT Tabby MH
le
le
le
le
le 4
le 4
le 4
le
GReaT Tabby MMLP+MM
le
le
le
le 4
le 4
le
le
GTT Non- Tonbyabby
le
4
le
GTT Tabby MMLP
le
GTT Tabby MH
le
le
GTT Tabby MMLP MH
16
1c
Table 22: Learning rates for Plain-trained LLMs of varying sizes in Section
3.2
Plain Training
Travel
Diabetes
House
Base Pythia 14M
le 4
lc 4
lc 4
Tabby MH Pythia 14M
le 6
le 4
le 4
Base Distilled- GPT2
le 4
4
4
Tabby MH Distilled- GP12
le 1e-4 4
le 4
le 4
Base GP12
le - 4
le 4
le 4
Tabby MH GPT2
le Re-4 4
le 4
le 4
Base Pythia 160M
le 4
le 4
le 6
Tabby MH Pythia 160M
4
4
Base Pythia 410M
le 6
le 4
le 6
Tabby MH Pythia 410M
le 6
le 6
1e
Base Llama 3.2 1B
le 6
4
1e
Tabby MH Llama 3.2 1B
le 6
1e 1
1e 6
Base Llama 3.1
1e 6
1c
1c
Tabby MH Llana 3.1 8B
1e 6
1c 6
lc 6
E.6 Hyperparameters for All Experiments
We list the learning rates chosen for Section 3.1 in Table 21 Section 3.2 in Table 22 and Section 24 in Table 24
We select the learning rate that yields lowest training loss from the set {le 3, le 4, le 6, 16,1e-00 8]. For
non- LLM methods in our experiments, we use the hyperparameters recommended by their respective papers.
28

---

Published in Transactions on Machine Learning Research (01/2026)
Table 23: Learning rates for GReaT (plus TapTap)- -trained LLMs of varying sizes in Section 3.2.
GReaT + TapTap Training
Travel
Diabetes
House
Base Pythia 14M
le 4
le 6
le
Tabby MH Pythia 14M
le 6
le 6
le 4
Base Distilled-GP1Y
le 4
le 4
le 4
Tabby MH Distilled-GPTT
le - 4
le - 4
le - 4
Base GPT2
le - 4
le - 4
le -
Tabby MH GPT2
le 4
le 4
le
Base Pythia 160M
1e 4
1e 6
1e 4
Tabby MH Pythia 160M
6
Base Pythia 410M
lE 6
lE 6
Tabby MH Pythia 410M
le
6
Base Llama 3.2 1B
le
4
Tabby MH Llama 3.2 1B
le - 4
1e 4
le 4
Base Llama 3.1 8B
le
le
Tabby MH Llama 3.1 8B
le - 4
le 4
le -
Table 24: Learning rates for JSON Glaucoma (Manoj 2024 experiment presented in Scction 3.4
Glaucoma
Base DGPT2
le 4 le 4
Tabby MH DGPT2
29


Reviews:
[{"review_id": "CvNQIhQZ2Z", "reviewer": "Reviewer_i7HK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Tabby, a post‑training architectural modification for transformer LLMs that replaces selected blocks with Mixture‑of‑Experts (MoE) so that each tabular column (or structured feature) is modeled by its own expert.  The best‑performing variant Tabby Multi‑Head swaps the LM head for a per‑column MoE head. The paper also introduces Plain, a simple finetuning recipe that serializes each row as text with an end-of-column token `<EOC>`, enabling per-column loss tracking and straightforward parsing at inference. Together, Tabby+Plain target higher-fidelity tabular (and nested JSON) synthesis using pretrained LLMs rather than training tabular models from scratch. The authors claim gains in machine learning efficacy (MLE) across six datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission's claims are convincingly supported by quantitative evidence: benchmark tables show Plain-trained Tabby-MH achieves the best or near-best MLE on most studied datasets, even reaching the non-synthetic upper bound on Diabetes, Travel, and Adult.  Aggregate performance-profile curves further give Tabby-MH the highest AUP across methods, indicating superior overall performance.  And on a nested-JSON glaucoma dataset, Tabby-MH matches real-data MLE while attaining the lowest discrimination score, outperforming baselines.", "audience_yn": "Yes", "audience_explanation": "This paper introduces Tabby, a per-column MoE add-on for pretrained LLMs, paired with a simple ``Plain'' row-as-text training scheme, and demonstrates clear, practical advances as follows:\n\n\n1. **Targeted architectural idea (per-column MoE) with simple and clear motivation.**\nThe per-column expert heads increase expressivity for heterogeneous feature types while still allowing cross-column dependencies, and the Multi-Head variant is especially effective. The change is conceptually simple and can be added to existing pretrained language models without needing to retrain them completely.\n\n2. **Plain training is practical and transparent.**\nRepresenting each row as a linearized text sequence with explicit `<EOC>` token keeps fine-tuning and inference simple.\nDespite its simplicity, **Plain** consistently matches or outperforms some more complex LLM-based tabular training methods in the experiments.\n\n3. **Extends beyond flat tables to structured (nested) data.**\nThe recursive application to JSON attains parity with real data and outperforms non‑LLM and diffusion baselines in that setting, supporting the generality claim.", "requested_changes": "## Weaknesses / Limitations\n\n1. **Coverage of strongest recent diffusion methods**\nThe comparison includes some diffusion-based baselines (Tab-DDPM, Forest Diffusion) but does not cover the most recent SoAT diffusion models (e.g., Tabsyn, TabDiff). Consequently, it remains difficult to pinpoint Tabby’s position relative to the newest diffusion methods. Notably, TabDiff leverages the **Diffusion LLM** concept for tabular data synthesis, which is conceptually related to this paper's approach.\n\n[Tabsyn] Zhang et al. -- Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space,  ICLR 2024\n\n[TabDiff] Shi et al. -- TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation, ICLR 2025\n\n2. **Extremely high-cardinality categorical fields.**\nThe approach relies on tokenizing raw column values as text, but columns with millions of distinct IDs (mostly unseen in pretraining) are not explicitly studied. It is unclear how Tabby behaves when most category values are effectively OOV strings.\n\n\n## Suggestions for Improvement\n\n1. **Include recent diffusion baselines** \nBuilding on the Weaknesses section, please add **TabSyn** and **TabDiff** as baselines to clarify how **Tabby** and **Plain** compare with recent tabular diffusion models, and evaluate Tabby/Plain on the datasets used in those papers (at least two beyond **Adult**, which you already cover). \n\n\n2. **Expand on handling high-cardinality categorical features**  \nDiscuss how Tabby (and Plain) behave with columns that have extremely high cardinality or free-form IDs, and whether additional preprocessing (e.g., hashing, learned embeddings) is needed at scale.", "broader_impact_concerns": "There is no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7x4dzx5Q4r", "reviewer": "Reviewer_1YRv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Tabby, a post-training architectural modification to transformer-based language models that enables higher-fidelity tabular and structured data synthesis through column-specific Mixture-of-Experts (MoE) layers. The work pairs this architecture with Plain, a simplified training technique that outperforms existing LLM-based approaches (GReaT, Tabula) while being computationally more efficient. The proposed approach represents the first application of MoE specifically to LLM-based table generation, and achieves strong empirical results on multiple tested datasets despite being more straightforward and parameter-efficient than baseline approaches. Beyond tabular data, authors also highlight strong performances on other structured data such as hierarchical JSONs, suggesting more general capabilities across structured data modalities.\n\nThe main strengths of this work are:\n\n- The architectural innovation of using MoEs for modeling columns in tabular data, stemming from the intuition that columns are distinct-but-interdependent features that benefit from specialized modeling, is sound and leads to considerable gains in downstream performances.\n\n- The evaluation is quite comprehensive, spanning 9 synthesis methods across 6 datasets.\n\n- The use of a per-column loss simplifies the analysis of models during training to identify challenging data that require further processing, as illustrated in the example of Appendix E.4. \n\n- The simple Plain approach also makes the method easier to adopt and less dependent on an adequate choice of additional procedures, e.g. tabular pretraining for TapTap.\n\n- Authors provide detailed hyperparameter evaluations in Table 17-20 to ensure the reported results are reproducible.\n\nThe weaknesses of this work are the following:\n\n- The inclusion of more modern approaches as baselines, e.g. HARMONIC [1] and CLLM [2] for LLM-based methods or CTAB-GAN+ [3] would have provided additional evidence that the proposed method obtains strong performances compared to recent work in this domain.\n\n- The problematic results of GReaT for the Rainfall dataset in Table 2 are not justified in light of practical improvement. It would have been helpful to clarify what could be the cause of such failures, and why the Tabby approach resolves them.\n\n[1] Wang et al. 2024. HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection. https://arxiv.org/abs/2408.02927v1\n[2] Seedat et al. 2024. Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes. https://arxiv.org/abs/2312.12112\n[3] Zhao et al. 2023. CTAB-GAN+: Enhancing Tabular Data Synthesis. https://arxiv.org/abs/2204.00401", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's core architectural claim, i.e. that MoE modifications improve LLM-based tabular synthesis, has some empirical support across various datasets and model scales. Table 2 shows that Plain Tabby MH achieves the best performance on 4 out of 6 datasets, although the Tab-DDPM method is within range for two of those datasets. The extensibility claim for JSON data (Table 4) is particularly interesting, with Tabby being the only method that reaches the non-synthetic upper-bound MLE (0.97).\n\nThe first claim, \"Plain-trained Tabby models generate higher-quality tabular data than prior approaches,\" relies on comparison with methods that are not very recent (e.g. GReaT from 2022). The inclusion of methods like the ones mentioned above might have provided more convincing evidence of the usefulness of this adaptation in the context of tabular data synthesis.\n\nThe second claim, \"Tabby allows smaller LLMs to achieve similar or better synthetic data fidelity than LLMs with higher parameter counts\", is confirmed by results in Table 3, but the small margins between non-Tabby and Tabby methods puts in question whether Tabby proves useful in this setting.\n\nThe two remaining claims are adequately supported by evidence provided by the JSON evaluation in Section 3.3, and the analysis of loss curves on the House dataset in Section 3.4.", "audience_yn": "Yes", "audience_explanation": "This paper addresses a critical gap at the intersection of foundation models and structured data, a topic of growing importance as the field recognizes the ubiquity of tabular data, despite it receiving far less attention than text/images. Several factors make this work highly relevant to TMLR's audience:\n\n- Van Breugel & van der Schaar's ICML 2024 position paper \"Why Tabular Foundation Models Should Be a Research Priority\" explicitly called for architectural innovations enabling cross-dataset generalization for tables. Tabby directly responds to this call with the first architecture modification (vs. pure fine-tuning approaches) designed explicitly for tabular generation.\n\n- Tabular data dominates real-world ML applications—healthcare records, financial transactions, scientific experiments, business analytics. Improved synthesis methods impact privacy-preserving data sharing (healthcare, finance), with the potential to enable data augmentation through the safe sharing of synthetic versions of proprietary datasets. The variety of datasets tested in this work and the extension to JSON establishes its generality beyond narrow application domains.\n\n- The first application of MoEs for tabular data opens potential avenues for further improvements in this area, e.g. an feature-type-specific experts (numerical vs. categorical) improve performance? From an interpretability perspective, expert specialization might improve transparency by enabling feature-importance analysis through expert weights.\n\nFinally, the effectiveness of the presented Plain method challenges conventional wisdom from previous approaches, such as Tabula and GReaT, requiring additional processing steps. The fact that small Tabby models exceed large baseline LLMs contradicts the field's general scaling trend, suggesting architectural specialization provides greater returns than brute-force parameter increases for structured domains.", "requested_changes": "- The inclusion of additional baseline comparisons with more recent methods in Section 3.1 would confirm the validity of the first claim made in this paper. In particular, beyond [1], [2] and [3], authors could consider also [4] and [5], which represent current diffusion SOTA for tabular data synthesis. If these methods cannot be run, provide detailed explanation of why (e.g., code unavailability, computational constraints) and acknowledge this limitation explicitly.\n\n- Regarding the privacy claims in Section E.1, while common Discrimination and DCR metrics were provided (Tables 6-7),  these were recently shown by [6] to be inadequate for identifying privacy leakages. In light of this, claims about privacy should be supported with a proper MIA evaluation.\n\n- Justify \"efficient\" and \"lightweight\" claims with quantitative evidence compared to baselines, not only in terms of parameter count but also training and inference time. Various other methods provide training time references (e.g. 9+ hours for GReaT, Table 6), and those would also be helpful here to conduct a fairer evaluation of different methods' efficiency.\n\n- Some additional clarity regarding the implementation would be useful. In particular, for Tabby MH: which specific layer(s) are replaced with MoE? (only final output layer, or multiple heads?). A mathematical characterization of the gating function beyond the \"Gated Mixture-of-Experts\" mention would also add clarity. Finally, regarding initialization: \"weights for each block in Λa are initialized to equal the weights of La\" (Section 2.3), does this mean all experts start identical then diverge during training?\n\n\n\n[4] Shi et al. 2024. TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation. https://arxiv.org/abs/2410.20626\n[5] Zhang et al. 2024. Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space. https://arxiv.org/abs/2310.09656\n[6] Yao et al. 2025. The DCR Delusion: Measuring the Privacy Risk of Synthetic Data. https://arxiv.org/abs/2505.01524", "broader_impact_concerns": "The paper includes a \"Broader Impact Statement\" (Section 5) acknowledging dual-use concerns. The current statement provide a sufficient, if slightly generic, overview of the current risks associated with synthetic tabular data generation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8qFdv7QOod", "reviewer": "Reviewer_XGrn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Tabby is a architectural modification to transformers to enable more effective modeling of tabular data by assigning a small set of parameters to modeling each field in the table or other structured data (i.e., by designating N heads in the final layer, each responsible for generating the content of one column). This, coupled with a technique for formatting tabular data for training called Plain, enables much better learning of tabular datasets and realistic synthetic data generation in-domain.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission's key claims are that the architectural modification Tabby and data processing modification Plain improve tabular modeling. The paper presents results for these both individually and together across several reasonable datasets, and using a number of metrics for tabular data learning.", "audience_yn": "Yes", "audience_explanation": "While I don't work in this area, so I cannot say for sure, the work motivates itself well in the domain of tabular data modeling and appears to engage deeply with the relevant literature, in addition to presenting improvements over existing methods. I also think the idea of a method that enables monitoring loss per-column is quite nice, and I could see this being useful.", "requested_changes": "Clarifications:\n* While I don't think it's incorrect to describe this as an MoE, the paper is not always clear on the detail of routing to individual heads (or attention/MLP blocks, for Appendix E). It's my understanding that the tokens corresponding to the $i$th column are always routed to the $i$th block-- is this correct? If so, this could be clarified in the text. The general reader's understanding of an MoE will be a model with a learned router, possibly one aiming to load balance across experts.\n* Could you clarify in the training section: you state that the loss for each column can be computed separately, which makes sense for the differentiated heads. Do you backprop between columns, or at the end of a batch of rows? I imagine it's the latter, but clarification would be useful. \n* Why does Tabby add so many parameters in Table 3? This makes me concerned that I've misunderstood something about the work. Why does adding Tabby for Travel triple the parameter count of GPT-2?\n\nExperimental requests:\n* I'm curious about adapting across table formats. The presumed advantage of the prior methods is that, once you have learned something about tabular data, it would be easier to train on a new tabular task. (1) Is this true? And (2) if so, can you provide results showing whether Tabby makes it more difficult to perform this transfer, since it requires architectural adaptation to each dataset? I think the paper should contain either this result or some discussion of tradeoffs here. \n* I'm not really convinced by the argument about parameter sharing in 2.4. _Does_ Tabby work well with parameter sharing, and how much does this limit the scaling up of cost with table size? I don't think the argument that this can scale to extremely large tables is necessary for the work, so I would also be okay with just modifying claims in this section to clarify that this is a direction for future work; however, as written, I think the claim needs experimental support.\n\nOptional, but might strengthen the work:\n* It's a bit hard to see from Figure 4 why Tabby is better than GTT; a different visualization might help.\n* It's very interesting to me that GPT-2 and related models outperform modern 1-8B LMs on this task. Some discussion of this or pointers to discussions of this in tabular modeling would be nice.", "broader_impact_concerns": "The broader impacts are sufficiently addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)