LATABLE: TOWARDS LARGE TABULAR MODELS

## Abstract

Abstract Tabular data is one of the most ubiquitous modalities, yet the literature on tabular generative foundation models is lagging far behind its text and vision counterparts. Creating such a model is hard, due to the heterogeneous feature spaces of different tabular datasets, tabular metadata (e.g. dataset description and feature headers), and tables lacking prior knowledge (e.g. feature order). In this work we propose LaTable: a novel tabular diffusion model that addresses these challenges and can be trained across different datasets. Through extensive experiments we find that LaTable outperforms baselines on in-distribution generation, and that finetuning LaTable can generate out-of-distribution datasets better with fewer samples. On the other hand, we explore the poor zero-shot performance of LaTable, and what it may teach us about building generative tabular foundation models with better zero- and few-shot generation capabilities.

## 1 Introduction

Motivation. Foundation models (FMs) in the image and text domains epitomize the value of large scale training, achieving impressive results and pushing the boundaries of AI capabilities. In this work, we propose a tabular diffusion model that can be trained across vastly different datasets, and could possibly be the basis for a tabular foundation model.

Tabular foundation models, or Large Tabular Models (LTMs), lag far behind their text and vision counterparts, and existing LTM research focuses on representation and supervised learning []. Yet, despite this *underrepresentation* in research, tabular data is *ubiquitous* in the real world—including in medicine [, ], finance [], census data [], data science competitions [, ], and natural science []. A generative LTM could transform these fields, through enabling few- and zero-shot generation of synthetic data (e.g. for privacy, debiasing, data democratization, augmentation, adaptation, simulation, etc. []), better imputation, and providing a base model that can be finetuned to other tasks (e.g. representation learning or prediction). Few-shot capabilities are especially interesting for the tabular domain, where datasets are often smaller. In Box A we expand on applications of a generative LTM.

Aim. Developing a generative LTM for tabular data is hard; different datasets cover vastly different features, there is no prior knowledge in the form of feature order, and data is messy—e.g. different datasets use different formats, contain different features, and can contain categorical, numerical, and missing values. As a result of this, training models meaningfully on large amounts of heterogeneous data remains underexplored.

We aim to take a step towards Large Tabular Models by developing a versatile tabular generative model with the following desiderata []:

- D1
Cross-dataset generation. We need a generative model to be able to generate different tables, which requires it being able to generate different features and variable number of features.
- D2
Categorical and numerical feature generation. Tables consist of combinations of both continuous and discrete data, and we want to be able to model each.(^1^11In contrast, an LLM-based approach like [] implicitly models each numerical feature as a series of discrete tokens, see Section 2.)
- D3
Use textual context. To achieve generating across tables, the model will need to understand contextual metadata—including the dataset description, feature names, and allowed categories (for categorical features).
- D4
Equivariance w.r.t. column order. Table column order is usually arbitrary, so we want an LTM to be equivariant to this. In other words, define generative model $G:\mathcal{S}^{L}\times\mathcal{M}\rightarrow\mathcal{X}^{L}$ with $\mathcal{S}$ the column-wise input space (e.g. feature names), $\mathcal{X}$ the output space, and $\mathcal{M}$ metadata (e.g. dataset description); for input $\mathbf{s}\in\mathcal{S}^{L}$, $r\in\mathcal{M}$, and $T$ some permutation, we desire $G(T(\mathbf{s}),r)=T(G(\mathbf{s},r))$.

Contributions.
Our contributions are as follows:

- 1.
We introduce LaTable, a novel tabular diffusion model that satisfies these desiderata and can thus be trained across vastly different tabular datasets (including different features, number of features, and different types) and with context of table metadata.
- 2.
We empirically show how LaTable outperforms existing generative models significantly on in-distribution generation, and when finetuned on new, out-of-distribution datasets.
- 3.
We explore the limitations of LaTable in terms of zero-shot capabilities. We argue that this stems from the limited pretraining of LaTable and discuss the challenges ahead for generative LTM research. Our findings points towards the central role that the training data should play in future research.

## 2 Related work

Tabular single-dataset generative models. Tabular data is challenging due to its lack of structural meaning (cf. images), mixed-type variables, and often limited size. A growing body of work is developing generative models tailored to tabular data [, , ], including diffusion/score-based models [, , ]. Nonetheless, most methods ignore or naively one-hot-encode categorical data (e.g. feature names and categories), which loses information and context that could help overcome data availability issues (D3).(^2^22Of course, without cross-table training, it would not be possible to learn much from the dataset description, feature names, and categories, due to lack of diversity—each dataset provides just one description, and not many unique features or categories to learn from.) Naturally, methods developed for single datasets are also not designed for cross-datasets generation (D1, or equivariance w.r.t. column order (D4).

LM-based approaches.
Pretrained language models (LMs) may provide a solution to cross-dataset generation with (D1-D4), as they contain general knowledge and can represent strings as numerical vectors in a space structured by language supervision.
These works all convert the rows of tables into strings that can be processed by the LM. The string generated by the LM is then converted back to a tabular format. Borisov et al. [] finetune GPT-2 [] on tables. They aim to achieve approximate equivariance (D4) through randomly permuting feature permutations, and sample tokens autoregressively from GPT’s predicted token probabilities to model distributions (cf. sample only the most likely sentences). Solatorio and Dupriez [] note that []’s approach retains GPT-2’s original vocabulary, yet that most tokens may not appear in the tables of interest. They instead use a fixed-set vocabulary[], which reduces the chances of generating invalid samples and thus improves efficiency. Last, Zhao et al. [] show that an untrained, smaller LM can generate more accurate data and do so more cheaply than []. We note that in contrast to us, previous works [, , ] train/finetune their model on just a single tabular dataset—i.e. they do not attempt cross-dataset training and do not aim to create a generative model that generalizes beyond the training data.

The advantage of LM-based generation is the simplicity and the fact that it does not require any manual preprocessing of data, since the raw data can be presented to the LM directly (e.g. prompting and in-context examples). However, adapting LMs as LTMs comes with serious disadvantages []. First, LMs are expensive, during both finetuning and inference. A 1-min training job for CTGAN, takes a whopping $540\times$ longer for [] (see their Appendix B.5, Table 6).
One of the core sources of this inefficiency is the linearization of table rows as sentences, and generating these autoregressively. For tabular data, where many columns are typically numerical, this is problematic—it means that a single numerical variable is implicitly modelled as an autoregressive series of categorical variables (e.g. $1.23$ is modelled as $1\rightarrow.\rightarrow 23$). Hence, generating this number with an LM requires 3 expensive forward calls to the model (plus one for the field separation token that follows). Fitting the whole data in context might also become difficult with datasets containing hundreds of columns. Furthermore, the multinomial training objective of LM is not apt at approximating continuous distributions—e.g. Renda et al. [] show how LMs do not accurately generate simple distributions (e.g. uniform) and van Breugel and van der Schaar [] visualize how modelling a simple Gaussian distribution with an LM is non-trivial. By opting for an end-to-end numerical representation of the data and a diffusion model framework, LaTable circumvents these limitations.

## 3 Method

Summary. We propose LaTable, which satisfies the desiderata. LaTable uses an encoder-only transformer as backbone for mixed-type diffusion. The input of the model consists of noised feature values (i.e. from the forward noising process), but also the dataset description, column names, conditioning mask (in case of conditional generation), and boolean missingness mask. All metadata strings (e.g. dataset description and feature names) and categorical features are first encoded using a pretrained LLM into a fixed-length embedding. All inputs are subsequently mapped to a common hidden space using separate element-wise MLPs, and added to provide the direct input of the transformer. The output of the transformer is decoded on a feature-by-feature basis, where categorical features are mapped to probabilities over the original features. See Figure 1.

### 3.1 Set-up and notation

We index datasets with $k$, samples with $i$, and features with $j$.(^3^33To avoid excessive clutter, we will leave out indices when they are irrelevant or clear from context.) Let us assume we have access to a metadataset $\mathcal{D}=\{D^{(k)}\}_{k=1}^{n_{D}}$, where each dataset $D^{(k)}$ consists of a description, column names, and the data itself. Let $d_{k}\in\mathbb{N}$ denote the number of columns for dataset $k$. Feature spaces will generally differ for each dataset, hence let us define the $j$th feature space for dataset $k$ as $\mathcal{X}^{(k)}_{j}$—i.e. for categorical data we let this be a finite set of strings, for numerical columns it may be some subset of $\mathbb{R}$. Let us denote the set of all strings with $\mathcal{S}$. We can thus write, $D^{(k)}=(r^{(k)},\mathbf{s}^{(k)},\mathbf{X}^{(k)})$, with $r^{(k)}\in\mathcal{S}$ the dataset description, $\mathbf{s}^{(k)}\in\mathcal{S}^{d_{k}}$ feature names, and samples $\mathbf{X}^{(k)}_{i}\in\prod_{j=1}^{d_{k}}\mathcal{X}^{(k)}_{j}$. Finally, let us denote the indices of the categorical features by $\mathcal{I}_{cat}^{(k)}$.

### 3.2 Generative model choice

We use a diffusion model [] (i.e., score-based model []) for modelling the data distribution. Diffusion models have increased in popularity thanks to their capability to produce samples of higher quality than GANs and VAEs. We use the discrete-time formulation with Denoising Diffusion Implicit Model (DDIM) noise scheduler [].

To satisfy the desiderata, however, we need to think carefully about what architecture backbone we use for the diffusion model, how we model both continuous and categorical variables, and how the tabular metadata fits in.
Let us look into each element of LaTable in more detail, and how it helps to satisfy desiderata D1-D4.

### 3.3 Satisfying desiderata

#### 3.3.1 Transformer backbone ( D1 , D4 )

We use an encoder-only transformer at the core of LaTable. The transformer backbone is useful for our setting, as it allows variable-length input and output—this allows us to train across datasets (D1). We *do not* use a positional encoding for the input, such that the transformer is equivariant w.r.t. the input (D4). In contrast, a recurrent architecture would not necessarily satisfy this equivariance property.(^4^44Importantly, removing the positional encoding means the transformer cannot know which column is which. We remove this undesirable symmetry by adding column information to each input element, in the form of a feature name embedding—see Section 3.3.2.) We denote the transformer’s hidden dimension by $d_{h}$.

#### 3.3.2 Using context ( D3 )

To make use of context (D3), we will encode the category names, column names, and dataset description using a pretrained LLM encoder. Note that the dataset description is the same across all features, but that the other metadata are defined on the feature level. By using a pretrained LLM encoder, we avoid having to learn embeddings for descriptions and categories from scratch. It also enables us to use textual similarities, even if our tabular data is limited. For example, one dataset may contain column “gender” and another “sex”, and we need not learn from the data alone that these are closely related.

We denote the pretrained LLM by $f_{\mathrm{LM}}$. Text encodings are fixed length $d_{f}$ for any input string $\mathcal{S}$, such that $f_{\mathrm{LM}}:\mathcal{S}\rightarrow\mathbb{R}^{d_{f}}$. Whenever we encode text data, we will denote this with a bar, e.g. $\bar{\mathbf{r}}^{(k)}$, as follows:

- 1.
Each string dataset description $r^{(k)}$ is encoded by the LLM into a vector $\bar{\mathbf{r}}^{(k)}$.
- 2.
Each categorical variable is converted into a string “[column name $s$] is [category $c$]". Categories $(c_{1},...,c_{n})$ are then encoded by the LLM and stacked into an *embedding matrix* $\bar{\mathbf{C}}^{(k)}_{j}$
$\bar{\mathbf{C}}^{(k)}_{j}=(\bar{\mathbf{c}}_{1},...,\bar{\mathbf{c}}_{n}).$
(1)
- 3.
Feature names $(s_{j})_{j}$ are encoded by the LLM into vectors $(\bar{\mathbf{s}}_{j})_{j}$

We condition on the encoded metadata by mapping the dataset description $\bar{\mathbf{r}}$ and feature embeddings $\bar{\mathbf{s}}_{j}$ to size $d_{h}$ using shallow MLPs $g_{r},g_{s}$ and adding it to the transformer input (see Figure 1 in blue).

For the encoder $f_{\mathrm{LM}}$, we use model WhereIsAI/UAE-Large-V1 for three reasons: it is open source and easily available,(^5^55[https://huggingface.co/WhereIsAI/UAE-Large-V1](https://huggingface.co/WhereIsAI/UAE-Large-V1)); it achieved SOTA on the Massive Text Embedding Benchmark (MTEB) [] until December 31 2023; and it is very lightweight (size: 1.34 GB). All encodings are cached to disk before training and testing to avoid LLM forward calls during training.

#### 3.3.3 Mixed-type variables ( D2 )

In contrast to standard LLM transformers and diffusion models, we need to generate numerical values as well as categoricals (D2). Let us start with the numerical variables.

Numerical variables. For numerical variables we can use relatively standard diffusion. Given some time step $t\in T$, the forward denoising process is applied to all numerical features $x_{j}$, giving the noisy $(x_{j})_{t}$. A shallow MLP $g^{i}_{n}:\mathbb{R}\rightarrow\mathbb{R}^{d_{h}}$ is used to map each numerical variable to the transformers input space. The transformer is subsequently applied to the sequence of all variables. Subsequently, each sequence element in the transformer’s output corresponding to a numerical variable is mapped back to scalars using output network $g^{o}_{n}:\mathbb{R}^{d_{h}}\rightarrow\mathbb{R}$. For each numerical $(x_{j})_{0}$, this yields output $(\hat{x}_{j})_{0}$, and the loss (e.g. sample-loss, $\epsilon$-loss, or $v$-prediction) is computed per feature.

Categorical variables. To handle categorical variables, we draw inspiration from Continuous Diffusion for Categorical Data []. The original CDCD learns high-dimensional embeddings for each category separately, before applying $L_{2}$ normalization to each embedding (after which noise is added for the denoising). During training, CDCD adds noise to these categorical embeddings (identical to continuous diffusion), and trains a model that predicts probabilities for the original categories. This model is trained using some classification loss, e.g. cross-entropy. During sampling, the outputted probabilities are used to predict the score in the continuous space. Specifically, they use *score interpolation*—which is effectively a weighted mean of all category embeddings, where weights are given by the probabilities predicted by the score model.

In our case, we do *not* want to learn embeddings and predict probabilities for all possible categories in all datasets. By learning an embedding separately for each category, this loses context that is captured in the category name (e.g. how categories female and woman are related). It also makes it hard to scale to very large number of total categories, especially if some of these categories are hardly ever observed. Instead, we use the pretrained LLM $f_{\mathrm{LM}}$ for acquiring ${d_{f}}$-dimensional embeddings for each category $c$, which *do* carry the contextual knowledge of the category name.(^6^66As discussed in Section 3.3.2, for all categorical features we embed the categorical $c$ as $\bar{\mathbf{c}}=$“[feature name] is [$c$]”.) We fine-tune these embeddings using a shallow MLP followed by an $L_{2}$ normalization layer (similar to CDCD), which we together denote by $g_{f}:\mathbb{R}^{d_{f}}\rightarrow S^{d_{f}}$.(^7^77Without the normalization layer, it would be difficult for the model to give embedded categories with small $L_{2}$-norm a high probability (see Eq. 2). We add the shallow MLP’s finetuning, to avoid projecting embeddings that are quite different in $\mathbb{R}^{d}$ to the same point on the unit sphere.) The forward diffusion process corrupts these vectors by adding Gaussian noise, giving $\bar{\mathbf{c}}_{t}$. Similar to the numericals, we use a network $g^{i}_{c}:\mathbb{R}^{d_{f}}\rightarrow\mathbb{R}^{d_{h}}$ to map these noisy embeddings to the transformer’s input space, and network $g^{o}_{c}:\mathbb{R}^{d_{h}}\rightarrow\mathbb{R}^{d_{f}}$ is applied to the categorical sequence elements in the transformer’s output—giving predicted embeddings in $\mathbb{R}^{d_{f}}$.

One key question remains: considering each categorical feature may have a different number of categories, how do we map the transformer output (in $\mathbb{R}^{d_{f}}$) back to the individual categories? We achieve this using an attention-like layer. Letting $\hat{\mathbf{c}}_{j}\in\mathbb{R}^{d_{f}}$ be the output of the model, we let the final predicted probability for some category $i$ be proportional to the similarity of the generated vector and the original categories’ embeddings (see Eq. 1):

$$ $p(c_{i})=\text{softmax}(g_{f}(\bar{\mathbf{C}})^{T}\hat{\mathbf{c}})_{i}, \forall i,$ (2) $$

thereby allowing us to predict probabilities for each datasets’ and categoricals’ categories individually.
Identical to CDCD, we use cross-entropy loss $\mathrm{CE}$ for the categorical variables and use score interpolation. Combining the numerical and categorical losses, we get:

$$ $\mathcal{L}(x,\hat{x})=\frac{1}{d_{k}}\left[\sum_{j\in\mathcal{I}_{cat}^{(k)}} \mathrm{CE}(x_{j},\hat{x}_{j})+\sum_{j\not\in\mathcal{I}_{cat}^{(k)}}\mathrm{ MSE}(x_{j},\hat{x}_{j})\right].$ (3) $$

Bringing it together. Overall, the model thus consists of a number of independent MLPs $g_{\cdot}$, that allow us to bring differently sized variables (dataset description, feature column name embeddings, categorical features, numerical features) all to the same space $d_{h}$. The input of the transformer consists of a sequence of inputs $h_{j}\in\mathbb{R}^{d_{h}}$, $j=1,...,d_{k}$ , with $h_{j}=g^{i}_{c}((\bar{x}_{j})_{t})+g_{r}(\bar{r})+g_{s}(\bar{s}_{j})+g_{t}(t)$ for categoricals, and $h_{j}=g^{i}_{n}(x_{j})_{t})+g_{r}(\bar{r})+g_{s}(\bar{s}_{j})+g_{t}(t)$ for numericals. The transformers output sequence is again decoded using independent shallow networks $g_{c}^{o},g_{n}^{o}$, to provide score estimates for both numerical and categorical variables. See Figure 1 for a full overview.

Figure: Figure 1: The diffusion pipeline, for both training and generation. Note that the whole pipeline is flexible with respect to the number of numerical and categorical features as input. The LLM is encoder is frozen and the transformer is an encoder-only model without positional encodings or causal masking. Additional conditioning (e.g. missingness mask, conditioning information) can be trivially added to the transformer input, or through cross-attention layers.
Refer to caption: extracted/5651947/figures/architecture.png

## 4 Experiments

In this section we explore the tabular data generation capabilities of LaTable. We show how LaTable achieves significantly better in-distribution generation, as well as few-shot out-of-distribution generation through minor finetuning. These observations suggest that LaTable’s data generation capabilities generalize across tables. At the frontier of these capabilities, we discuss zero-shot tabular data generation as a limitation of LaTable. We explore how this relates to the limited diversity and quality of the training data, drawing lessons for future research into generative LTMs.

Data. We combine curated metadatasets [] and [] from OpenML, which respectively cover classification and regression tasks. After removing image datasets, these total 83 sets (678k samples, 3021 unique columns)—see Figure 4 for number of samples and features per dataset. We split up these datasets into in-distribution $\mathcal{D}_{\mathrm{id}}$ and out-of-distribution $\mathcal{D}_{\mathrm{ood}}$, including respectively 78 and 5 datasets. In contrast to other large metadatasets (e.g. TabLib []) that may contain (processed) duplicate sets, the datasets we use are curated, hence there is no data leakage. All experimental details can be found in Appendix B.

### 4.1 In-distribution generation

We use independent train-val-test splits of each set in $\mathcal{D}_{\mathrm{id}}$, and train LaTable on all training sets in $\mathcal{D}_{\mathrm{id}}$. We compare the generation quality of LaTable to popular synthetic data baselines. We choose tabular generators in different classes, including TVAE, CTGAN [], TabDDPM (diffusion) [], and ARF (tree-based) []. Note that for each of these models, we train a new model on each individual dataset and for each seed. We compare downstream model train-on-synthetic-test-on-real AUC performance (of an XGBoost classifier and taking the label of the original dataset); synthetic data metrics precision and recall [, ]; and density and coverage []. Note that precision and density aim to quantify realism or fidelity of samples, whereas recall and coverage measure diversity.

As we observe in Table 1, LaTable achieves significantly better results than baselines on most metrics. The closest competitor is the tree-based ARF, which scores comparably on downstream AUC and recall but significantly poorer on the other metrics. Metric Recall gives a high score when data samples are very diverse (i.e. the synthetic distribution has a large support), but this does necessarily reflect well on data quality (see Fig. 1b in []). This might explain why recall is high for all methods, even though they fail on other metrics.

**Table 1: Average performance of each generative model across 78 OpenML datasets in $\mathcal{D}_{\mathrm{id}}$. We report the mean and standard deviation over 5 random seeds. Downstream AUC is computed for an XGBoost model trained on synthetic, tested on real data (only including classification datasets). LaTable is significantly better on all metrics except recall.**
| *Generator* | *Down. AUC* [$\uparrow$] | *Density* [$\uparrow$] | *Coverage* [$\uparrow$] | *Precision* [$\uparrow$] | *Recall* [$\uparrow$] |
| --- | --- | --- | --- | --- | --- |
| Real | 0.895 (0.007) | 0.961 (0.005) | 0.955 (0.002) | 0.953 (0.003) | 0.945 (0.001) |
| ARF | 0.853 (0.009) | 0.700 (0.008) | 0.832 (0.003) | 0.836 (0.002) | 0.942 (0.001) |
| CTGAN | 0.842 (0.008) | 0.584 (0.006) | 0.741 (0.003) | 0.720 (0.002) | 0.912 (0.004) |
| TVAE | 0.828 (0.007) | 0.739 (0.010) | 0.805 (0.004) | 0.812 (0.007) | 0.877 (0.002) |
| TabDDPM | 0.757 (0.014) | 0.414 (0.015) | 0.535 (0.022) | 0.469 (0.021) | 0.823 (0.011) |
| LaTable | 0.874 (0.006) | 0.865 (0.009) | 0.900 (0.001) | 0.866 (0.004) | 0.900 (0.002) |

It is insightful to more closely analyze the origin of LaTable’s gains against the baselines. In Figure 2 we plot the metrics for each individual dataset and as a function of the number of samples in each dataset. We observe that LaTable performs more reliably for datasets with smaller sample size, and on average better across the board. This can be understood as a consequence of LaTable being trained on other datasets besides the small datasets. Hence, this suggests that LaTable outperforms the baselines by transferring data generation capabilities across tables. We note that despite hyperparameter tuning attempts, TabDDPM performed poorly for small datasets.

Figure: (a) Density
Refer to caption: extracted/5651947/figures/plot_iid_n_samples_stats.prdc.density.png

### 4.2 Zero- and few-shot out-of-domain generation

An important characteristic of foundation models is their ability to be adapted to new datasets, preferably using relatively little data (i.e. few-shot). In this section we thus aim to use the trained model $G$ on some target datasets $D_{T}\in\mathcal{D}_{\mathrm{ood}}$ that it has not seen during training, and see how it compares to baselines.

For LaTable, we would ideally want good zero-shot performance—i.e. to generate new data based on the new data’s feature names and metadata alone. However, we will show that this constitutes the limits of LaTable’s capabilities and pretraining. A more realistic scenario is to adapt LaTable to the new datasets through finetuning. To measure the effect of how much data is needed for finetuning, we artificially reduce the target dataset to a smaller number of samples. We compare to baselines that are trained on the target dataset alone.

In Figure 3 we plot the performance of LaTable and baselines as a function of the number of target samples, which is averaged over 5 seeds and 5 different datasets in $\mathcal{D}_{\mathrm{ood}}$. Baseline *Real* is again the real training data, but in contrast to last section, this dataset is now very small.

As aforementioned, the zero-shot performances of LaTable does not match the baselines—it has not seen enough data during training to generalize well. When finetuned, however, LaTable exhibits a more promising behavior. On *Density* (i.e. fidelity) it achieves an almost perfect score of 1, on par with real data. Among the baselines, TVAE stands out. However, its density larger than 1 suggests that the model collapsed to a high-density region of the distribution. When we look at *Coverage* (i.e. diversity), LaTable appears even stronger. We hypothesize that due to the pretraining and its metadata, it is able to generate more diverse realistic samples in a few-shot setting. This hypothesis is reinforced by noting that LaTable achieves significantly better diversity than the real data—this is possible, considering the real training data includes just a few samples.

Figure: (a) Density
Refer to caption: extracted/5651947/figures/plot_ood_stats.prdc.density.png

## 5 Challenges to generalization

Let us explore possible directions for future research into better few- and zero-shot LaTable.

Why LaTable achieves poor zero-shot performance.
We cannot expect good zero-shot performance of LaTable, as $\mathcal{D}_{\mathrm{ood}}$ is not covered by $\mathcal{D}_{\mathrm{{id}}}$. In Appendix C.1 we analyse the features of $\mathcal{D}_{\mathrm{id}}$ and $\mathcal{D}_{\mathrm{ood}}$. We observe that most features in $\mathcal{D}_{\mathrm{ood}}$ do not have a close feature in $\mathcal{D}_{\mathrm{id}}$. Similarly, even for the features that are observed during training, there may be a significant shift in the feature distribution—let alone, how it interacts (e.g. correlates) with other features.

Improving out-of-distribution performance through scaling.
The straightforward solution to this problem is to increase the training dataset size—this has been one of the key drivers behind text and vision FM progress [, , , ]. Multiple large tabular metadatasets exist: WebTables [] consists of 10M crawled HTML tables, WikiTables [] includes 580K Wikipedia tables, and the recent Tablib [] includes 627M crawled tables with 827B context tokens.

Though these metadatasets are large, unfortunately they primarily include a significantly different type of table. In Appendix C.2 we show they consist of smaller tables, both in terms of features and samples (see Figure 7). We also show this affects LaTable’s generation. WikiTables is curated, covers a wide range of topics with context from the Wikipedia article, and is guaranteed to not have overlap with $\mathcal{D}_{\mathrm{ood}}$ (vs Tablib or WebTables, which may bias our results), hence let us consider this for training LaTable. We discard any table with less than 20 samples, which yields 100k+ tables, and repeat the experiment from Section 4.2 for a model trained on WikiTables. Though we observe (Figure 8) comparable performance to the original LaTable when finetuned, the zero-shot performance is significantly worse.

Going forward. Our findings emphasize the importance of good quality data in improving generalization performance of LaTable. The issue of data quality may be resolved by better understanding of large metadatasets like TabLib, and using heavy-handed curation and selection. To avoid data leakage between the training and evaluation sets, this also requires robust tools for identifying and removing duplicates, which is hard for tabular datasets since preprocessed and filtered copies are common. Another possible avenue for future work is to use more specialized databases for training, e.g. of high-throughput biological data [].

## 6 Discussion

Limitations.
LaTable can be extended and improved in multiple ways:

- 1.
Scale and data. As discussed in the previous section, LaTable is a relatively small model trained on a relatively small dataset. Scaling up is one of the key challenges for future research.
- 2.
Extending variable and table types. We restrict ourselves to numerical and categorical data, but future work could consider datetime, full string descriptions, time-series, and relational databases to increase applicability and impact.
- 3.
Bias. In this work, we did not explore possible bias in the data and pretrained LLM, or how LaTable copies it. More research into bias is required before applying LaTable to the real-world, see broader impact below.

Broader impact. We believe the broader impact of LaTable, and continued work in this area, is primarily positive. LaTable can enable better synthetic data using fewer samples, which could promote more responsible AI—e.g. to improve minority representation using data augmentation [, , ], ML model testing through data simulation [], and data democratization through private synthetic data []. The latter is especially interesting, since few-shot generation may require a smaller privacy budget than standard synthetic data—after all the model need not learn the full distribution from the private data and may be less likely to memorize this data [].

Nonetheless, we need to acknowledge the risk: LaTable may make errors and may reflect or exacerbate societal biases that are present in the data, or in the pretrained LLM embeddings. More research into possible biases is required. The quality and fairness of generated data should always be evaluated before applying LaTable to real-world sensitive settings like healthcare and finance.

## Appendix A Broader impact of generative LTMs

The potential impact of a foundational tabular models is large. Generative models require a lot of data to be trained, inhibiting their use in smaller domains. On the other hand, if we would able to learn from multiple datasets, combine knowledge, and translate knowledge from one dataset’s domain to another, this could enable generating data and making predictions with less data. We highlight some applications in Box A.

## Appendix B Implementation details

Encoding example. As mentioned in 3.3.2, we use text encoder WhereIsAI/UAE-Large-V1 for encoding all string values. Let us consider an example:

**Table 2: This describes the dataset.**
| Age | Gender | Income |
| --- | --- | --- |
| 42 | Male | 30,000 |
| 33 | Female | 46,000 |

In this case, we encode and cache “This describes the dataset” ( $\bar{\mathbf{r}}$), feature names “Age”, “Gender”, and “Income” ($\bar{\mathbf{s}}_{j}$), and “Gender is Male” and “Gender is Female”.

### B.1 Data

Data is loaded from OpenML dataset benchmarks [, ]. Datasets with more than 500 columns are discarded as these correspond mostly to image datasets, which results in 83 datasets. We take 78 datasets for training LaTable ($\mathcal{D}_{\mathrm{id}}$) and 5 datasets for testing out-of-distribution performance ($\mathcal{D}_{\mathrm{ood}}$)—the latter includes OpenML datasets with dataset IDs 31 (credit-g), 37 (diabetes), 44 (spambase), 1063 (kc2), and 1570 (adult). Additional train-val-test splits are created for each dataset individually, and all training sets in $\mathcal{D}_{\mathrm{id}}$ are combined for training LaTable. The number of samples and features per dataset is visualized in Figure 4. Numerical features are normalized.

Figure: Figure 4: Number of samples per dataset plotted against number of features, with in-distribution datasets $\mathcal{D}_{\mathrm{id}}$ in blue and $\mathcal{D}_{\mathrm{ood}}$ in orange. Note: double log-scale.
Refer to caption: extracted/5651947/figures/n_features_n_samples.png

### B.2 Model and training

For LaTable, we use the architecture described in Section 3 and visualized in Figure 1. All MLPs $g_{\cdot}$ consist of 3 layers, and the transformer consists of 10 layers. All hidden layers are of dimension $1024$. We train and finetune for 500 epochs using Adam, with a cosine scheduler and learning rate $5\cdot 10^{-5}$. These values were chosen based on the in-distribution validation loss and validation generation performance, and are kept the same across all experiments. During training we weight points by their respective dataset size to ensure small datasets are not ignored, with dataset $k$’s weight $w^{(k)}\propto\frac{1}{\sqrt{|D^{(k)}|}}$. We use $T=200$ inference steps for DDIM, and use the standard Diffusers implementation of DDIM. Note that zero-shot generation, we *do* assume we know the mean and standard deviations of numerical variables (as we retain scaling) and we *do* assume which categories are allowed to be generated for each categorical feature (i.e. $\mathbf{\hat{C}}$ is provided).

### B.3 Baselines and metrics.

We use the Synthcity library [] for baselines and metrics. Baselines use standard recommended settings for all datasets. For TabDDPM, we use 1000 denoising steps. We have tried changing the MLP size, but this did not yield reliably better results. To compute the metrics, we generate $\min\{|D_{\mathrm{test}}|,5000\}$ samples. For the “Real” baseline in Figure 3, we oversample the small real datasets to acquire the required number of points for computing the metrics.

### B.4 Compute

The model as described above can be trained on a 16GB GPU with batch size 32. Training LaTable on all datasets in $\mathcal{D}_{\mathrm{id}}$ takes around 2 days on a single GPU. Finetuning takes about 3 seconds per sample (e.g. 5 minutes for 100 samples). Generation speed is around 30 samples per second. For model development, we trained 33 models with different hyperparameters on all of $\mathcal{D}_{id}$.

## Appendix C Improving out-of-distribution generation (accompanying Section 5 )

### C.1 Comparing in- and out-of-distribution datasets

Though the number of features and samples in datasets is comparable across $\mathcal{D}_{\mathrm{ood}}$ and $\mathcal{D}_{\mathrm{id}}$ (see Figure 4), the small size of $\mathcal{D}_{\mathrm{id}}$ means it does not cover $\mathcal{D}_{\mathrm{ood}}$’s features. We plot the cosine similarities between all feature embeddings of $\mathcal{D}_{\mathrm{ood}}$ and $\mathcal{D}_{\mathrm{id}}$ in Figure 5. Though there is some semantic overlap in features, about three quarters of the features in $\mathcal{D}_{\mathrm{ood}}$ have no similar feature in $\mathcal{D}_{\mathrm{id}}$.

Even when similar feature names appear in both datasets, trivially these may still pertain to vastly different features. Many datasets contain a column “class”, yet these have vary different meaning and categories. Similarly, marginals (and joint) distribution can be significantly shifted—see Fig 6

Figure: Figure 5: Features in $\mathcal{D}_{\mathrm{id}}$ do not cover all features in $\mathcal{D}_{\mathrm{ood}}$. Cosine similarity matrix between feature embeddings in $\mathcal{D}_{\mathrm{ood}}$ and $\mathcal{D}_{\mathrm{id}}$. For 0.752 of the ood-features, all id-features are dissimilar ( similarity$<0.8$). Note: the three little diagonal lines refer to dataset “Kc2 Software Defects” (OpenML ID 1063) and 3 similar datasets that contain the same features.
Refer to caption: extracted/5651947/figures/cosine_similarity_matrix.png

Figure: (a) Education
Refer to caption: extracted/5651947/figures/education_hist.png

### C.2 Using other metadatasets

In Figure 7 we compare the number of features and samples in OpenML, WikiTables, and Tablib. We observe that the latter two have significantly different characteristics. This is to be expected—many online datasets consist of informational tables (with few samples and features). TabLib consists of a large number of datasets with many features. This is likely due to a good amount of their data being sourced from Github, which may contain wide SQL and other database structures, which do not resemble typical tabular ML tasks.

In Figure 8 we show that indeed, training on WikiTables does not yield better out-of-distribution performance. Training on WikiTables for 100 epochs took 24 days on a single GPU.

Figure: Figure 7: Large online metadatasets have significantly different characteristics than ML tabular datasets. Plotting different sources’ number of samples and features per table. Note: double log-scale.
Refer to caption: extracted/5651947/figures/n_features_n_samples_per_source_kde.png

Figure: (a) Density
Refer to caption: extracted/5651947/figures/plot_compare_others_ood_stats.prdc.density.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 1
presentation: 1
questions: Can the authors clarify what WhereIsAI/UAE-Large-V1 refers to in L171?
soundness: 1
strengths: - The authors provide a clear motivation for the need for tabular generative models and present a model designed to meet the specific requirements of tabular data generation. The related works section is well-written, highlighting why LLM-based approaches are not optimal compared to their diffusion-based approach.
- The authors conducted comprehensive experiments, examining critical factors beyond general model performance, such as scalability, cross-dataset training procedures, and the impact of including textual context.
summary: The paper introduces LaTable, a diffusion-based generative model for table generation. LaTable is designed to support flexible generation with varying numbers and types of features, utilizes additional context like dataset descriptions and feature information, and demonstrates equivariance with respect to column order. The authors conducted experiments showing that model performance scales with both model and table sizes, and that LaTable outperforms several baselines. Results also indicate that training across multiple datasets, incorporating textual context, and exposing the model to more data samples (e.g., table rows) or increasing the model size enhance its performance.
weaknesses: - The evaluation setup is unclear. The authors mention Cardio, URL, WiDS, Insurance, and Heloc as test datasets, citing Stoian et al. However, only URL, WiDS, and Heloc are covered in that paper; details on Cardio and Insurance datasets are not disclosed, and relevant citations for these datasets are missing. Additionally, the authors do not provide clear references to the baselines (e.g., it is unclear which papers CTGAN, TVAE, ARF, DDPM, and GREAT correspond to in L343).
- The authors state that all methods are fine-tuned on the test dataset (L348) but then fit a CatBoost model on the generated data from models fine-tuned on the test set (L346) to predict table elements from the test set itself (L347). Is this interpretation correct? If so, the metrics’ significance is unclear. For instance, a model that generates only the exact data it was fine-tuned on would make the generated synthetic training set overlap with the test set, which would improve CatBoost performance, but the metrics would not indicate the model’s ability to generate novel synthetic data instead of memorized training data.
- The paper's presentation is poor and requires improvement. References are incorrectly formatted (e.g., missing parentheses), figures exceed boundary limits (e.g., Figure 2), and the phrase “scaling law” is mentioned 14 times without citing a single scaling law paper.
- No results are provided for multiclass classification or regression tasks. Although some discussion is included in L467, quantitative comparisons on other tasks between the authors’ method and other baselines are necessary to showcase the general capability and utility of LaTable.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Will you consider including additional metrics beyond downstream performance, such as low-order and high-order statistics [1]?
2. Will you consider adding more recent baselines, such as TABSYN [1] and TabDDPM [2]?
3. Figure 2 illustrates the effects of training set size and model parameters but does not address the impact of the number of categories and numerical features. Will you consider investigating this aspect?
4. Do you plan to release the codes?

[1] Zhang, Hengrui, et al. "Mixed-type tabular data synthesis with score-based diffusion in latent space." arXiv preprint arXiv:2310.09656 (2023).
[2] Kotelnikov, Akim, et al. "Tabddpm: Modelling tabular data with diffusion models." International Conference on Machine Learning. PMLR, 2023.
soundness: 2
strengths: The paper clearly outlines four primary design goals—cross-dataset generation, handling of categorical and numerical features, use of textual context, and column order equivariance—and effectively aligns these with specific model design choices. Additionally, it identifies scaling laws unique to tabular data, which is valuable given that this area has not been thoroughly explored within the scope of tabular foundation models.
summary: The work introduces a generative model LaTable designed specifically for tabular data generation. Recognizing the ubiquity and challenges of tabular data, the authors focus on managing heterogeneous tabular data features, such as categorical and numerical variables, across various datasets. LaTable leverages metadata to improve the generation process and is built upon an encoder-only transformer. The model is evaluated through few-shot learning scenarios to validate its cross-dataset generalization capabilities. Additionally, LaTable shows early signs of scaling laws, commonly observed in foundation models in other domains.
weaknesses: 1. LaTable shows limited robustness on non-binary classification tasks, such as multi-class classification and regression, suggesting constrained generalization across different task types.
2. The descriptions of datasets and baseline models are brief and lack detail.
3. The evaluation metrics are limited, primarily focusing on downstream performance.
4. Figure 2 is oversized.
5. Although the paper acknowledges issues of data bias and fairness, it does not explore practical approaches to detecting or mitigating these biases in real-world applications.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: None
soundness: 2
strengths: - The paper addresses the underexplored domain of large-scale tabular data modeling, a departure from traditional focus areas in foundation models such as text and vision.
- The model meets several carefully formulated desiderata: cross-dataset generation, mixed-type handling, use of textual metadata, and equivariance to column order. The authors thoroughly answer each desiderata in their model design.
- LaTable represents an important step toward creating generative models that can be applied to diverse tabular datasets.
summary: The paper introduces "LaTable," a novel generative model for tabular data based on diffusion techniques, specifically aimed at large-scale tabular data generation. LaTable displays early signs of "scaling laws" observed in foundation models. The authors empirically demonstrate that their model outperforms existing generative models in ood settings.
weaknesses: I mostly found it hard to understand the architecture of the model and the training objectives you used on it. I am from outside the tabular data community so this may be the reason, but I think that it should be clear to people outside the community as well. It is clear that you very carefully designed the architecture to meet all your requirements but I wasn't sure in the end what is the input/output, how you train everything end-to-end. A more higher level description is required instead of diving in directly to satisfying desiderata.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 1
questions: Can you explain the generation procedure? Where is the noise inserted?

Can you differentiate LaTable from other large tabular models, e.g. TabPFN? How does it compare?
soundness: 2
strengths: Strengths:

* The present work is one of the first works that trains a generative transformer model across several dataset. The collection of datasets used to train the model presented is bigger than of any other works I am aware of, thereby making a reasonable step towards scaling up tabular models.

* Evaluation through ablation studies : I like the evaluation done through several ablation studies, showing the usefulness of the individual components. The results support the respective design choices.

* The Limitation and Discussion Section is honest and insightful.
summary: This work proposes LaTable, a Diffusion model for tabular data generation that can be pretrained on large collections of tabular datasets and used for unconditional and conditional generation on unseen datasets in a few-shot manner. The model is evaluated in a few-shot generation setup where it performs comparably against other techniques.
weaknesses: Weaknesses
* The write-up is quite hard to understand and the notation seems overwhelming at some points. I know the basics of denoising diffusion models, but I was not quite able to follow in what space the diffusion happens and how exactly it is mapped back and forth into the tabular representation. Figure 1 doesn’t help much to gain a better understanding. In particular, I would have appreciated a section which would detail the generation procedure. I do not understand where in Figure 1 noise is input to generate synthetic samples and in diffusion models there are different generation paradigms as well, such as latent diffusion etc. I do not think the write-up is very accessible in its current form.
* Related Work. This is not the first attempt to build a tabular foundation models trained on multiple datasets. Notable approaches include TabPFN (Hollmann et al., 2023). I also wonder how LaTable compares to other approaches (although mainly focused on classification), such as Yak et al. (2023) or Zhu et al. (2023). It is unfortunate, that these competing approaches are neither discussed nor compared in the evaluation. 
* Conditional Generation / Classification is not evaluated. The authors describe how conditional generation can be implemented with LaTable, but do not test is as far as I see. This also trivally allows to use LaTable for the classification task (by conditioning on all tabular features and letting the model generate the label). Here, it would be insightful to compare LaTable’s performance to models such as TabPFN, XTab, or GREAT. Also in addition, zero-shot and fine-tuning with the entire dataset should be considered for a comprehensive evaluation.
* The evaluation only uses ML Efficiency (Train Synth., Test Real, TSTR). For a comprehensive picture, the performance of a model trained on the real data should be included as an upper baseline to assess the data quality gap in the TSTR table. In addition, there could be further data quality metrics, including metrics such as the Discriminator metric (e.g. used in Borisov et al., 2023) where a model is trained to differentiate between original and synthetic data and its performance is reported. Also some quantitative results could complement the evaluation.

The overall impression of the submission suggests that not much care was taken to prepare the current manuscript for submission and several important things have been neglected:
* The citation format is incorrect, citep is not used properly
* The Appendix and Supplementary materials are missing
* There are several formatting issues, e.g., Figure 2
* Text in formulas should be wrapped in \text, e.g. \text{softmax} (eqn. 1, 2)
* There is no code available.

While these points may be fixed, I think for submission at a respected venue such as ICLR more care should be taken.

**Summary.** Overall, the submission seems to be rushed and I think a thorough revision is needed. This should include a more accessible write-up, studying classification performance and additional data quality metrics, and comparing with other attempts for building large tabular models.


Typos: 
* Please check if “e.g.” should be followed by a comma.
* l. 237: transformer‘s
* l. 207 mathbb{R}  instead of normal R
* Caption of Figure 2: Capitalization of LaTable 

----------------

**References**

Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., and Shoaran, M. XTab: cross-table pretraining for tabular transformers. In Proceedings of the 40th International Conference on Machine Learning (pp. 43181-43204), 2023

Yak, Scott, Yihe Dong, Javier Gonzalvo, and Sercan Arik. "IngesTables: Scalable and Efficient Training of LLM-Enabled Tabular Foundation Models." In NeurIPS 2023 Second Table Representation Learning Workshop. 2023.

Hollmann, Noah, Samuel Müller, Katharina Eggensperger, and Frank Hutter. "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second." In The Eleventh International Conference on Learning Representations, ICLR 2023.

Vadim Borisov, Kathrin Sesler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. ICLR 2023.

Meta-review: This paper introduces LaTable, a new diffusion-based generative model designed for tabular data that can be trained across multiple datasets. The authors present LaTable as one of the first attempts at building a tabular foundation model, addressing key challenges like handling heterogeneous feature spaces, incorporating metadata, and maintaining column order equivariance. The work demonstrates early signs of scaling laws similar to those observed in other foundation model domains, and shows superior performance compared to baselines in few-shot out-of-distribution data generation scenarios.

The reviewers unanimously provided negative scores and raised several significant concerns about the the paper’s presentation and scope. The reviewers seemed to be generally confused about the model architecture and training objectives. Besides, reviewers pointed out limited evaluation and lack of comparison with other recent tabular foundation models (though authors note these focus on supervised rather than generative learning). While reviewers acknowledge the paper's novel contributions toward scalable tabular generation, they recommend rejection due to the need for substantial improvements in presentation quality, broader experimental evaluation beyond binary tasks, and more comprehensive comparison with related work. The authors' rebuttal clarifies the few-shot evaluation setup and reasoning behind metric choices, but reviewers maintain their assessment that the current manuscript requires significant revision to meet publication standards.

Decision: Reject