Diffusion Transformers for Tabular Data Time Series Generation
Authors: Fabrizio Garuti
Enver Sangineto
Simone Luetto
Lorenzo Forni
Rita Cucchiara
Venue: ICLR 2025 Poster (2025)
Keywords: tabular data generation
time series
diffusion models
Primary area: generative models

Abstract: Tabular data generation has recently attracted a growing interest due to its different application scenarios. However, 
generating time series of tabular data, where each element of the series depends on the others,
remains a largely unexplored domain. 
This gap is probably due to the difficulty of jointly solving different problems, the main of which are the heterogeneity of tabular data (a problem common to non-time-dependent approaches) and the variable length of a time series.
In this paper, we propose a Diffusion Transformers (DiTs) based approach for tabular data series generation. Inspired by the recent success of DiTs in image and video generation, we extend this framework to deal with heterogeneous data and variable-length sequences. 
Using extensive experiments on six datasets, we show that the proposed approach  outperforms previous work by a large margin.

Published as a conference paper at ICLR 2025
DIFFUSION TRANSFORMERS FOR TABULAR DATA TIME SERIES GENERATION
Fabrizio Garuti",2,, Enver Sangineto Simone Luctto Lorenzo Forni' & Rita Cucchiara² Prometeia Associazione, Bologna, Italy, 2 University of Modena and Reggio Emilia, Italy, 3 Prometeia SpA, Bologna, Italy. i fabrii cabrrziio.o aur www.ggartutttpeeeegeiee.o com, enver. "enver..angineeee wag.eessssnnneettmmmmmeeeee...
ABSTRACT
Tabular data generation has recently attracted a growing interest due to its different application scenarios. However, generating time series of tabular data, where each element of the series depends on the others, remains a largely unexplored domain. This gap is probably due the difficulty of jointly solving different problems, the main of which are the heterogeneity of tabular data (a problem common to non- time-dependent approaches) and the variable length of a time series. this paper, we propose a Diffusion Transformers (DiTs) based approach for tabular data series generation. Inspired by the recent success of DiTs in image and video generation, we extend this framework to deal with heterogeneous data and variable- se- quences. Using extensive experiments on six datasets, we show that the proposed approach outperforms previous work by a large margin.
INTRODUCTION
Time series of tabular data are time-dependent sequences of tabular rows, where each row is usually composed of a set of both numerical and categorical fields. Tabular data time series are widespread in many real-life applications, and they can represent, e.g., the temporal sequence of the financial transactions of a given bank user or the clinical data a patient during her hospitalization. Gen- erating time series of tabular data is particularly important due the limited availability of public datasets in this domain. For instance, despite private banks usually own huge datasets of financial transactions of their clients, they rarely make these data public. A generator trained on real dataset can synthesize new data without violating privacy and legal constraints. Besides privacy preservation (Jordon et al. 2018; Abdelhameed al. 2018 Assefa et al.) 2020 Efimov al. 2020 Hermandez et al. 2022, Qian et al. (003)) other common applications of (non necessarily time dependent) tab-
ular data generation include: imputing missing values (Zheng & Charoenphakdee, 2022] Gulati & Roysdon 2023 Borisov et al., 2023; /Zhang et al. 2024). data augmentation and class balancing of
real training datasets (Che et al. [2017] Choi et al. 2017 Xu et al. 2019 Kim et al. 20222] Rizzato
ct al. 2023; Fonseca & Bacao (023). and promoting fairness (van Breugel et al.) 2021). However, despite the impressive success of generative AI methods for images, videos or text, generating tab- ular data is still challenging task due different reasons. The first problem is the lack of huge unsupervised or weakly supervised datasets for training. For instance, Stable Diffusion (Rombach) al. 2022) was trained on LAION, weakly supervised dataset of 400 million image-text pairs crawled from the web, while tabular data generation models need to be trained on datasets which are several orders of magnitude smaller. The second reason which makes tabular data generation difficult is the heterogeneity of their input space. In fact, while, for instance, images are composed of numerical pixel values, with high correlation between adjacent pixels, and textual sentences are sequences of categorical words, tabular data usually contain both numerical and categorical fields (e.g., the transaction amount and the transaction type). Thus, a generative model for tabular data must jointly synthesize both numerical and categorical values, and its training must handle this in- homogeneity (Sec. 2 In case of time series of tabular data, which is the topic addressed in this paper, the additional temporal dimension further increases this difficulty introducing the necessity to model the statistical dependencies among the tabular rows the series is composed of, as well as the need to deal with variahle-length input.

---

Published as a conference paper at ICLR 2025
among the very few generative deep learning methods for time series of tabular data with heterogeneous field values, and they are both based Transformer (Vaswani et al. 2017) trained autoregressively to predict the next input token. Specifically, both methods convert the numerical fields into categorical fea- tures, creating field-specific token vocabularies. A time series is then represented sequence of concatenated tokens, which are predicted by the network autoregressively. Despite this strategy is effective, the main drawback is the limited diversity of the generated sequences. In fact, an uncon- ditional generation of a time series starts with a specific "start-of- "start-of-sequence" token and proceeds by selecting the next token using the posterior on the vocabulary computed by the network. However, even when non-deterministic sampling strategy is used (e.g., randomly drawing from the posterior on the next token), an autoregressive (AR) network tends to generate similar patterns when it is not conditioned on a specific user query. For instance, there is no unconditional sampling mechanism in TabGPT, since it can only predict the future evolution of an initial real sequence provided as query.
TabGPT (Padhi et al., 2021) and REaLTabFormer (Solatorio & Dupriez 2023)
In this paper, we follow a different direction and we use a Diffusion Model (DM) based paradigm. which is known to be particularly effective in covering multi-modal data distributions (Ho et al. 2020 DMs have recently been applied to the generation of single-row tabular data (Kim et al. 2022 Kotelnikov et al., 2023; Lee et al., 2023) Kim et al. 2023) Zhang et al.) (024), empirically showing their superiority both in terms of diversity and realism with respect other generation paradigms /Zhng et al., 2024). However, none of the existing DM based single-row generation approaches
can directly be used to synthesize tabular data time series. Indeed, these methods assume relatively
short, fixed-length input sequences and use denoising networks based on MultiLayer Perceptrons
(MLPs). To solve these problems, we propose to use a Diffusion Transformer (DiT) (Peebles & Xie, 2023 which enjoys the benefits of both worlds: it is a Transformer, which can naturally deal with sequences, and is a DM, which can generate data with a high diversity and realism.
We call our method TabDiT (Tabular Diffusion Transformer) and we follow the Latent Diffusion Model (L.MM) (Rombach et al. 2022 Peebles Xiel (023) framework (Sec. 2), where DMs are trained on the latent space of a pre-trained and frozen variational autoencoder (VAE (Kingma & Welling, 2014)). Very recently, TabSyn (Zhang et al. 2024) exploits a similar LDM paradigm for single-row tabular data generation (Sec. 2). In TabSyn, both categorical and numerical field values are represented as token embeddings, which are fed to a Transformer based VAE. The latent space of this VAE is used the input space of an MLP-based denoising network. However, TabSyn can- not be used for tabular data time series generation because both its VAE and its denoising network cannot deal with long, variable-length sequences. To solve these issues, we propose decompose the representation problem by simplifying the VAE latent space and using the denoising network to combine independent latent representations. Specifically, since time series training datasets are usually too small to train a VAE to learn complex dynamics with variable length, we split each time series in set of independent tabular rows which are separately compressed by our VAE. In this way, the variational training is simplified and the time series chunking acts as data augmentation. Thus, differently from previous LDM work, our VAE latent space does not holistically represent a domain sample (i.e., a time series), but only its components. Modeling the combination of a sequence of embedding vectors in this latent space is delegated to the Transformer denoising network, which is responsible of learning the time-dependent distribution of the time series. Moreover, differently from DM-based single tabular row generation methods, we propose an AR VAE decoder, in which the generation of the next field value depends the previously generated values of the same row, which improves the overall consistency when compared with a standard parallel VAE decoder. Fur- thermore, since the representation of heterogeneous tabular fields still largely unsolved problem, we propose a variable-range decimal representation of the numerical field values, based on a se- quence of digits preceded by a magnitude order prefix. This representation is a trade-off between a lossless coding and the length of the sequence (Sec. 4). Finally. to generate variable-length time series, we follow FiT (Lu et al. 2024), a very recent LDM which deals with vuriable-resolution im- age generation by padding the sequence of embedding vectors fed to the DIT. However, at inference time, the image resolution sampled in FiT independently of the noise vectors fed to the DIT, which is reasonable choice since the resolution of an image is largely independent of its content. Con- versely, a time series of, c.g., financial transactions conditioned on the attributes of a specific bank client, can be longer shorter depending on the client and how they use their bank account. For this reason, we use the denoising network to also predict the generated sequence length, jointly with its content, and we do so by forcing our DiT explicitly generate the padding vectors defining the end of the time series.
2

---

Published as a conference paper at ICLR 2025
Due to the lack of a unified evaluation protocol for tabular data time series generation, we collect different public datasets and we propose an evaluation metric which extends single-row metrics to the time-series domain. For the specific case of unconditional generation, TabDiT is, to the best of our knowledge, the first deep learning method for unconditional generation of heterogeneous tabular data time series. In this case, we compare TabDiT with strong AR baseline, which implemented by merging the (discriminative) hierarchical architecture proposed in (Padhi et al. 2021 Luetto et al. 2023), with some of the architectural solutions we propose in this paper. In all the experiments, TabDiT significantly outperforms all the compared baselines, usually by a large margin. In summary, our contributions are the following.
We propose a DiT-like approach for tabular data time series generation, in which Trans- former denoising network combines latent embeddings of a non-holistic VAE.
We propose an AR VAE decoder to model intra-row statistical dependencies and a variable- range decimal representation of the numerical field values.
We propose to generate variable-length time series by explicit padding prediction.
We propose metric for the evaluation of tabular data time series generation, and we empir- ically validate TabDiT using different public datasets, showing the superiority of TabDiT with respect to the state of the art.
TabDiT can be used in both a conditional and an unconditional scenario, and it the first deep learning method for unconditional heterogeneous tabular data time series generation.
2 RELATED WORK
Diffusion Models. DMs have been popularized by Ho al. 2020); Dhariwal & Nichol (2021) who showed that they can beat GANs (Goodfellow et al. 2014) in generating images with a higher realism and diversity, reducing the modc-collapse problems of GANs. Stable Diffusion (Rombach
et al. 2022) first introduced the LDM paradigm (Sec. D. where training is split in two phases. In the first stage, VAE is used to compress the input image. In the second stage, DM is trained on the VAE latent space. DiT (Peebles & Xie 2023) adopts this paradigm and shows that the U-Net architecture (Ronneberger et al. 2015) used in Stable Diffusion is not crucial and it can be replaced by Transformer. Since a U-Net is based on image- seege--peccfic inductive biases (implemented with convolutional layers), we adopt DiT as our basic framework. Very recently, FiT (Lu et al.) (024) extended DIT to deal with variable-resolution images, by padding the embedding sequence up to a maximum length. We adopt a similar solution to generate variable-length time series. However, in FiT the padding vectors are masked in the attention layers and ignored both at training and at inference time, and the image resolution is sampled independently of the noise vectors. In contrast, we force our DiT to explicitly generate padding vectors, in this way directly deciding about the length of the specific sequence it is generating.
Single-row tabular data generation. Early methods for generating single-row tabular data include CTGAN (Xu et al.) (201)) and TableGAN (Park et al.] 2018), based on GANs, and TVAE (Xu et al.) 2019) based VAES. VAFS have also been employed more recently in other works, such as, for instance, GOGGLE (Liu et al., 2023), where they are used jointly with Graph Neural Networks. TabMT (Gulati & Roysdon| 2023) adopts Transformer with bidirectional attention model sta- tistical dependencies among different fields of a row. In GReaT (Borisov et al. 2022). a tabular row represented using natural language and Large Language Model (LLM) is used to conditionally generate new rows. However, the authors do not provide any mechanism for fully-anconditional sampling. Moreover, the LLM maximum input length limits the dataset size which can be used (Zhang et al. 2024). Finally, tabular field names and values are frequently based on jargon terms and dataset-specific abbreviations, which are usually -iittr-distiitttion for an L.M (Narayan al., 2022 Luctto et al. 2023).
Most of the recent literature on single-row tabular data generation has adopted a DM paradigm Kimm et al. 2022 Kotelnikov et al., 2023 Lee et al. 2023 Kim et al. 2023 Zhang et al. 2024).
and different works mainly differ by the way in which they deal with numerical and categorical
features. For instance, both TabDDPM (Kotelnikov al. 2023) and CoDi (Lee al. 2023 use standard Gaussian diffusion process (Ho et al. (020) for the numerical features and a multinomial diffusion process (Hoogeboom al. 2021 for the categorical ones. Specifically, CoDi uses
3

---

Published as a conference paper at ICLR 2025
categorical and numerical specific denoising network which are conditioned from each other. In STaSy (Kim et al.) 2023). both numerical and categorical features are treated as numerical and a self-paced learning strategy (Kumar et al. 2010) is proposed for training. Different from previous work, we propose a variable-range decimal representation that converts numerical features into a sequence of categorical values and allows both categorical and numerical features be uniformly represented as a sequence of tokens. TabSyn (Zhang et al., 2024) is the work which is the closest to our paper because it is also based on an LDM paradigm (Sec. D. However, TabSyn cannot be used for time series of tabular data because both its VAE and its ALse- denoising network cannot deal with variable length sequences. For this reason, in this paper we propose non-holistic VAE, which compresses only a chunk of the input sample (i.e., an individual row rather than an entire time series). this way, most of the representation complexity is placed on our DiT-based denoising network, which is responsible of generating temporally coherent sequences by combining VAE compressed embedding vectors. Finally, differently from TabSyn and other DM-based tabular
data generation methods (Kim et al.] 2022; Kotelnikov et al. 2023 Lee et al. 2023 Kim et al. 2023). we use an AR VAE decoder and we explicitly predict the time series length.
Tabular data sequence generation. SDV (Patki et al., (016)) is non-deep-leaming method based on Gaussian Copulas to model inter-field dependencies in tabular data. TabGPT (Padhi et al.) (202)) (Sec. D is a forecasting model, which can complete a real time series but it lacks sampling mech- anism for generation from scratch. Moreover, it is necessary to train a specific model for each bank client. In contrast, REaLTabFormer (Solatorio & Dupriez 2023) has a similar AR Transformer ar- chitecture but it focuses on conditional generation, in which a time series is generated depending, e.g., on the attributes of a specific bank client, described by a "parent table" (Sec. 3 In REaLTab- Former, numerical values are represented as fixed sequence of digits. We also represent numerical values as sequence of digits, but our representation depends on the magnitude order of the specific value, which results in shorter sequences and reduced decoding error (Sec. 4.1). Differently from (Padhi et al. 202T, Solatorio & Dupriez| 2023), our proposal is based on DM, and can be used for both conditional and unconditional generation tasks.
3 PRELIMINARIES
2022) is based on two separated Diffusion Transformer. The LDM paradigm (Rombach et al. training stages: the first using a VAE and the second using a Gaussian DM. The main goal of the VAE is to compress the initial input space. Specifically, in DiT (Peebles & Xie 2023), an image is compressed into a smaller spatial representation using the VAE encoder 2 = L(x). The VAE latent representation is then "patchified" into a sequence s = [21. 1...... =x.. Note that grouping "pixels" of 2 into patches is a procedure external to the VAE, which holistically compresses the entire image. denoising
Then, random noise is added to and its corrupted version is fed a Transformer-bant network (DiT) which is trained to reverse the diffusion process. More specifically, given a sample 8o extracted from the real data distribution (defined on the VAE latent space, 8y ~ g(so)), and a prefixed noise schedule (ā1, .... āT), the DM iteratively adds Gaussian noise for I diffusion steps: ((: = ( / faysp. (1 a)) t L..., T. Using the reparametrizztioon trick, s, can be obtained by: 8; v /0 + VI OCEE where e I) is noise vector. A denoising network is trained to invert this process by learning the reverse process: (0:- (s)) = -N (o (st, t), og t)).
Training is based on minimizing the variational lower bound (VLB) (Kingma & Welling 2014).
which can be simplified to the following loss function (Ho et al. 2020
((0) Simple
[lle (8,, 1)
where (+)()) is reparametrized into a noise prediction network (-). Following (Nichol& & Dhariwal 2021], DiT predicts also the noise variance (oo(-)), which is used to compute the KI.-divergence of the VLB in closed form. However, for simplicity, we do not predict the noise variance.
At inference time, a latent 27 is sampled and fed to the DiT network, which follows the reverse- process sampling chain for T' steps until 2o is generated. Finally, the VAE decoder ((-) decodes 20 into a synthetic image. For condional generation, DiT encodes the conditional information (e.g.. the desired class label of the image) using a small MLP. The latter regresses, for each DiT block, the parameters of the adaptive layer norm (B,y) and the scaling parameters (a) used in the residual connections. In (Peebles & Xie 2023), this is called adaLN- Zero block conditioning and it is used also to represent the timestep 1.
4

---

Published as a conference paper at ICLR 2025
Problem formulation. Tabular data are characterized by set of field names (or attributes) A {ai...., ax). where each aj E A is either a categorical or a numerical attribute. A tabular row r = [ei...., 2/| is sequence of le field values, one per attribute. If aj is numerical attribute, then My E IR, otherwise Vij E V,, where V, is an atribbte-specificc vocabulary. A time series is a variable- length, time-dependent sequence of rows x [ri...,, r.]. Given a training set X (21;--,,N empirically representing the real data distribution q(x), in unconditional tabular data time series generation the goal is train generator which can synthesize time series following g(x)). Moreover, inspired by (Solatorio & Duprizz 2023), for the conditional generation case, we assume to have "parent" table P associated with the elements in X. tabular row u [w..... w/. in P is associated with sequence I; E X and it describes some characteristics that affect the nature of x;- For instance, if z; E X is the history of the transactions of the i-th bank client, the corresponding row in P could describe the attributes (e.g., the age, the gender, etc.) of this client. In conditional generation task, given u, the goal is to generate a synthetic time series according to g(z)u).
4 METHOD
In this section present our approach, starting from unconditional generation scenario, which we will later extend to the conditional case. Our first goal is to simplify the VAE latent space: since the time series have variable length and complex dynamics, rather than representing their dis- tribution using a variational approach (Kingma & Welling, 2014), we use our VAE to separately compress only individual tabular rows, and then we combine multiple, independent latent represen- tations of rows using the DiT-based DM. In more detail, our VAE encoder & represents a tabular row r with latent vector z & (r). z E Z, which is decoded using the VAE decoder Dep The parameters + = x sp] of & and D. are trained with a convex combination of reconstruction loss and a KL-divergence (Kingma & Welling, 2014). The training set is = ...,,". MMM. where each rj E R is row extracted from one of the time series E X. Note that M >> N and that the rows in are supposed to be i.i.d., i.e., we treat the samples in R as independent from each other.
Hence, the semantic space of our VAE (Z Rd) will not embed any dependencdd among the rows of the same time series, since Eg and De cannot observe this relation.
Fo 11
8. P.
DET books
MLP
Ademion
Decide 14
60 Densising network
Tee + T
(b) VAE
[Par.]
Encader
4
O
Enioke
Figure 1: A schematic illustration of the denoising (a) and the VAE (b) network of TabDiT.
Fig. shows the proposed framework, which includes Eg. Dip and our DiT-based denoising network
Fo. At training time, the latter takes as input sequence of latent row representations of a specific time series. In more detail, given I; E X, x; = [r,,..., rr]. we compute 80 = 1 2,]
(r)())) .... (T)r and we use Su as explained in Sec. 3 to train Fe. where Fø t) implements tt. The timestep t is first encoded using small MLP and then its embedding vector is used to condition Fo (see later). Note that sy is time dependent sequence of row embeddings, thus the real data distribution (So)) we use to train Fg does include the statistical dependencies among tabular rows of a same time series. In other words, while 2 represents only individual rows, we use Fo to combine independent vectors lying in this space into sequence of time-dependent final-embedding vectors representing an entire time series.
Conditional generation. In conditional generation task, we want to condition Fo using row u of a parent table P (Sec. B). To do so, we first encode u using a specific encoder Cz. The architecture and the field value representations of Cu are the same Cp is disjoint from the VAE and it is trained jointly with Fe- The output vector c = Cp (u) is summed with the embedding vector of the timestep and fed to the MI. of the adal N--ero block
Es (Sec. 4.1p However,
5

---

of Published as a conference paper at ICLR 2025
conditioning mechanism (Sec. B). Moreover, following most of the image generation DM literature,
in the conditional generation scenario we also use Classifier-Fres Guidance (CFG). Specifically, the denoising network output is interpreted as the score function (Song et al. 2021 then the DM conditional sampling procedure can be formulated as (Peebles & Xie
2023):
Fo,, u, Fo,, + 8. (Fo(s,, Fo(0,,,,0) 1,)),
where (o,,,,, 1, is the unconditional prediction of the network and > 1 indicates the scale of the guidance (s corresponds to no CFG). At training time, for each sample F;, with probability Pa [0, 1] the condition is dropped and the network is trained unconditionally.
4.1 ENCODING AND DECODING IN THE VAE LATENT SPACE
Field value representations. For categorical attributes aj, use widely adopted tokenization approach (Zhang et al., 2024; Luetto et al. 2023; Padhi et al. 2021). in which each possible value Uy Vj is associated with token and lookup table of token embeddings transforms these tokens into the initial embedding vectors of &g. However, how to represent numerical values (0g E IR) in way which is coherent with categorical feature tokens is still open problem (Sec. ( and each method adopts specific solution (see Ap.. For instance, REaLTabFormer (Solatorio & Dupriez, 2023) converts vj into a sequence of digits and then treats each digit as a categorical feature. If the maximum possible value in X for the attribute aj is Umaz, and assuming, for simplicity, that a, can only take on positive integer values, then U, converted into a sequence each D& E ('O'.... ... '9'} corresponds to a digit in the decimal representation of fj and p is fixed sequence length corresponding to the number of digits necessary to represent Umar, Importantly, if the decimal representation of by requires less than p digits, the sequence is left-padded with zeros (ooltooio & Dupriez, 2023). This representation is lossless (App. A). but it leads to very long sequences which are frequently full of zeros. Indeed, the value distribution for a is usually Gaussian with very long tails. For instance, the "amount" field of bank transaction can range from tens of millions few cents, but most values smaller than 1,000 (e.g., 358).
[Di,... Dpl. where
To solve this problem, we propose a variable-range representation using a small, fixed number of digits preceded by magnitude order. For simplicity, let assume that uy is positive integer, thus:
V/ =
by 10 DR(,,) = [bm fombm-ze (b..bb....
(3)
k==
where m is the largest exponent and DR((;) is the representation of ny by means of sequence digits (e.g.. [35967]). Then, we represent fy using sequence Q defined as follows:
Q = 10, Dm, Dm-1,..., Dm-
(4)
In Eq. (4), O is the magnitude order prefix and it corresponds to m in Eq. (3). Specifically, in the tabular data domain we can assume that vy < 1000 hence, m E (O,..., 9} and O E {'O',... ..., '9'} is the token corresponding to m. The value of O is the first one that will be generated by D, when decoding the sequence representing a numerical value, which corresponds to predict its magni- tude order (m). We then encode the n most significant digits of Uy using Dm. Dm- Dm-ntt. where Dz E {'O', .... '9'} (m < S m n + 1) is the token corresponding to the digit by. C DR(0)) is the (m-depending) range we represent. For instance, if ey = 35967 and n = 4. then Q = ['4", '3', '5', '"'. Once decoded, Q can be used to com- pute the (possibly truncated) value of 0z. For instance, using Q |'4', '3', 3.." '5', '9', '6'], we get: f = 3 10a + 5 * 103 + 9 * 102 + 6 * 10 = 35960. We use = 4 regardless of attribute or dataset, and this value was chosen in preliminary studies as a trade-off between the length of the resulting sequences Q and the amount of truncated information (more details in App. A). On the other hand, if m < n (the most frequent case). no truncation is necessary and we right- pad Q with zeros. For instance, if , 35, then we use Q [',', '3', '5', '0', 0, ''O'. At decoding time, O is the first token generated by Dp. which is converted into m. If m < n. the last n - - tokens in Q are ignored because they are pero-padding digits (e.g., in Q [', "3', '5', '0', '0'), this corresponds to
and (onmb--....mm-4.

---

Published as a conference paper at ICLR 2025
the last 2 elements of Q). This implies that possible generation errors in the last m tokens are ignored. More formally, using our representation, the joint distribution over the tokens that the decoder should model is restricted to min(n + 1, m + 2) variables, as opposed to a joint distribution over p variables (p > m + 2) as in the case of the fixed digit sequence proposed in (Solatorio & Dupriez 2023), reducing the overall error probability (see App. A for more details).
Encoder and Decoder Architectures. Given a tabular row r, each categorical value is converted into a token and each numerical value is converted in sequence Q of I tokens (see above). After that, we use attribute-specific lookup tables to represent all the tokens as embedding vectors: ei....e,, where is the sum of the number of categorical attributes plus the number of the numerical attributes multiplied by (n + 1). The whole sequence e ei,..ey is fed to Eg, which, following (Zhang et al. 2024), is composed of two separated towers, respectively computing the mean (() and log variance (log or) of the latent representation 2 of r in 2. However, differently from (Zhang et al., 2024]. we use multi-head self-attention in &g because, in preliminary experiments, we found that this is more effective than single-head attention.
Using the standard VAE reparameterization trick, we obtain: = -,o..,, (N(, I), and # fed to the decoder Dep Moreover, differently from most DM-based tabular data generation approaches, we propose an AR decoder which is conditioned on z. Specifically, decoding a tabular row r starts with a special token [Start]. Every time the next-token is predicted, it is coded back and fed to De. which is a Transformer with 3 blocks. Each block alternates multi-head causal attention layers with respect to previously predicted tokens with (multi-head) cross attention layers the embedding vectors of z. For each categorical attribute az, final linear layer followed by softmax computes posterior over the specific vocabulary Vj. Similarly, if aj is numerical, it computes posterior for each of the n + 1 tokens in its variable-range representation Q. At inference time, we deterministically select token from each of these posteriors using arg max. Finally, the predicted value 1/ is autoregressively encoded into D. using the same field value representation mechanism used for the encoder (more details App.
4.2 VARIABLE-LENGTHI TIME SERIES
[ri..... r. and xz Tryl I; E Xx their length is usually different (T. * rj). Following FiT (Lu et al. 2024), we use maximum length Tmar and, for each I;, we append Traz Ti padding rows p (see below) to the right side of æ;. However, in FiT the padding tokens are used only to pack data into batches of uniform shape for parallel processing. and are ignored during the forward pass using a masked attention (basically, they are not used). Conversely, we use our padding rows to let the network decide the length of the time series it is generating. To do so, we add [EoS] token each categorical vocabulary Vj. included the vocabulary representing the digits, and we form the special row p = [[EoS] .... [EoS]]. During VAE training, with probability 0.05 we sample p and with 0.95 sample real row from R. During the denoising network training, if T; < Tmaz, I; is padded with p. which results in 8, being sequence of Tmaz latent vectors, where the last vectors correspond to the representation of p in Z and contribute to compute the loss. Finally, at inference time, we randomly sample Tmox vectors Z1,..., Tmaa in Z. which form ST = [=....., TTmazl 1. the starting point of the DM sampling chain. The ending point of the sampling chain su is split in Tmaz final vectors 2 z'.... which are individually decoded using Dzp We stop decoding as soon as we meet the first padding row p. More precisely, we stop decoding when we meet the first row containing at least one token [EoS]. In this way, we use Fo to predict the end of the time series jointly with its content.
Given two sequences
EXPERIMENTS
5.1
EVALUATION PROTOCOL.
Due to the lack of a shared evaluation protocol for tabular data time series generation, we propose unified framework which is composed of different public datasets, conditional and unconditional generation tasks and different metrics. More details are provided in App- C and E
Datasets. We use six public datasets, whose statistics are provided in App. Agel. Age2, Leaving, taken from (Fursov et al., 202T). and PKDD '99 Financial Dataset, taken from (Berka 1999), are composed of bank transaction time series of different real banks with different attributes. Fach time
7

---

Published as a conference paper at ICLR 2025
series is the temporally ordered sequence of bank transactions of given bank client. On the other hand, the Rossmann and the Airbmb datasets, used in (Patki et al. 2016)] Solatorio & Dupriez 2023), composed of, respectively, historic sales data for different stores and access log data from Airbnb users. These six datasets are widely different from each other both in terms of their attributes and their sizes, thus representing very different application scenarios. Two of these datasets, Age/ and Leaving, do not have an associated parent table, so they are used only for unconditional generation.
Metries. Since heterogeneous time series generation is relatively unexplored domain, there is also a lack of consolidated metrics. For instance, [Solatorio SSsttrcc & Dupriez (2023) use the Logistic Detection, which discriminative metric based on training a binary classifier to distinguish between real and generated data, and then using the ROC-AUC scores of the discriminator (the higher the better t) on an held-out set of real and synthetic data (App. [c. We also adopt this metric in Sec. [.33 to compare our results with RFal.TabFormee (a Random Forest) takes as input only an individual row of the real/generated time series, thus this criterion is insufficient to assess, c.g., the temporal coherence of a time series composed of different rows (e.g., see App. H.3 for few examples of time series generated by REaLTabFormer with incoherent sequence of dates). For this reason, in this paper we extend this metric using classifier which takes as input the entire time series instead of individual rows. Inspired by similar metrics commonly adopted in single-row tabular data generation (Liu et al.. 2023), we call our metric Machine Learning Detection (MID) and we measure the discriminator accuracy (the lower the better J. because it means that the discriminator struggles in separating the real from the synthetic distribution). To emphasize the difference with respect to the Logistic Detection used in Solatorio & Dupriez (023). use the suffix "SR" (single row) for the latter (LD-SR) and the suffix "TS" (MLD- TS) to indicate that our MLD metric depends on the entire time series. More specifically, in MLD-7S we use CatBoost (Prokhorenkova et al.) 2018) as the classifier, because is one of the most common non-deep learning based methods for heterogeneous tabular data discriminative tasks (Luetto et al.,
However, the classifier used in the Logistic Detection
2023 Gorishniy et al., 2021| Chen & Guestrin 2016), jointly with standard library (Christ et al.,
2018
extracting a fixed-size feature vector from time series (Luetto al.) 2023). We provide
more details on these metrics in App.[C. where we also introduce an additional metric based on the Machine Learning Efficiency (MLE) (Zhang et al. 2024 Kotelnikov et al. 2023).
5.2 ABLATION
In Tab. [ we use Age2, which, among the six datasets, is both one of the largest in terms of number of time series and one of the most complex for different types of attributes. The results in Tab. [ are based on an unconditional generation task and evaluate the contribution of each component of our method. The last row, TabDiT, refers to the full method as described in Sec. 4 while all the other rows refer to our full-model with one missing component. For instance, Parallel VAE refers to a standard, non-AR VAE decoder, in which all the fields of taauuarr row are predicted in parallel. No cross-att VAE differs from the VAE introduced in Sec. 4.1 because in Dip we remove the cross attention layers to z, and z is directly fed to Drp in place of the [Start] ] token. In all the other entries of the table we use our AR VAE as described in Sec. |.1 Fixed digit seq corresponds to the numerical value representation proposed in REaLTabFormer (Solatorio & Dupriez, (023), in which use p 7 digits (Sec. 4.1, which In Quantize, we follow TabGPT (Padhi et al. (021) and we convert each numerical feature into a categorical one, using quantization and field specific vocabulary. In Linear transf. we follow Tab- Syn (Zhang et al. 2024), where numerical features are predicted using a linear transformation of the corresponding last-layer token embedding of the VAE decoder. We refer to App. A for more details on these representations. In all the variants, we always use a coherent numerical value representa- tion in the corresponding VAE encoder. We also evaluated hybrid solutions, where VAE encoders and decoders have different representations of the numerical values, but we always obtained worse results than cases of coherent representations. Moreover, in W/o length pred we follow FiT (Lu) et al. 2024) and we use masked attention which completely ignores the padding rows. In this case, at testing time the sequence length is randomly sampled using a mono-modal Gaussian distribution fitted the length of the training time series. Finally, AR Baseline is a (strong) baseline based purely AR Transformer which we use to validate the effectiveness of the DM paradigm (see below).
enough represent all the numerical values in Age2.
In our AR Baseline, modify the hierarchical discriminative architecture proposed in (Padhi et al. 202T) and adopted also in (Luetto et al.) 2023) to create an AR generative model. Specifically, causal attention in the "Sequence Transformer" and we replace the "Field Transformer"
8

---

Published as a conference paper at ICLR 2025
Table 1: Ablation study on the Age2 dataset.
Method
Parallel VAE No cross-art VAE
Fixed digit
Quantize
Lincar transf
W/o longth pred
AR Baseline
TabDiT
MLD-IS
622
51.2
64.0
83.8
83.7
61.0
68.3
50.8
with one of the towers of our VAE encoder architecture. Moreover, we use our AR VAE decoder architecture to predict the output sequence. Note that we use the VAE architectural components but we do not use variational training and the entire network trained end-to-end using only a next- token prediction task, following (Padhi et al. 2021; [Solatorio & Dupriez, [023). The numerical features are represented using our variable-range decimal representation. In short, this baseline is a
purely AR Transformer where we merged the hierarchical architecture used in (Padhi et al.) 2021
Luetto et al. 2023) with our decoding scheme and our feature value representation.
Tab. 1 shows that all the components of the proposed method are important, since their individual removal always leads to a significant decrement the MLD. Specifically, the numerical value rep- resentation has high impact on the results. For instance, both Linear transf, adopted in (Zhang al. 2024). and Quantize, used in (Padhi et al., 202T), lead to a drastic worsening of results. On the other hand, our AR Baseline, which is based on the same input representation and shares important architectural details with TabDiT, is largely outperformed by the latter, showing the advantage of our LDM-based approach. Finally, the difference between No cross-att VAE and the full method is sub- tle, showing that the cross attention layers in Di can be replaced by directly feeding z to the decoder, as long as the latter has an AR architecture (see App. D.3 for additional ablation experiments).
5.3 MAIN EXPERIMENTS
Following the protocol adopted in (Solatorio & Dupriez 2023), all the experiments of this section have been repeated three times with different random splits of the samples between the generator training data, the discriminators' training data and the testing data (see App- C and E for more details). For each experiment, we report the means and the standard deviations of the MLD-TS and the LD-SR metrics. In App. ApI. we show additional experiments using other metrics App. C
Unconditional generation. We are not aware of any unconditional generative model for time se- ries of heterogeneous tabular data with public code or published results. Indeed, REaLTabFormer is conditional method, while TabGPT is a forecasting model, both lacking of an unconditional sampling mechanism (Sec. [ and D For this reason, we can only compare TabDiT with our AR Baseline (Sec. 5.2). The unconditional results in Tab. and 3 show that TabDiT outperforms the AR Baseline in all the datasets by a large margin. For instance, in the largest dataset (Agel), Tab- DiT improves the MLD-TS score by more than 27 points compared to the AR baseline. On Age2, TabDiT outperforms the AR Baseline by more than 17 points, achieving an almost ideal situation in which the real and the generated distributions cannot be distinguished by each other (discrimination accuracy = 50.43%, very close to the chance level). Note that the Age2 results are slightly different from those reported in Tab. 1 because they were obtained averaging 3 different runs.
Conditional generation. We indicate with "child gt-cond" the conditional generation task in which the parent table row used for conditioning, is ground truth, real row extracted from the testing dataset (Sec. 3.. Specifically, we use all the elements u E Prest (Sec. 3) to condition the generator networks. Moreover, make comparison with (Solatorio & Dupriez (2023) possible, we also fol- low their protocol and we indicate with "child" the conditional generation task on the time series (x) where also the conditioning information (u) automatically generated. Specifically, in our case, we generate u by training a dedicated DiT-based denoising network and a corresponding AR VAE on P. These networks have the same structure and are trained using the same approach described in Sec. but using P instead of X (more details in App.B) B). Finally, in (Solatorio & Dupriez (003) "merged" indicates the evaluation of the joint probability of generating both x and u. Using MLD-TS and LD- SR, this is obtained by concatenating u with either æ or with its individual rows respectively, and then feeding the result to the corresponding discriminator (see App. In Tab. 4 we indicate with the results of REaLTabFormer and SDV which we report from (Solatorio & Dupriez 2023). while in all the other cases they have been reproduced by us using the corresponding public available code. Specifically, while using Fabboomee with real data-conditioning task ("child gt-cond")

---

Published as a conference paper at ICLR 2025
is easy, that was not possible for SDV. The choice of REaLTabFormer and SDV follows (Solatorio & Dupriezz 2023). where the selected baselines are those which have open-sourced models.
The results in Tab. 4 confirm the results in Tab. 2 and 3 Across all datasets and tasks, and with both metrics, TabDiT outperforms all the baselines by a large margin, often approaching the lower bound of 50% MLD-TS accuracy. In the "child gt-cond" task, AR Baseline is the second best most of the time, while in "child" and "merged" the second best is Raa.TabFormmr. We believe that the reason of this discrepancy is most likely due to the fact that both the "child" and the "merged" task evaluation depend on the quality of the single-row parent generation, in which REaLTab better results (see App. B). Finally, in App. [. we present additional experiments using the MLE (Sec. [5.]]. where we also show how the generated data can be effectively used to replace real data for classification tasks (Sec.
gets
D. and in App. D.2 we extend the results of this section to larger datasets.
Table 2: Unconditional generation results on Rossmann, Airhmb and PKDD '99.
Rossmann
Airbub
PKDD'99 Financial
Method
MLD-TS&
LD-SR 1
MLD-TS
LD-SR 1
MLD-TS L LD-SR
AR Baseline (ours) TabDiT (ours)
97.80.2.20
49.97.3.25
77.23
26.43
92.87.1.s 85.53
71.93-1.4
82.60.3.0
77.0725.7
55.07
78.07
79.10.c.0
Table 3: Unconditional generation results on Age2, Agel and Leaving.
Age2
Agel
Leaving
Method
MLD-TS
LD-SR f
MLD-TS
LD-SR ?
MLD-TS 1 LD-SR
AR Baseline (ours)
83.47 67.53g0m
91. 91.20 as
74.23g1.0
75.33ga 62.33.cm
69
TabDiT (ours)
50.43:1.a
87.00
63,93,3.0
76.00_1.3
Table 4: Conditional generation results.
Values reported from Solatorio & Dupriez 2023
Rossmann
Airbab
Age2
99 Financial
Method
Task
MID-TS.
LD-SR 1
MLD-TS J
LD-SR f
MID-TS
LD-SR 1
MLD-TS.
LD-SR I
SDV
child
99.63 10.65
6.53"-9.0
93,30
2.00"
96.03 1011
44.80.1.71
97.95
11.42
6.53
merged
100.00 2000
2.80
21
2.00"
96.27 2010
37.63g1
98.12
8/77200
child gt-cond
98,90
60.63-241
63.63
86.17 41.29
66.77 10.22
77.90-0.5
97.87-a.5
21.97.0.3
REal.TahFormer
chili
64.83
$2.08
57.77
30.45
$2.97
27 30
59.33
21 21.50
merged
74.43 15.00
28.33
76.97 2914
113.00 21
52.10 17
75.53 2:
58.77 13
26.00.1 mt
AR Baseline
child gr-cond
95.57
71 60 27.45
$7.97
82.77
69.97
80.73 2n
68,33
81. 21
99.63
8.72
12.33
13.03
79.03
79.07
.4.38
(ours)
merged
99.633.044
19.70-9.0
93.50
8.53 a
81.30 na
48.03-a4.
83.33,4.16
38.73 -4.22
child gr-cond
72.20.3.30
82.90
51.10
98.07 .0.27
$1.40 vo
84.60
59.50210.51
81.20
TabDiT (ours)
60.44
13.10
81.10
50.47
6..44
79.13
merged
71.83:2.77
38.63.1.04
14.63....6
47.37-2245
51.53
78.93.0.44
$4.03
23.20.o.0
CONCLUSIONS
We presented TabDiT, an LDM approach for tabular data time series generation. Differently from the common LDM paradigm, where an entity domain holistically represented in the VAE latent space, we split our time series in individual tabular rows, which are compressed independently one from the others. This simplifies the variational learning task and avoids the need represent variable length sequences in the VAE latent space. Then, a DiT denoising network combines embedding vectors defined in this space into final sequences, this way modeling their temporal dynamics. Moreover, our DiT explicitly predicts the end of the generated time series to jointly model its content and its length. Furthermore, we proposed an Autoregressive VAE decoder and a variable-range decimal representation of the numerical values to encode and decode heterogeneous field values. Using extensive experiments with six different datasets, we showed the effectiveness of TabDiT, which largely outperfoms the other baselines in both conditional and unconditional tasks. Specifically, TabDiT is the first network showing results for unconditional generation of time series of tabular data with heterogeneous field values.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by the EU Horizon project ELIAS (No. 101120237). We thank the Data Science team of Prometeia S.p.A. for their valuable collaboration and for providing useful discussions and insights that contributed this work.
REFERENCES
Saad A. Abdelhameed, Sherin M. Moussa, and Mohamed E. Khalifa. Privacy-preserving tabular data publishing: comprehensive evaluation from web to cloud. Computers & Security, 72: 74-95, 2018.
Samuel A. Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E. Tillman, Prashant Reddy, and Manuela Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on Al in Finance, ICAIF, 2020.
Petr Berka. Workshop notes on Discovery Challenge PKDD'99. 1999. https://sorry use.
cz/ www.berrkhhhho cha Saaerka/challengeppeee.cc lenge/pkdd /99/ber99 ktm
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Lan- guage models realistic tabular data generators. In ICLR, 2023.
Zhengping Che, Yu Cheng, Shuangfei Zhai, Zhaonan Sun, and Yan Liu. Boosting deep learning risk prediction with generative adversarial networks for electronic health records. In IEEE Interna- tional Conference on Data Mining (ICDM), 2017.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, and Jimeng Sun. Gen- crating multi-label discrete patient records using generative adversarial networks. In Proceedings of the 2nd Machine Learning for Healthcare Conference, 2017.
Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W. Kempa-Liehr. Time series feature extraction on basis of scalable hypothesis tests (tsfresh python package). Neurocomputing, 307:72-77,2018.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs image synthesis. In NeurlPS, 2021.
Dmitry Efimov, Di Xu, Luyang Kong, Alexey Nefedov, and Archana Anandakrishnan. Using gen- erative adversarial networks to synthesize artificial financial datasets. arXiv:2002.02277, 2020.
Joao Fonseca and Fernando Bacao. Tabular and latent space synthetic data generation: a literature review. Journal of Big Data, 10(1), 2023.
Ivan Fursov, Matvey Morozov, Nina Kaploukhaya. Elizaveta Kovtun, Rodrigo Rivera- Castro, Gleb Gusev, Dmitry Babaev, Ivan Kireev, Alexey Zaytsev. and Evgeny Burnaev. Adversarial attacks on deep models for financial transaction records. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining (KDD), 2021.
Ian Goodfellow, Jean Abudet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In NeurIPS, 2021.
Manbir S Gulati and Paul F Roysdon. TabMT: Generating tabular data with masked transformers. In NewrlPS, 2023.
Mikel Hernandez, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin. Synthetic data generation for tabular health records: systematic review. Nearocomputing, 493:28 45, 2022.

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In NeurIPS, 2021.
James Jordon, Jinsung Yoon, and Mihaela van der Schaar. PATE-GAN: Generating Synthetic Data with Differential Privacy Guarantees. In ICLR, 2018.
Jayoung Kim, Chaejeong Lee, Yehjin Shin, Sewon Park, Minjung Kim, Noseong Park, and Jihoon Cho. SOS: Score-hased Oversampling for Tabular Data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2022.
Jayoung Kim, Chaejeong Lee, and Noseong Park. Stasy: Score-based tabular data synthesis. In ICLR, 2023.
Diederik P. Kingma and Max Welling. Auto-Ene Encooinn Variational Bayes. In JCLR, 2014.
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Bahenko. TabDDPM: modelling tabular data with diffusion models. In ICML., 2023.
M. Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NeurlPS, 2010.
Chaejeong Lee, Jayoung Kim, and Noseong Park. CoDi: co-evolving contrastive diffusion models for mixed-type tabular synthesis. ICML, 2023.
Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, and Mihaela van der Schaar. GOGGLE: Generative modelling for tabular data by learning relational structure. In ICLR, 2023.
Zeyu Lu, Zidong Wang. Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. FiT: Flexible Vision Transformer for Diffusson Model. 1wXiv:2402.12366 2024.
Simone Luetto, Fabrizio Garuti, Enver Sangineto, Lorenzo Forni, and Rita Cucchiara. One trans- former for all time series: Representing and training with time-dependent heterogeneous tabular data. arXiv:2302.0633 2023.
Avanika Narayan, Ines Chami, Laurel Orr, and Christopher Ré. Can foundation models wrangle your data? Proc. VLDB Endowment, 16(4):738-745) 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre L.. Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In IEEE International Conference Acoustics, Speech and Signal Processing, ICASSP, 2021.
Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Young- min Kim. Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment, 11(10):00711000 2018.
Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. The synthetic data vault. In IEEE Interna- tional Conference on Data Science and Advanced Analytics (DSAA), 2016.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Liudmila Ostroumova Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In NeurlPS, 2018.
Zhaozhi Qian, Bogdan- Cebere, and Mihaela van der Schaar. Syntheity: facilitating innovative use cases of synthetic data in different data modalities. arXiv:2301.07573. 2023.
Matteo Rizzato. Julien Wallart, Christophe Geissler, Nicolas Morizet, and Noureddine Boumlaik. Generative adversarial networks applied to synthetic financial scenarios generation. Physica A: Statistical Mechanics and its Applications, 623, 2023. ISSN 0378-4371.
12

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. Medical Image Computing and Computer-ssisted Intervention (MIC- CAI), 2015.
Aivin V. Solatorio and Olivier Dupriez. REaLTabFormer: Generating realistic relational and tabular data using transformers. arXiv:2332.02041, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar. Decaf: Generating fair synthetic data using causally-awar generative networks. In NeurlPS, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrIPS, 2017.
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. In NeurlPS, 2019.
Hengrui Zhang, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score- based diffusion in latent space. In ICLR, 2024.
Shuhan Zheng and Nontawat Charoenphakdee. Diffusion models for missing value imputation in tabular data. In NewrlPS Table Representation Learning (TRL) Workshop, 2022.
A NUMERICAL VALUE REPRESENTATIONS
In this section, we provide more details on the numerical value representation approaches used by previous work and in our ablation analysis in Sec. 5.2, as well as on our variable-range representa- tion. For clarity, we adopt the same terminology and numerical examples used in Sec. 4.1
TabGPT (Padhi et al. 2021 applies a quantization which associates by with bin value B, which is then treated as a categorical feature. The disadvantage of this representation is an information loss due to the difference between the decoded bin B and the actual value vy.- Indeed, B corresponds to the center of the numerical interval assigned to the B-th bin during the quantization phase. This coding-decoding scheme corresponds to the entry Quantize in Tab. [
TabSyn Zhang et al. 2024) applies a linear transformation when coding and another linear transformation (i.e., linear regression) when decoding back the embedding vector of the last-layer decoder to ey. However, also this solution is sub-optimal, because linear regression struggles in respecting some implicit value distribution constraints. For instance, the admissible values for aj are only integers, linear regression layer may predict a number with a decimal point. This scheme corresponds to the entry Linear transf in Tab. []
Finally, Fixed digir seq in Tab. corresponds to the sequence of digits L (see Sec. used in REaLTabFormer (Solatorio Dupriez (023). We believe that one of the reasons why this rep- resentation is sub-optimal with respect to our variable-range decimal representation Q (Sec. 4.1 and (52) that its longer length leads, on average, to a more difficult decoding problem. For in- stance. in the Age2 dataset, the "amount" attribute needs p 7 digits to represent Umez, Thus, the length L is p = 7. When the VAE decoder should decode numerical value, the probability of error is given by the (complement of the) joint distribution of all the digits of its representation L. For instance, if the target value is Uj = 35, then, the decoder should generate this sequence: L == ['O', '0. '0', '0', '0', 3', '5']. The probability of error when generating L is:
PL 11 = II DI .D.1),
(5)
13

---

Published as a conference paper at ICLR 2025
which, in case of by 35, is:
PL('0'00', '0', "000,"03330000000000000000000000 P2 '0',"""''''"0""""", '0','",,'''"" (0)]
(6)
On the other hand, in case of Q (Sec. 4.1 since we use AR decoding, once the magnitude order prefix O has been generated, we can convert O in its corresponding value m (Eq. (3) and use m to ignore possible cero-padding tokens on the right side of the sequence. Specifically, if n, the probability of error is given by:
T
Pq = 1- 10 = (P((O) II
Pe(Dm-x 10, Dm..... Dm-k+ A=
which, in case of of = 35, Q 13',''',3 '5, '0, '0' and m is:
1 [=T((1))(((3))))2(((1})
The comparison between Eq. (6) and Eq. ( intuitively shows that Q is much easier generate than L if vy is small. More formally, if we assume that all the digit generations have, average, the same probability to be correct, i.e., that, on average: (O) PL (D,) and (Dx D&-1)
Po(Dm-alO Dm, ..., D then, if m < 2, from Eq. 5 and follows that Pt, Pq.
The proposed representation can be easily extended to negative numbers and non-integer values. For instance, if the admissible values for a; include negative numbers, then we prepend in Q token S corresponding to the sign of U₁: Q [5, O, Dm, Dm-1,..., Dm-n+1l where S E ('-', '+'.. Note that, in case of negative numbers, also L should be extended to include S (Solatorio & Dupriez, 2023). On the other hand, in case of rational numbers, needs to be extended to include token representing the decimal point (Solatorio & Dupriez, 2023), while our variable rrage representation Q remains unchanged. For instance, if Ug = 3.5, then we have Q = ['O,," '3', '5', '5','',
chosen. We used the Age2 dataset (Fursov et al. 2021). adopted for most of our ablation studies. For each numerical values fy in Age2, we compute DR(;;) (Eq. (3)) and remove from DR(U;) the possible subsequence of only zeros on its right (e.g., from [87600] we remove [00].. Note that these all-zero subsequences do not lead any truncation error. Let Significant(v;) be the part of DR(v;) that remains after this cut (e.g., (876)). Finally, we computed the average (Jus) and the
Finally, provide below more details how
in the previous example, Significan((e,) standard deviation (ass) of the lengths of all the sequences Significant v;) in the training dataset, getting μĄş = 2.26 and Øs = 0.47, respectively. The value n = 4 was chosen as the first integer greater than #ss + 20ss. In this way, most of the training data Age2 do not need any truncation when represented using Q. The value n 4 was used in all the other datasets. In App. H.1 we show some qualitative results which compare to each other the distributions of some numerical values generated using the representations presented in this section.
B TABDIT FOR SINGLE TABULAR ROW GENERATION
Generating single tabular rows is out of the scope of this paper, in which we focus on the (more challenging) generation of time series. However, following the protocol proposed in (Solatorio) & Dupriez 2023). in Sec. 5.3 we show conditional experiments where the "parent" row table is automatically generated (tasks "child" and "merged"). In order to generate a row u following the empirical distribution in P (Sec. B.. we adapted our TabDiT to single row generation task. Specifically, we train a parent-table dedicated VAE encoder &P. and decoder Ppp as well as a dedicated DiT-based denoising network J.P. These networks all have the same structure those used for time series generation (Sec. 4p. including the same number of layers and parameters. The only difference is that the sequence of embedding vectors fed to Fop corresponds to individual field
14

---

Published as a conference paper at ICLR 2025
Table Parent generation results using the LD-SR i metric (mean and standard deviation over three runs). "hese values are reported from Solatorio & Dupriez (2023).
Method
Rossmann
Airbnb
Age2
PKDD'99 Financial
SDV
31.77* 13.41
7.37
20.72
55.97
42.14
37.87
22.50
REaLTabFormer
81.04
+4.54
89.65*
+1.92
98.13
+2.73
98.70
+2.17
AR Bascline (ours)
52.47 +9.31
13.33 =3.12
63.03
+3.38
62.97 #8.13
SR-TabDiT (ours)
91.60 23.80
84.53
20.45
91.90
10.92
81.77 24.72
value embeddings of a single tabular row in the VAE latent space. Specifically, if y EP (u) is the latent representation of u = [w..... wn] E P. we split y in h separate vectors, 8/: .... Bh corresponding to the final embeddings of the field values wi: W/, in the last-layer of E Then, 0p"
po the sequence of embeddings used in the forward process for training FOP is su = ly..... y.l. The
rest of the training and sampling procedures follows the method presented in Sec. 4 We call this method Single Row TabDiT (SR-TabDiT).
For the AR Baseline, is simply concatenated on the left-side of æ, and we autoregressively gener- ate the sequence [4. x) starting from a [SoS] token. Tab. 5 shows the results on the "parent" (alone) generation task following the protocol used in (Solatorio Dupriez (023). In most datasets, RE- aLTabFormer beats SR-TabDiT, which is the second best. However, as mentioned above, the goal of this paper is to propose time series generation approach, and we have developed a single-row generation model only to make a comparison with REaLTabFormer possible using the "child" and the "merged" tasks proposed in that paper. Note also that we have not optimized our method for this task and, most likely, simply reducing the number of layers and parameters of SR-TabDiT may help in regularizing training. However, we leave the study of how to better adapt TabDiT to a single-row generation task as future work. Moreover, note that in Tab, 4 TabDiT largely outperforms REaLTab- Former also in the "child" and the "merged" tasks despite it is conditioned on generated parent rows with lower quality compared to REaLTabFormer. Indeed, success these tasks necessarily de- pends on the quality of the row u that was generated before the conditional process. Finally, we note that in Tab. 4 in most datasets the AR Baseline beats REaLTabFormer in the "child gt-cond" task, in which the parent row is not generated, but underperfoms Raa.TabFormer in the "child" and the "merged" tasks. Most likely, the reason is that also our AR Baseline is disadvantaged due to lower quality parent row generation.
C METRICS
Besides the discriminative metries described in Sec. 5.1 in the single-row tabular data generation literature there are many other evaluation metrics, which however we do not believe suitable for the time series domain. For instance, low-order statistics include statistics computed either using the values of an individual tabular attribute or statistics such as the pair-wise correlation between the values of two numerical attributes (Zhang et al. 2024). However, in time series composed of dozens of rows, each row composed of different fields, statistics on a single field or pairs of fields not very informative. Similarly, we do not use high-order statistics (c.g., a-precision and 3-recall) (Zhang et al.) (2024), because they are single-row criteria and they usually use a fragile neirest-neighhor like approach in the data space estimate the distribution coverage.
On the other hand, we believe that the most useful metric is the MLD-TS proposed in Sec. 5.1 for which we provide below additional details. The CatBoost (Prokhorenkova et al. 2018) discriminator is trained using a balanced dataset composed of both real and synthetic data (separately generated by each compared generative method). Then, a separate testing set, also composed of 50% real and 50% generated data, is used to assess the discriminator accuracy. Real training and testing data do not include data used train the generator. Thus, basically the real data are split in: samples used to train the generator, samples used (jointly with synthetic data) to train the discriminator, and samples used (jointly with other synthetic data) test the discriminator. We randomly change these three splits in each of the run used to compute the results in Sec. 5.3
'The evaluation protocol code is available at: https://gi warooo//aabrii http://aaaabiiiiigttrrrt///b.b
15

---

Published as a conference paper at ICLR 2025
We use the same training-testing protocol for the Logistic Detection, which, following (Solatorio & Dupriez 2023), is defined as: LD-SR = 100 x (1 - (RA),, where:
1 F
Σ max(0.5, ROC AUC) 2 1.
PRA
In Eq. ( ROC and AUC indicate the ROC-AUC scores, computed using Random Forest trained and tested using single tabular rows. F = 3 is the number of cross-validation folds, in which training and testing of the discriminator is repeated F times, keeping fixed the generator weights.
We use this metric with its corresponding publicly available code (Solatorio & Dupriez (023) for fair comparison with RFaa.TabFormer (Solatorio & Dupriez, 2023) and SDV (Patki et al.] 2016).
Specifically, in the conditional generation scenario, we follow (Solatorio & Dupriez 2023) and we evaluate the LD-SR for the "merged" task by concatenating with all the rows r extracted from generated time series I. The Random Forest is then trained and tested on this "augmented" rows. In case of MLD-7S, we concatenate u with the exed-dimensee feature vector extracted from using the feature extraction library (Christ et al. 2018) (Sec. 5.1 and we use the "augmented" feature vector train and test CatBoost.
Finally, we introduce an additional metric based on the Machine Learning Efficiency (MLE) (Zhang al. 2024 Kotelnikov al. 2023), which is based on the accuracy of a classifier trained on gener- ated data and evaluated on real data testing set. MLE can also be used to simulate an application scenario in which, for instance, the generated data are used to replace real data (e.g., because pro- tected by privacy legal constraints, see Sec. [ in training and testing public machine learning methods. In this case, the classifier accuracy, when trained with synthetic data (only) is usually upper bounded by the accuracy of the same classifier trained the real data. Similarly to MLD-TS, we adapt this metric (which we refer to as MLE-TS) to our time series domain using CatBoost as the classifier, fed using fixed-size feature vector extracted from given time series (Christ al., 2018) (Sec. D. For simplicity, we always use binary classification tasks, whose lower bound is 50% (chance level), and in App. D.I we show how these tasks can be formulated selecting specific attributes of the parent table to be used as target labels.
D ADDITIONAL EXPERIMENTS
D.1 MACHINE LEARNING EFFICIENCY
In this section, we show additional experiments using the MLE-TS metric introduced App. Since this metric is based on training a classifier to predict a target label, we can use MLE-TS only evaluate conditional generations, where these labels can be extracted from the parent table of the corresponding dataset (individual time series or rows are not labeled). Moreover, the "merged"" task cannot used in this case because, as defined in (Solatorio & Dupriez, 2023), at inference time, it involves the concatenation of the generated data with the conditioning parent table (App. C). the latter used to extract the ground truth target labels. Specifically:
In Rossmann, we use the binary attribute "Promo2", indicating the presence of a promo in that store.
In Airbnb, we use the attribute "n sessions" which indicates the number of sessions opened by a user. We binarize this attribute predicting whether the user has opened more than 20 sessions (n_sessions >= >=20).
In Age2, we use the attribute "age", indicating the age of a customer. In particular, we predict whether the customer over 30 years old (age 30).
In PKDD' '99 Financial, belongs to. We predict whether the customer is located in Moravia or in Bohemia (including Prague).
the attribute "region" which indicates the region customer
In Tab. 6 "Original" indicates that the classifier has been trained on real data and tested real data, and it is an ideal value of the expected accuracy when the same classifier trained on synthetic data. In the same table, "child" and "child gt-cond" refer the tasks used in Tab. Specifically, in both
16

---

Published as a conference paper at ICLR 2025
cases the classifier is trained with the generated data and tested on real data. However, in case of "child gt-cond". we use the real parent table row values as the classification target labels (see above). Conversely, in case of "child", we use the generated parent table rows. This is because, in realistic scenario, the "child" task corresponds to a situation in which the parent table data are missing and they are generated as well (Sec. (533). thus we coherently use the synthetic parent table values to label the corresponding generated time series. The real time series are always labeled with their corresponding real parent table values.
The results in Tab. conffirm those reported in Tab, - showing that TabDiT significantly outperforms all the other baselines in all the datasets. Specifically, in Age2, TabDiT even surpasses the ideal "Original" accuracy (61.57 versus 60.07), being very close to the ideal case in all the other datasets. We believe that these results show the effectiveness of the synthetic time series generated by our method, which can potentially be used to replace real data in machine learning tasks when, e.g., the real data cannot be made public.
Table 6: Conditional generation results using the MLE-TS metric (mean and standard deviation over three runs).
Method
Task
Rossmann
Airbnb
Age2
PKDD'99 Financial
Original
66.70 28.00
100.00 20.00
60.07
21.74
61.67 22.95
SDV
child
54.10 #2.60
93.97 #1.37
54,43 #1.00
61.30 #4.81
child gt-cond child
53.33 25
60.87
54.53+1.42
61.10=2.44
REaLTabFormer
53.37 88
61.97 11.67
54.97
61.47_8.20
AR Baseline
child gt-cond child
59.27 _1.27
100,00 10.00
58.2311.90
62.03 15.41
(ours)
50.37 26
59.9012.46
55.77
60.73_3.54
child gt-cond child 65.93
64.43_11.44
100.0010.00
61.57
61.30_1.70
TabDiT (ours)
99.6310.10
61.23
62.60_3..
D.2 LARGER SCALE EXPERIMENTS
In this section, we use a much larger dataset to show the potentialities of our method to be scaled. Since, as far as we know, heterogeneous time series datasets considerably larger than those used in Sec. [3] are not publicly available, we used a private dataset which we call Large Scale Bank Data, and which was provided by an international bank Large Scale Bank Data consists of several hundred million real bank account transactions of private customers. From this dataset, we randomly selected 100K client bank accounts, corresponding to approximately 87.3M transactions (i.c., rows), which we use to train TabDiT and the AR Baseline. Moreover, we selected another set of 10K bank accounts (not included in the training set) for evaluation. Furthermore, with Large Scale Bank Data use longer time series, setting Tmax 100, which corresponds to an average of one month of bank transactions of a given customer (see App. for more details).
In Tab. 7 we compare TabDiT with AR Baseline using the "child gt-cond" task. Note that we were not able to use REaLTabFormer on this large-scale datasets for computational reasons. Indeed the long length of the time series (Tmaz 100), combined with the larger number of time series fields (9, as reported in Tab. g. led to memory usage and time complexity problems during training with RBaL.TaoFormer. Conversely, both the hierarchical architecture of AR Baseline (Sec. 5.2) and our LDM approach (Sec. [ allow a much faster and lower memory consumption training, which made possible to use a huge dataset like Large Scale Bank Data.
In Tab. 7 we report the MLD-TS, the LD-SR, and the MLE-TS metric values. These results show that TabDiT significantly outperforms AR Baseline. Moreover, even when using a large-scale dataset, TabDiT approaches the lower bound of 50% MLD-TS accuracy, achieves score higher than 80% LD-SR, and is able approach the upper bound of MLE-TS obtained using the real data.
Forr both privacy and commercial reasons, this dataset cannot he released.
17

---

Published as a conference paper at ICLR 2025
Table 7: Large scale conditional experiments using the "child gt-cond" task and Large Scale Bank
Data.
Method
MLD-TS
LD-SR f MLE-TS 1
Original
73.22
AR Baseline (ours)
58.03 56.77
83.33
70.41
TabDiT (ours)
83.72
71.38
Table 8: 9KDD'99 Financial dataset. Analysis of the influence of the CFG hyperparameter values using the MLD-TS & metric (mean and standard deviation over three runs).
Pd
2
3
4
0.001
83,33 =3.61
72.67
=3.52
78.33
+7.86
71.13
#9.38
69.33
#7.92
0.005
80.47
=6.02
73.00
8..84
72.47
#14.95
59.50
+10.53
75.00
#12.30
0.010
77.87 10.40
70.80
10..83
69.50
#13.95
76.83
16.40
73.97
#9.51
0.100
80.67 44.74
71.10
69.13
66.77
=12.79
69.63
D.3 ADDITIONAL ABLATIONS
In this section we present additional ablation studies. In Tab. we use the "child gt-cond" task to investigate the influence of the CFG and its hyperparameter values and Pa (Sec. (4) For these ex- periments we use PKDD '99 Financial dataset instead of Age2 (used in all other ablations) because, on Age2, TabDiT achieves a nearly ideal MLD-7S score (~ 50%) even without CFG, so there is no room for improvement. and the results reported in the table clearly shows that CFG is beneficial for conditional generation of tabular data time series, a finding which aligned with the empirical importance of CFG in the image generation literature (Peebles & Xie 2023). Using these results, selected the values Pa = 0.005 and tasks. Although a dataset-dependent hyperparameter tuning may likely lead even better results, we opted for simpler and computationally less expensive solution based on dataset-agnostic CFG hyperparameters.
Tab. 8 the scale value = 1 corresponds to non CFG (see Eq. (2). )
and we used these hyperparameter values in all the datasets and conditional
For training our VAE we adopt the scheduling proposed in (Zhang et al. 2024), where the recon- struction loss and the KL-divergence loss are balanced using a coefficient 3 which weights the importance of the latter. The value of 3 starts from an initial Bmaz and, during training, it is pro- gressively and adaptively reduced. We refer to (Zhang et al., (024) for more details. Fig. P we use Age2 to show the impact of 3maz. which is evaluated jointly with the number of diffusion steps of the denoising network (Sec. 3}. evaluate the values of 3maz and T jointly because they are strongly related to each other. This ablation shows that using relatively small number of dif- fusion steps (c.g., greater than 100) is sufficient to get good results, and that there is no significant improvement with very long trajectories. Conversely, higher value of Bmaz. corresponding to higher regularization of the latent space, improves the results. In all the datasets and tasks, we use 200 diffusion steps and Binax 5.
E DATASETS
In Tab. 9 report the main characteristics of the datasets used in our experiments, We use six real- world public datasets: Age2"| Age Leaving the PKDD '99 Financial datase the Rossmann store sales datase and the Airbnb new user bookings datase The first three datasets have been
Age2 dataset Fursov et al.
2021
Agel dataset (Fursov al.
202
Leaving datasct (Fursov et al
2021
PKDD' 99 Financial dataset (Berka
1999
Rossmann store sales dataset
Solatorio & Dupriez
2023
Airbnb new bookings dataset (Solatorio & Dupriez
2023
18

---

Published as a conference paper at ICLR 2025
10
MLD-TS
Ditfusion
Figure 2: Age2 dataset. Analysis of the influence of the number of diffusion time steps T jointly with the Smax value using the MLD-TS & metric.
Table 9: Dataset statistics.
Age2
Agel
Leaving
PKDD'99 Financial
Rossmann
Airbnb
Large Scale
Bank Data
Total dalasel rows
3,652,757
14,117,905
490,513
1,056,320
68,015
192,596
9,040,010
Total time series
43,289
50,000
5,000
4,500
1,115
10,000
110,000
Training samples
38,961
45,000
4,000
3,600
892
8,000
100,000
Testing samples
4,328
5,000
1,000
900
223
2,000
10,000
Tenas
50
50
50
50
61
50
100
Time series fields
6
8
5
Time series numerical fields
2
2
Time series categorical fields
3
5
Time series date/time fields
1
Parent fields
5
16
Parent numerical fields
O
I
Parent categorical fields
4
8
11
Parent date/time fields
0
1
O
3
previously used in Fursov al. (2021), while the last two datasets have been used in SSllatooo & Dupriez] (2023). Age2, PKDD '99 Financial, Rossmann and Airbub include both parent and time series data, and they can be used for the conditional generation tasks. Conversely, Age/ and Leav- ing do not include the parent table, thus we used them only for the unconditional generation task. Original source, copyright, and license information are available in the links in the footnotes.
In Rossmann and Airbnb, we use the same training and testing splits created in (Solatorio & Dupriez 2023). Specifically, Rossmann, we use 80% of the store data and their associated sales records for training the generator. We use the remaining stores as the testing data (see App. CM for more details on how testing data are split for training the discriminators). Again following (Solatorio & Dupriez 2023). we limit the data used in the experiments from the years 2015-2016 onwards, spanning the Airbub dataset, we consider a random sample of 10,000 users for the experiment. We take 8,000 part of our training data, and we assess the metrics using the 2,000 users in the testing data. We also limit the users considered to those having at most 50 sessions in the data. Regarding Age2, Agel, Leaving and PKDD '99 Financial, datasets, without any data filtering. The only exception is for the PKDD '99 Financial parent table where used the following fields: district id, frequency, city, region, and the account creation date.
months of sales data per store. Moreover,
the entire
Tab. also includes the statistics of Large Seale Bank Data (App. D.2). In our experiments with this dataset, used both its parent table and its time series, without any data filtering.
19

---

Published as a conference paper at ICLR 2025
Table 10: Dptaset-specific hyperparameter values.
Age2
Agel
Leaving
PKDD'99 Financial
Rossmann
Airbmb
Large Scale
Bank Data
TahDiT VAE
Training epochs
50
100
50
2,000
300
Training iterations
175,950
215,100
43,200
49,150
120,000
40,300
378,800
TahDiT
Training epochs
150
2,000
2,000
4,000
180
Denoising network
Training iterations
45,750
52,800
64,000
58,000
28,000
50,400
118,400
SR-TabDiT VAE
Training epochs
1,000
5,000
6,000
20,000
Training iterations
39,000
20,000
6,000
180,000
SR-TabDiT Denoising
Training epochs
1,000
3,000
6,000
3,000
network
Traiing iterations
170,000
54,000
54,000
120,000
Table 11: Dataset-independent hyperparameter values for the VAE.
Hyperparameter
Value
Optimizer
Adam
Learning rate
5e-05
Training dropout
0.1
Batch size
1,024
Model size (parameters)
2M
VAE Encoder Transformer layers
3
VAE Encoder Transformer heads
8
VAE Encoder hidden size
72
VAE Decoder Transformer layers
3
VAE Decoder Transformer heads
VAE Encoder hidden size
72
VAE latent space size (d)
792
Sman
5
Omin
0.05
0.7
patience
5
F IMPLEMENTATION DETAILS
In this section, we provide additional implementation details jointly with the values of the hyperpa- rameters used our experiments. Tob shows the number of training epochs and iterations for both the VAE and the denoising network (TabDiT and SR-Te TabDiT). Since Large Scale Bank Data was used only for a "child gt-cond" task, we did not train a SR-TabDiT with this dataset.
The model hyperparameter values in Tab. and 12 are shared by all the datasets. Moreover, both TabDiT and SR-TabDiT share the same hyperparameter values, both for the VAE and the denoising network.
Tab. I shows the VAE hyperparameters. The encoder & and decoder Dp of the VAE architecture have the same number of layers and the same number of heads, and the same hidden size. The latent space size of the VAE the same as the DiT-based denoising network F hidden size d, in order to have the denoising network work directly in the latent space of the VAE model.
The DiT-based denoising network F& hyperparameters are detailed in Tab. 12 Most of the hyper- parameter values are borrowed by the Dit-B model presented in (Peebles & Xie 2023). We use standard frequency-basee positional embedding, the same as DiT, the only difference being that we have single dimension input (the time series length) rather than a 2D image. The main hyperpa- rameter the denoising network that we change is the number of diffusion steps (T), which needs to be adapted our tabular data time series domain (see App. D.3
20

---

Published as a conference paper at ICLR 2025
Table 12: Dataset-independent hyperparameter values of the denoising network.
Hyperparameter
Value
Optimizer
AdamW frequency-based
Positional encoding
Learning rate
le-04
Dropout
0.1
Batch size
128
Model size (parameters)
140M
DiT depth
12
DiT num heads
12
Hidden size (d)
792
Diffusion steps
200
Pd 8
0.005
4
Table 13: Dataset-specific total training time (measured in hours).
Age2
Agel
Leaving
PKDD'99
Rossmann
Airbnb
Large Scale
Financial
Bank Data
VAE
3h
3h
2h
8h
Denoising network
5h
5h
5h
5h
3h
3h
26h
G COMPUTING RESOURCES
All the experiments presented in this paper have been performed internal compute node com- posed of:
2 CPUs AMD EPYC 7282 16-Core, for a total of 32 physical and 64 logical cores,
256 Gb RAM,
4 GPUs Nvidia RTX A6000, each with 48 Gb of memory each, for a total of 192 Gb.
Table 13 shows the training time for the VAE and the denoising network on each dataset.
H QUALITATIVE RESULTS
H.1 NUMERICAL FIELD REPRESENTATIONS
In this section, we show some qualitative results using the numerical representation methods eval- uated in Tab. and presented detail in App. A Specifically, we use the Airbnb dataset and select the "secs clapsed" attribute. This is the field with the highest variability, with values ranging from to 1.8M, mean of 3.3K and standard deviation of approximately 13K.
In Fig. to 6 we compare to each other the distributions of all evaluated numerical representations. In every plot, the axis is based on logarithmic scale and fixed number of 50 bins for both the real and the generated numerical values. For each bin, in the y axis we show the percentage of tabular rows that contain numerical values that belong to that bin.
In Quantize (Fig. 3). the "secs_elapsed" field values are quantized using equal-size bins of 360 seconds (I hour) and we generate the bin center. As depicted in the figure, this method struggles in representing small numerical values, which are all grouped in a few bins. On the other hand, the Linear transf representation (Fig. tends to under-sample the tails of the distribution. In the last two figures, qualitatively evaluate the Fixed digit seq (Fig. 5) and representation (Fig. (). The corresponding distributions show that our variable-range representation more accurate in reproducing numerical values. As mentioned in App. A believe that of
variable-range decimal
21

---

Published as a conference paper at ICLR 2025
the reasons why the Fixed digit seq representation is sub-optimal is that its longer length leads, average, to a more difficult decoding problem.
30
Tabular row real
Tabular row real generated
generated
25
Nr tabuua row (perc.)
20
Nr tabular row (perc.)
100
100
10
secs, elapsed (log)
10a
10a
102
20
10a
secs, elapsed (log)
Figure 3: Real and generated distributions of the values of the "secs. peesselapsed" attribute using Quantize as the numerical field value represen- tation.
Figure 4: Real and generated distributions of "secs elapsed" using Linear transf.
Tabular
Tabular real
real generated
Nr of tabuler row (perc.)
Nr of tabular row (perc.)
10a
10:
10a
102
10'
10P
10A
10a
10a
10
10a
10a
elapsed (log)
elapsed (log)
Figure 6: Real and generated distributions of "secs elapsed" using variable-range decimal representation.
Figure 5: Real and generated distributions of "secs clapsed" using Fixed digit seq.
H.2 TIME SERIES LENGTH
In Fig. and we show the distribution of the real and the generated time series lengths on Airbnb. In the first plot, use the W/o length pred (Sec.) 5.2) method: at inference time the sequence length randomly sampled using a mono-modal Gaussian distribution fitted on the length of the training time series. On the other hand, in Fig. 8 we series length. Fig. / and show that, in the latter case, the time series length distribution is more accurately reproduced.
padding rows (Sec. 4.2) to predict the time
22

---

Published as a conference paper at ICLR 2025
Fig. 9 and I0 show the distributions of the difference between the real and the generated time series length using a "child gt-cond" task. Specifically, given a real parent table u. representing a client of generate time series # using both our full method (Fig. I0p and W/ length pred (Fig. 9). In both cases we compute the difference between the length of the generated series t (Tg) and the length of the real time series & (7x) associated with The shorter the difference, the better the method in predicting the real length. Fig. 9 and I0 show these differences for the two methods. Specifically, Fig. since Tæ sampled from mono-modal Gaussian fitted on the training set (Sec. 4.2). it is independent of u. As a result, the denoising network cannot predict the correct time series length. On the other hand, Fig. [0] shows that, in case of our full-method, the distribution of the difference between and Tz is very close to zero, showing the effectiveness of predicting the series length jointly with its content (Sec.
the Airbnb dataset,
Time serie
Time serie
generated
generated
of time series (perc.)
of time series (perc.)
1.0
20
30
40
50
10
30
50
Time serie longth
Time serie longth
Figure 7: Distribution of the lengths of the real and the generated time series, using W/o length pred method.
Figure Distribution the lengths of the real and the generated time series, using our padding row generation (sec. 4.2)
Nr of time seffes (perc.)
Nr of time series (perc.)
60
40
1
20
-40
220
20
Time serie lnngth difference (tg Tt)
length difference
Figure 9: Distribution of the differences be- tween the real and the generated time series lengths when using Wo length pred.
Figure 10: Distribution the differences be- tween the real and the generated time series lengths when using padding row prediction.
23

---

Published as a conference paper at ICLR 2025
H.3 TIME SERIES EXAMPLES
In this section, we show some examples of real and generated time series to qualitatively evaluate the generation results. We use the "child gt-cond" task the PKDD'99 Financial dataset and on the Airbnb dataset. We also use the unconditional generation task on the Leaving dataset.
first provide an example of real parent table row, used as the conditional ground truth information, then we show the first ten rows of the correspond- ing real time series, and finally the generated one. In these examples, REamer In the unconditional task, we provide the first ten rows of real time series ex- ample (Fig. d time series example generated using TabDiT (Fig. [8), and time series example generated using AR baseline (Fig. 19p.
In the conditional task (Fig. Fig. 16).
compare TabDiT with
The results in Fig. 13 and Fig. 19 show that both Fab and the AR baseline make some errors in generating correctly time-ordered dates in the time series. This may be an important error in real-world applications.
Real parent table
district id
frequency
city
region
year
month
day
61
POPLATEX MESIONE
Trebic
south Moreiia
94
17
Real time series
Year
Month
Day
trans
operation
s.ymmoll
amount trans
balance
PRUEM
VILAO
13910.0
48991.9
VYDAI
FREVOD
3757.0
4513d9
VVDAI
4440.0
56
10
VYDAJ
WYBE
None
30000
3/0949
96
10
2/
VYUNJ
VYBER
None
5,000
37814.
VVDW
WBBR
SLUZBY
14.
3:970.6
PRIEM
URCX
5600.
PRUEM
VKLAD
None
13010.0
40180.6
56
11
12
VYDA/
PREVOD NA UCET
UYER
3757.0
36401.6
Figure 11: PKDD'99 Financial dataset: an example a real time series and its corresponding real parent table row.
Real parent row
district id
frequency
city
region
year
month
day
POPLATEK MESIONE
Trebic
south Moradia
94
17
Generaled Lime series
Month
Day
type
operation
symbol
18
VYaS
20
WOJU
WYBER
None
2500.
16990
96
29
VODIJ
VYSER
SLUZBY
14.6
16750
29
PRUBB
None
UROK
80.5
16750.0
PREVOD
3146.
14410
WSFFI
19000
13
PR) IRM
Via AD
Nona
57180
26530.0
95
31
VY3U
VYEER
SLUZBY
14.6
26080
0
875
Figure 12: PKDD'90 '99 Financial dataset: an example of generated time series conditioned on ground truth parent table row using TabDiT.
24

---

Published as a conference paper at ICLR 2025
Real parent table row
district.id id
frequency
region
year
month
POPLATDX MESIONE
Generaled Lime seriess
Day type trans
operation
symbol
amount
90
11
YYUU
PREVOD NA UCEI
76.8
136/5.2
96
12
16
WDDJ
PREVOD NA UCET
SIPO
4508.0
19001 2
12
WDKJ
VYBER
SLUZBV
1372.
14904A
PRUEM
PREVIID UCTU
DUCHOD
1800
16431.9
PREVCID
12
WDW
VYBOR
None
4527.
15727.0
96
12
VYBER
S11078
172" 0
11981 1
WDBJ
PREVOD NA LCFT
SIO
7:6
66755.1
WDKU
VYBER
19241.6
Figure 13: PKDD'99 '99 Financial dataset: example of generated time series conditioned ground truth parent table row using REaLTabFormer. The sequence of the dates not chronologi- cally ordered.
Heal parent table row
Compatitioniatance
PramolSinceWWek
CompatiiienOeeeSneelle
CompettionnparnnneMMenn PramozSinceller
StoreType
Assortment
Promolnterval
Promož
12003
40.
20120
10.0
20140
AMPA
Real time
Open
Promo
StateHoliday
SchoolHoliday DayORWeck
647.)
696E
29
(59.)
(412
28
753.0
83280
27
50560
5050 53572
23
2010 4033
27
Figure 14: Airbnb dataset: example of real time series and a real parent table row.
Real parent table row
Dantetiiinniiaanee
Prone2SinsoWekk
CompetlionOpuniissalae CompetSSonopniinnaMMott
PromozSinceleer
StoreType
Awsrtment
Promolnterval
Promo2
1000
2014
Generated time series
Open
StateHelddyy
SchcolHoliday
DayOlWeck
Customerr
Sales
day
503,
585.0
505.0 50940
7
1
0
5710 67500
7
598. 79000
<710
4020
4000 42480
2718 37220
Figure 15: Airbnb dataset: an example of a generated time series conditioned on ground truth parent table row using TabDiT.
25

---

Published as a conference paper at ICLR 2025
Real parent table row
CompetlionOsaans
Wook
Compatliondpordinsseee
Compalloon@ondinnallnnt
PromsoSirnalier
StuneType
Awortment
Promslntared
Promož
005
7012
100
2014
knnpphhiOrt
Generated time series
Open
Promo
StateHolldyy
SchooHoolday
DayOfWWekk
day
484
295
8110
7
28
291
5856
1134
9011
984
4010
7
28
688
4380
Figure 16: Airbnb dataset: an example of a generated time series conditioned on ground truth parent table row using REaLTabFormer.
year
day
type
cx.eagoyyry
currency
Syp=1
POS
2017
10
4
59
42
typo1
POS
010
2989
1300.00
2017
11
0
0
+yp=1
ROS
810
9411
100 (0)
2017
12
0
0
Sype1
POS
810
9411
312.00
201/
12
Type1
PCS
N0226
2017
Oypel
PO6
184.00
Typet
WOLJARMLHAAINEK
2017
11
0
Opel1
POS
810
5601
4027.00
2017
11
typo1
ROS
810
3681
496.00
2017
11
Oypt
ios
010
7022
990.00
Figure 17: Leaving (unconditional task): an example real time series.
year
day
hour
second
channel type
catagory
currency
amount
2017
20
0
tyas?
PO5
810
5411
756,0
2017
4
22
0
typs2
POS
810
8999
16030
2017
22
typs2
5912
5030
2017
tyne?
M230
2017
5
tyde2
749.0
2017
5
12
14
43
type2
WD ATM RDS
810
6011
10030
2017
5
21
0
Iyuw2
POS
810
5411
10680
2017
WOJAAMM_RDS
2017
type2
WD_ATM_OR IER
2017
5
22
typo2
010 5111
857.1
Figure 18: Leaving (unconditional task): an example of a generated time series using TabDiT.
year
day
channel.tyye trx category
currency
2017
2
10
Iypet
WD_ATMLRRS
010
6011
40000.00
2017
3
20
0
bypet
POS
010 5331
17860.00
2017
4
typet
POS
5954
33640.00
typet
2017
4
9
41
typet
POS
810
5014
490.00
2017
5
3
0
typet
POS
810
5942
102.00
2017
5
typot
5411
4990U00
5
tynet
2017
5
111
23
6
typet WD AIM RDS
810
$912
4564100
2017
6
15
17
29
typet
WT ATM ROS
810
3381
0.35
Figure 19: Leaving (unconditional task): an example of a generated time series using AR baseline. The sequence of the dates not chronologically ordered.
26


Reviews:
[{"review_id": "9DPNzfdRh4", "reviewer": "Reviewer_Eh8R", "summary": "This paper proposes a system based on Diffusion Transformer to generate tabular data series. Particularly the difficulties of handling heterogeneity and variable length of tabular data are tackled. The paper makes a couple of proposals such as 1) transformer denoising combined with latent encoding of VAE, 2) Autoregressive VAE decoder, 3) several data encoding techniques. Experimental results are presented with multiple datasets using proper metrics for the tasks.", "questions": "- In VAE decoder, one naïve architecture is to feed z instead of [Start] without using the attention from z. Would it work at all, and is there ablation study to compare this and the proposed approach?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Each proposal is clearly described and well motivated to solve the targeted issue. Experiments show that the proposed system outperforms the baseline methods in a large margin across different datasets for both unconditional and conditional generation tasks.", "weaknesses": "There is a minor question around VAE encoder/decoder which are listed in Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DPY3oPGrMr", "reviewer": "Reviewer_61wY", "summary": "The paper introduces a novel method for generating time series of tabular data using DiTs. The authors highlight the challenges in this domain, such as the heterogeneity of tabular data and the variable length of time series. They propose the TabDiT framework, which extends the LDM paradigm to handle the complexities of tabular data. The method involves decomposing the representation of time series into independent tabular rows compressed by a VAE and combined by a Transformer-based denoising network. Through extensive experiments on six datasets, the paper demonstrates the superiority of TabDiT over existing methods in both conditional and unconditional generation tasks.", "questions": "- How does the proposed method scale with increasing dataset size and complexity? Have you tested the method on larger datasets, and if so, what were the results?\n\n- The paper includes several hyperparameters, such as the number of digits for numerical representation and the length of the latent sequence. How sensitive is the performance to these hyperparameters, and how were they chosen?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Utilizing DiT for tabular data time series generation seems an innovative contribution to the field, which extends DiT to multi-modal data distributions that could pave the way for future research.\n\n- The authors conduct extensive experiments on six diverse datasets, covering a wide range of application scenarios. This includes both conditional and unconditional generation tasks, providing a thorough assessment of the method's performance. The ablation study also strengthens the credibility of the paper results.\n\n- The proposed method is methodologically sound, with detailed explanations of the VAE encoder/decoder, the denoising network, and the variable-length sequence generation process. The introduction of a variable-range decimal representation for numerical values and an autoregressive VAE decoder are novel and improve the quality and consistency of the generated data.", "weaknesses": "- While the paper introduces new evaluation metrics for tabular data time series generation, these metrics may not be universally accepted or standardized. The authors should provide a more in-depth comparison with existing metrics and justify their choices. \n\n- Some parts of the methodology, particularly the training process and parameter settings, could benefit from more detailed explanations. Providing clear guidelines and examples would help in better understanding and implementing the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nbjwbOZhC", "reviewer": "Reviewer_PrJn", "summary": "This work introduces a Diffusion Transformers (DiTs) based approach for generating time series of tabular data, addressing the challenges of data heterogeneity and variable sequence lengths. Inspired by the success of DiTs in image and video generation, the framework is extended for tabular data series. Extensive experiments on six datasets demonstrate that the proposed method significantly outperforms previous approaches, with the code to be made public upon acceptance of the article.", "questions": "See weaknesses section", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper introduces a novel Diffusion Transformers (DiT)-like approach combined with an Auto-Regressive (AR) Variational Autoencoder (VAE) decoder and explicit padding prediction mechanisms. This design effectively addresses the complex challenges associated with generating time series of tabular data, such as maintaining intra-row statistical dependencies and managing variable-length sequences. This comprehensive framework ensures that the generated data preserves the intricate relationships within each row and adapts to varying sequence lengths, which are critical for accurate and realistic data generation.\n\nThe authors propose a new metric specifically tailored for the evaluation of tabular data time series generation. This metric provides a standardized and objective means to assess the quality and performance of generated data, facilitating more reliable comparisons across different methods. By introducing this metric, the paper sets a benchmark for future research in this domain, promoting the development of more robust and effective data generation techniques.\n\nThe proposed method is rigorously evaluated using a diverse set of public datasets, demonstrating its versatility and robustness across various types of tabular data. This extensive experimental validation highlights the method's superior performance compared to existing approaches, showcasing its applicability to a wide range of real-world scenarios. The thorough evaluation not only underscores the effectiveness of the approach but also provides a solid foundation for its adoption and further development in the field.", "weaknesses": "While the paper introduces numerous model modifications aimed at addressing the challenges of tabular data time series generation, it lacks ablation experiments to isolate and evaluate the impact of each modification. Ablation studies are crucial for understanding the contribution of individual components to the overall performance of the model. Without these experiments, it is difficult to determine which modifications are most effective and how they interact to improve the model's capabilities.\n\nThe paper does not include any case studies that showcase the generated tabular data time series. Given that the primary focus of the research is on generating realistic and useful time series data, it is surprising that the paper does not provide examples to illustrate the differences between the outputs of various models. Including case studies would provide valuable insights into the practical implications of the proposed method and allow readers to visually assess the quality and realism of the generated data.\n\nThe paper does not present experiments that demonstrate the practical utility of the generated time series tabular data. It is important to show how the generated data can be used in real-world applications, such as improving predictive modeling, anomaly detection, or decision-making processes. Without such experiments, it is challenging to gauge the practical relevance and impact of the proposed method. Demonstrating the usefulness of the generated data in specific applications would significantly strengthen the paper's contributions and highlight its value to practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)