Title: ClavaDDPM: Multi-relational Data Synthesis with Cluster-guided Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: diffusion models

synthesis

tabular data

generative models
Abstract: Recent research in tabular data synthesis has focused on single tables, whereas real-world applications often involve complex data with tens or hundreds of interconnected tables. Previous approaches to synthesizing multi-relational (multi-table) data fall short in two key aspects: scalability for larger datasets and capturing long-range dependencies, such as correlations between attributes spread across different tables. Inspired by the success of diffusion models in tabular data modeling, we introduce 
 \textbf{C}luster \textbf{La}tent \textbf{Va}riable guided \textbf{D}enoising \textbf{D}iffusion \textbf{P}robabilistic \textbf{M}odels (ClavaDDPM). This novel approach leverages clustering labels as intermediaries to model relationships between tables, specifically focusing on foreign key constraints. ClavaDDPM leverages the robust generation capabilities of diffusion models while incorporating efficient algorithms to propagate the learned latent variables across tables. This enables ClavaDDPM to capture long-range dependencies effectively. 
 Extensive evaluations on multi-table datasets of varying sizes show that ClavaDDPM significantly outperforms existing methods for these long-range dependencies while remaining competitive on utility metrics for single-table data.

ClavaDDPM: Multi-relational Data Synthesis with Cluster-guided Diffusion Models

Abstract

Recent research in tabular data synthesis has focused on single tables, whereas realworld applications often involve complex data with tens or hundreds of interconnected tables. Previous approaches to synthesizing multi-relational (multi-table) 2 data fall short in two key aspects: scalability for larger datasets and capturing long-range dependencies, such as correlations between attributes spread across different tables. Inspired by the success of diffusion models in tabular data modeling, we introduce C luster La tent Va riable guided D enoising D iffusion P robabilistic M odels (ClavaDDPM). This novel approach leverages clustering labels as intermediaries to model relationships between tables, specifically focusing on foreign key constraints. ClavaDDPM leverages the robust generation capabilities of diffusion models while incorporating efficient algorithms to propagate the learned latent variables across tables. This enables ClavaDDPM to capture long-range dependencies effectively. Extensive evaluations on multi-table datasets of varying sizes show that ClavaDDPM significantly outperforms existing methods for these long-range dependencies while remaining competitive on utility metrics for single-table data.

1 Introduction

Motivation. Synthetic data has attracted significant interest for its ability to tackle key challenges in accessing high-quality training datasets. These challenges include: i) data scarcity [14, 53], ii) privacy [2, 19], and iii) bias and fairness [46]. The interest in synthetic data has extended to various commercial settings as well, notably in healthcare [18] and finance [36] sectors. The synthesis of tabular data, among all data modalities, is a critical task with approximately 79% of data scientists working with it on a daily basis [45]. While the literature on tabular data synthesis has predominantly focused on single table (relation) data, datasets in real-world scenarios often comprise multiple interconnected tables and raise new challenges to traditional single-table learning [38, 3, 12, 22]. These challenges have even enforced a join-as-one approach [15, 17], where the multi relations are first joined as a single table. However, with more than a couple of relations (let alone tens or hundreds of them as in the finance sector) this approach is neither desirable nor feasible.

Challenges. Synthetic Data Vault [35] and PrivLava [5] are recent efforts to synthesize multirelational data using hierarchical and marginal-based approaches. These methods exhibit significant

limitations in processing speed and scalability, both with respect to the number of tables and the domain size of table attributes, and they often lack robustness in capturing intricate dependencies. Alternatively, diffusion models have emerged as powerful tools for data synthesis, demonstrating remarkable success in various domains [37]. These models are particularly noted for their strong capabilities in controlled generation. Despite their potential, the application of diffusion models to tabular data synthesis has been limited to unconditional models [25, 50, 28, 24], leaving a gap in effectively addressing the multi-table synthesis problem.

Solution. To address these challenges, we introduce ClavaDDPM (Cluster Latent Variable guided Denoising Diffusion Probabilistic Models). Our novel approach leverages the controlled generation capabilities of diffusion models by utilizing clustering labels as intermediaries to model the relationships between tables, focusing on the foreign-key constraints between parent and child tables. This integration of classifier guidance within the diffusion framework allows ClavaDDPM to effectively capture complex multi-table dependencies, offering a significant advancement over existing methods.

Contributions. In this work, we: 1) provide a complete formulation of the multi-relational modeling process, as well as the essential underlying assumptions being made, 2) propose an efficient framework to generate multi-relational data that preserves long-range dependencies between tables, 3) propose relationship-aware clustering as a proxy for modeling parent-child constraints, and apply the controlled generation capabilities of diffusion models to tabular data synthesis, 4) apply an approximate nearest neighbor search-based matching technique, as a universal solution to the multi-parent relational synthesis problem for a child table with multiple parents, 5) establish a comprehensive multi-relational benchmark, and propose long-range dependency as a new metric to measure synthetic data quality specific to multi-table cases, and 6) show that ClavaDDPM significantly outperforms existing methods for these long-range dependency metrics while remaining competitive on utility metrics for single-table data.

2 Related work

Single-table synthesis models. Bayesian network [48] is a traditional approach for synthetic data generation for tabular data. They represent the joint probability distribution for a set of variables with graphical models. CTGAN [47] is a tabular generator that considers each categorical value as a condition. CTAB-GAN [52] includes mixed data types of continuous and categorical variables. Several studies have explored how GAN-based models can contribute to fairness and bias removal [44, 45]. In privacy, GAN-based solutions boosted with differential privacy have not been as successful as their Baysian-network-based competitors [34, 51]. Recent popular Diffusion Models, [20, 40, 42, 41], offer a different paradigm for generative modeling. TabDDPM [25] utilizes denoising diffusion models, treating numerical and categorical data with two disjoint diffusion processes. STaSy [24] uses score-based generative modeling in its training strategy. CoDi [28] processes continuous and discrete variables separately by two co-evolved diffusion models. Unlike the previous three which perform in data space, TabSyn [50] deploys a transformer-based variational autoencoder and applies latent diffusion models. Privacy and fairness research for diffusion models are currently limited to a few studies in computer vision [26, 11, 16].

Multi-table synthesis models. There have been few proposals for synthetic data generation for multi-relational data. A study proposed this synthesis through graph variational autoencoders [31], the presented evaluation is nevertheless very limited. The Synthetic Data Vault [35] uses the Gaussian copula process to model the parent-child relationship. SDV iterates through each row in the table and performs a conditional primary key lookup in the entire database using the ID of that row, making a set of distributions and covariance matrices for each match. This inhibits an efficient application of SDV to the numerous tables case. PrivLava [5], synthesizes relational data with foreign keys under differential privacy. The key idea of PrivLava is to model the data distribution using graphical models, with latent variables included to capture the inter-relational correlations caused by foreign keys.

3 Background

Multi-relational databases. A multi-relational database R consists of m tables (or relations) ( R 1 , . . . , R m ) . Each table is a collection of rows, which are defined over a sequence of attributes.

One of the attributes, let's consider the first attribute without loss of generality, is the primary key of table R , which serves as the unique identifier for each row in the table. No rows in the same table have repeated values for the primary key attribute. We use Berka database [4] as our running example in this work, as in Figure 1. Note the Account ID , the primary key for the Account table in Berka .

Given a table R j , we say a relation R i has a foreign key constraint with R j , or R i refers to R j , if R i has an attribute known as foreign key that refers to the primary key of R j : for every row r i ∈ R i , there exists a row r j ∈ R j such that r j 's primary key value equals to r i 's foreign key value. For example, the Account ID of the Loan table refers to the primary key of the Account table. If an account row is removed from the Account ID table, so would all the referring rows in the Loan table to this account, for foreign key constraint to hold. Note that the primary key of a table can consist of multiple attributes. In this paper, we focus on the case of a single attribute that is common in practice. Also note that all keys are considered row identifiers and are thus not treated or modeled alongside the actual table attributes in this work.

A multi-relational database under foreign key constraints forms a directed acyclic graph (DAG),

̸

with the tables R being the set of nodes, and E being the set of edges. In addition, for R i referring to R j , we also call this a parent-child relationship , where R j is the parent and R i is the child . We use the maximum depth to denote the number of nodes on the longest path in G . Figure 1 shows the corresponding graph to Berka database and its maximum depth is 4 .

Multi-relational synthesis problem. Given a multi-relational database R = { R 1 , . . . , R m } , we would like to generate a synthetic version ˜ R = { ˜ R 1 , . . . , ˜ R m } that has the same structure and foreign-key constraints as R and preserves attribute correlations within R , including 1) the intercolumn correlations within the same table; 2) the intra-group correlations within the same foreign key group; 3) the inter-table correlations. The first aspect has been well defined, measured, and tackled in the literature of single-table synthesis [52, 25, 50] while the other two aspects are raised due to foreign-key constraints between tables [5]. For instance, in Berka database (Figure 1), the foreign key constraint between the Loan table and the Account table via Account ID adds an important intra-group correlation for the combinations of loans associated with an account and many 1-hop inter-table correlations between columns in the Loan table and the columns in the Account table. Even for the Loan table and the Demographic table that are indirectly constrained by foreign keys, their columns are correlated as well, e.g., how is the average salary in a district related to the status of loans, an example for 2-hop inter-table correlation.

Classifier-guided DDPM. DDPM [20] uses two Markov chains, a forward chain that perturbs data to noise through a series of Gaussian transitions, and a reverse chain that converts noise back to data with the same number of steps of Gaussian transitions (Equation 2).

Prior work [40] shows that given label y , the conditional reverse process has the form

By approximating log p ϕ ( y | x t ) using Taylor expansion around x t = µ , the conditional reverse process (Equation 3) can be approximated with a perturbed Gaussian transition [10]

where C is a constant and g = ∇ x t log ( P ϕ ( y | x t )) | x t = µ computed from the classifier P ϕ .

4 ClavaDDPM

Here, we elaborate on the training and synthesis process of ClavaDDPM, and each design's rationale.

4.1 Modeling generative process for two-table relational databases

Notations. Consider a database of two tables R = { R 1 , R 2 } , e.g. { Loan, Account } in Berka , where the child table R 1 refers to parent table R 2 . To model the entire database, we first use X and Y as the variables for the child table R 1 and parent table R 2 , respectively (dropping their primary key attributes and indexing their respective row variables starting from one). In this section, we use boldface to represent random variables. e.g. Y ∼ Y , where Y is the data of R 2 , and Y is the random variable Y being sampled from. In addition, we use subscript to represent the parent row some data or random variable refers to. e.g. x j represents the child random variable who refers to parent y j . Refer to Appendix A for a complete list of notations used and the corresponding design choices.

Assumptions. 1) The parent table has no constraints itself. Hence, we can follow previous work on single-table synthesis [13, 25, 47, 50, 52] to make an i.i.d assumption on the rows in the parent table. The parent table Y can be modeled as a list of i.i.d. row variables { y j | j = 1 , . . . , | R 2 | } , where j is the index or the primary key value of the j th row, and each row follows a distribution p ( y ) .

2) The i.i.d assumption does not apply to the child table rows ( x j 's) as they are constrained by their respective parent rows. Consider two loans associated with the same account id; if one's status is in debt ('C'), the other one is likely so too. To capture this dependency, we make a Bayesian modeling assumption that, although child rows associated with the same parent row are not independent, they are conditionally independent of child rows associated with other parent rows, given their respective parent. For example, consider an account table (parent) and a loan table (child). Loans related to the same account (i.e., the same parent) are not independent due to shared account-specific factors. However, loans from different accounts can be considered conditionally independent when accounting for their respective account-level information. Hence, we model X by { g j | j = 1 , . . . , | R 2 | } , where each group g j = { x i j | i = 1 , . . . , ∣ ∣ g j ∣ ∣ } represents a set of child table rows referring to the parent row y j .

3) Without violating the assumptions made above, we further make an i.i.d assumption on ( g j , y j ) , which leads to an approximated distribution for the parent-child tables:

where X = ∪ | R 2 | j =1 g j and g j = { x 1 j , . . . , x | g j | j } . This model allows us to capture the inter-table correlations (the correlation between tuples from different tables) and the intra-group correlations.

Modeling. Despite the simplified formulation with several aforementioned assumptions, learning the distribution p ( g j , y j ) is non-trivial. In particular, ( g j , y j ) cannot be flattened into a matrix form for learning since the set structured attributes in g j , e.g., the size of a group variable g j is not fixed.

A naive solution is to model a conditional distribution of the group given the parent row

Direct modeling of Equation (6) still has the same issue as before for the foreign key group g j , which can take an arbitrary number of child rows. In particular, when modeling g j = f ( x j ) for some function f , there is no trivial structured support for g j if we model for g j using only the attributes or features of the child rows. Furthermore, the conditioning space of the parent row y can be very

large (e.g., Account table has a domain size of more than 11,000), which can lead to poorly learned conditional distribution if we treat y as labels in the classifier-guided DDPM. The original space of y is high-dimensional and noisy and does not guarantee any spatial proximity or smoothness. In the context of deep modeling, this drastically worsens the quality of conditional sampling.

To address these two challenges, we introduce latent random variables c such that g j is independent from y j conditioned on c

With this assumption, we get an indirect modeling of inter-table correlations through c :

Compared to Equation (6), when c is selected to be lying on a low-dimensional, compact manifold, the latent conditional distribution p ( g j | c ) will be easier to model than p ( g j | y j ) .

For each foreign key group g j , we model its size explicitly with a variable s j . By making an assumption that s j is conditionally independent from its child row variables { x 1 j , . . . , x s j j } given the latent random variable c , we essentially defined a generative process for g j : first sample its size s j ∼ s j , then sample g j child row variables. In addition, we make an i.i.d assumption on the child row variables given the latent variable. Hence, we have

Putting all together, we have the final formulation of the generative process for a two-table case:

Unlike our naive method that uses a direct modeling of p ( x j | y j ) , we model p ( x j | c j ) , which greatly reduced the condition space, thus better capturing the inter-table correlation between X and Y . On the other hand, the intra-group correlations are intrinsically addressed, because our modeling of ( g j , y j ) and the corresponding dependency assumptions enforce that two child rows are drawn from the same distribution if and only if they belong to the same foreign key group.

Based on Equation (10), we introduce our generative process for two-table case.

Phase I: Latent learning and table augmentation : (1) Learn latent variable c on the joint space ( X ; Y ) , such that each parent row y j corresponds to a learned latent variable c j . (2) Augment the parent table into T Y = ( Y ; C ) , where C corresponds to the latent variable values c j for each row y j .

Phase II: Training : (3) Train diffusion model p θ ( y, c ) on augmented table T Y and child diffusion model p ϕ ( x ) . (4) Given learned latents and child table, train classifier p ψ ( c | x ) . (5) Estimate the foreign key group size distributions conditioned on latent variables p ( s | c ) .

Phase III: Synthesis : (6) Synthesize the augmented parent table ˜ T Y = ( ˜ Y ; ˜ C ) ∼ p θ ( · , · ) . (7) For each synthesized latent variable ˜ c j ∈ C , sample group size ˜ s j ∼ p ( · | ˜ c j ) . (8) Given ˜ s j , sample each child row within the foreign key group x i j ∼ p ϕ,ψ ( · | ˜ c j ) , where p ϕ,ψ performs classifier guided sampling by perturbing p ϕ with the gradient of p ψ . We denote steps (7) and (8) by ˜ X ∼ p ( ·| C ) .

4.2 Extension to more parent-child constraints

We learn the latent variables between a parent and a child pair in a bottom-up fashion (starting from the leaf nodes in G ) and pass all the latent variable values to the parent table for the next set of latent variables at higher levels. Given a parent-child pair ( Y, X ) , the child table X also

has k leaf node children, Z 1 , . . . , Z k . Let c X,Z i represent the latent variables learned on the joint space ( X ; Z i ) . The augmented table for X is formed by appending all its latent variable values, i.e., T X = ( X ; C X,Z 1 ; . . . ; C X,Z k ) . Then, the latent variable c Y,X is learned on the joint space of ( Y ; T X ) instead of ( Y ; X ) . Therefore, our latent learning process follows a bottom-up topological order, ensuring each child table is already augmented by the time we learn the latent variable to augment its parent.

The training phase and the synthesis phase are similar to the two-table case, by handling the parentchild tables in a top-down topological order using the augmented tables. We detail the end-to-end algorithms for the complex data in Appendix B. However, we would like to highlight a special case when a table X has multiple parents Y 1 , . . . , Y k . During synthesis, we will have k synthetic latent variable ˜ C 1 , . . . , ˜ C k corresponding to the k parents, and thus k copies of synthetic child tables ˜ X 1 ∼ p ( · | ˜ C 1 ) , . . . , ˜ X k ∼ p ( · | ˜ C k ) . Unifying these diverged synthetic tables presents a challenge and we present a universal solution in Section 4.3.3.

Extending the model to include more tables allows for capturing longer-range dependencies, beyond just those between adjacent tables. For example, as shown in Figure 1, the dependency between the Demographic table and the Credit Card table can also be captured and quantified. Further details are provided in Section 5.

4.3 Design choices for ClavaDDPM

We detail how design decisions for ClavaDDPM meet our goals and align with our assumptions.

4.3.1 Relationship-aware clustering

Given the conditional independence between the parent row and its foreign key group (Equation (7)), it is important to model the latent variable c such that it can effectively capture the inter-table correlation within the same foreign key group. In ClavaDDPM, we learn c using Gaussian Mixture Models (GMM) in the weighted joint space of X and Y , denoted as H = ( X ; λ Y ) , where λ is a weight scalar controlling the importance of child and parent tables when being clustered. Concretely, we consider k clusters, and model the distribution of h = ( x ; λ y ) with Gaussian distributed around its corresponding centroid c , i.e., P ( h ) = ∑ k c =1 P ( c ) P ( h | c ) = ∑ k c =1 π c N ( h ; µ c , Σ c ) .

Note that diagonal GMMs are universal approximators, given enough mixtures of Gaussian distributions [49]. Therefore, we can further enforce diagonal covariance, i.e., Σ c = diag ( . . . , σ 2 l , . . . ) , which, being properly optimized, immediately satisfies our assumptions that the foreign key groups are conditionally independent of their parent rows given c . In addition, the family of Gaussian Process Latent Variable Models (GPLVM) [30, 27, 33] has been used as an embedding technique to find low-dimensional manifolds that map to a noisy, high-dimensional space. This satisfies our need to learn a stochastic map between the noisy parent space and a condensed latent space. Thus, we can achieve a better trade-off by sacrificing some information fidelity during this quantization process while making the conditional space better shaped.

However, such clustering in the joint space ( X ; λ Y ) could potentially lead to inconsistency when we create the augmented table T Y = ( Y ; C ) . Though we add a weight λ to the parent rows such that child rows with the same parent rows are likely to be assigned to the same cluster, there is still some chance that they end with different clusters. In particular, for each parent row y j ∈ Y , its child rows are assigned to different clusters. In ClavaDDPM, we impose a majority voting step to find the most popular cluster label in each foreign key group and assign it to the parent row y j . In practice, the voting agree rates tend to be high, and this can be further enforced by assigning a higher weight to the parent table (increasing λ ) during GMM clustering. We evaluate the choice of λ and voting agree rates in our ablation study in Section 5.3.

While alternative latent learning algorithms could potentially be applied, such as TabSyn [50] that demonstrated the utility of latent encoding of tabular data with V AE, this work focuses on demonstrating the effectiveness of a simple diagonal Gaussian Mixture Model (GMM) for ClavaDDPM. Our experiments (detailed in Section 5) reveal that ClavaDDPM with a diagonal GMM achieves state-of-the-art results while maintaining low computational overhead. We leave the exploration of more complex latent learning techniques for future work.

4.3.2 Learning with DDPM

Gaussian diffusion backbone. We consider one of the state-of-the-art diffusion models for single tabular data, TabDDPM [25], as the backbone model. TabDDPM models numerical data with Gaussian diffusion (DDPM [20]), and models categorical data with multinomial diffusion ([21]) with one-hot encoding, and carries out disjoint diffusion processes. However, the modeling of multinomial diffusion suffers significant performance overheads, and poses challenges to guided sampling. Instead, ClavaDDPM uses a single Gaussian diffusion backbone to model both numerical and categorical data in a unified space, where categorical data is mapped to the numerical space through label-encoding. To be specific, for a categorical feature with m distinct values C = { c 1 , . . . , c m } , a label encoding E : C →{ 0 , . . . , m -1 } maps each unique category c i to an assigned unique integer value. For a table row x = [ x num ; · · · ; x cat i ; · · · ] , where x num represents all the numerical features and x cat i represent a categorical feature, we obtain the unified feature by x uni = [ x num ; · · · ; E ( x cat i ) ; · · · ] . Based on this encoding, we learn p θ ( y, c ) on the augmented parent table T Y = ( Y ; C ) through training a Gaussian diffusion model on the unified feature space ( Y uni ; E ( C )) .

Classifier guided synthesis. As defined in Equation (10), we model p ( x i j | c j ) by leveraging classifier-guided sampling of diffusion models, following [10]. In practice, with the sheer power of diffusion models, we jointly model p ( x | c ) for the entire table without distinguishing j . First, we train a Gaussian diffusion model p ϕ on child table row x , with its reverse process modeled with x t ∼ N ( x t +1 ; µ ϕ t +1 , Σ ϕ t +1 ) . Then, we train a classifier that classifies cluster labels based on x . The conditional reverse process can be approximated by x t | c ∼ N ( x t +1 ; µ ϕ t +1 + η Σ ϕ t +1 g ψ t +1 , Σ ϕ t +1 ) , where g ψ t +1 = ∇ x t +1 log ( p ϕ ( c | x t +1 )) and η is a scale parameter controlling the strength of conditioning. One can regard η as a hyper parameter measuring the trade-off between single-table generation quality and inter-table correlations, to be demonstrated by our ablation study in Section 5.3.

4.3.3 Multi-parent dilemma: matching

Consider the case where some child table X has two parent tables Y 1 , Y 2 . Our parent-child synthesis modeling paradigm would lead to two divergent synthetic child tables ˜ X 1 ∼ X | Y 1 , and ˜ X 2 ∼ X | Y 2 Each synthetic table encodes its own parent-child relationship, i.e. the foreign keys. Combining ˜ X 1 and ˜ X 2 so that the synthetic child table contains foreign keys from both parents p 1 and p 2 is non-trivial, and we call it a multi-parent dilemma. One possible approach is to explicitly constrain the model sample space of X | Y 2 to be the synthetic data ˜ X 1 , as used in PrivLava [5]. However, this approach is not applicable to diffusion models that sample from a continuous space.

We provide a universal solution for all generative models. Consider some real data point x with two parent rows y j 1 and y k 2 . Ideally, some synthetic data point ˜ x following the same distribution as real data point x should be sampled from x | y j 1 , y k 2 . This can be approximated by finding the intersection of two conditional distributions x | y 1 and x | y 2 . Specifically, we estimate ˜ x by finding two synthetic data points ˜ x 1 ∈ ˜ X 1 and ˜ x 2 ∈ ˜ X 2 , such that ˜ x 1 ∼ x | y j 1 and ˜ x 2 ∼ x | y k 2 , and the two points are close enough. We reason as follows: although ˜ x 1 was sampled from x | y 1 , as long as it is close enough to some other synthetic data point ˜ x 2 sampled from x | y 2 , then ˜ x 1 will also be within in the high density region of the distribution x | y 2 , indicating a high probability that ˜ x 1 follows x | y 1 , y 2 . Symmetrically, the same reasoning also holds for ˜ x 2 .

Therefore, we can estimate the true sample data point by ˜ x = f (˜ x 1 , ˜ x 2 ) if ˜ x 1 is close to ˜ x 2 , where f can simply be an interpolation between two data points in practice. We call this a matching process between two divergent synthetic tables ˜ X 1 and ˜ X 2 , and this can be done efficiently using approximate nearest neighbor search. Although we call this a "matching", it does not require finding a one-to-one mapping. Note that this estimate can be further improved by resampling ˜ X 1 and ˜ X 2 and estimate ˜ X with more data points rather than just a pair, and the trade-off is a larger computational overhead, and we leave this for future research. Empirically, sampling ˜ X 1 and ˜ X 2 only once is already strong, and an ablation study on the effectiveness of parent matching is in Section 5.3.

5 Evaluation

We evaluate ClavaDDPM's performance in multi-relational data synthesis, using both single-table and multi-tables utility metrics (including the new long-range dependency). We present an end-to-end comparison of ClavaDDPM to the SOTA baselines, followed by an ablation study for ClavaDDPM.

5.1 Experimental setup

Real-world datasets. We experiment with five real-world multi-relational datasets including California [6], Instacart 05 [23], Berka [4], Movie Lens [39, 32], and CCS [32]. These datasets vary in the number of tables, the maximum depth, the number of constraints, and complexity. Among all, Berka , Movie Lens , and CCS exhibits complex multi-parent and multi-children structures. We use Berka in our work for ablation study and model anatomy. Details can be found in Appendix C.1.

Baselines. We adopt two multi-relational synthesis models in literature as our baselines: PrivLava [5] as a representative of state-of-the-art marginal-based methods, and SDV [35] as a statistical method specially designed for multi-relational synthesis. We also introduce two multi-relational synthesis pipelines, SingleT(ST) and Denorm(D), as our additional baselines. SingleT learns and generates each table individually, but it also assigns foreign keys to each synthetic child table accordingly to the real foreign key group size distribution such that the group size information is preserved. Denorm follows the baseline idea that joins table first, but it is hard to join all tables into a single table. Hence, Denorm first applies single-table backbone model to generate the joined table between every parent-child table pair and then split it. For these two pipelines, we use CTGAN [47] and TabDDPM [25] as the single-table backbone models, representing the SOTA tabular synthesis algorithms with GAN-based models and diffusion-based models. The details can be found in Appendix C.2.

Evaluation metrics. We evaluate the quality of the synthetic data using: 1) cardinality to measure the foreign key group size distribution for the intra-group correlations; 2) column-wise density estimation (1-way) to estimate the density of every single column for all tables; 3) pair-wise column correlation ( k -hop) for the correlations of columns from tables at distance k , e.g., 0 -hop refers to columns within the same table and 1 -hop refers to a column and another column from its parent or child table; 4) average 2 -way , which computes the average of all k -hop column-pair correlations, taking into consideration of both short-range ( k = 0) and longer-range ( k > 0) dependencies. For each measure, we report the complement of Kolmogorov-Smirnov (KS) statistic and total variation (TV) distance 3 between the real data and the synthetic data, ranging from 0 (the worst utility) to 1 (the best utility). The reported results are averaged over 3 randomly sampled synthetic data.

We also consider higher-order single-table evaluation metrics for some representative tables as prior work [50]. We include their details and experiemntal results in Appendix D due to space constraints.

All experiments are conducted with an NVIDIA A6000 GPU and 32 CPU cores, with a time limit of 7 days. If an algorithm fails to complete within the time limit, we report TLE (time limit exceeded). Implementation details and hyperparameter specifics are in Appendix C.3.

5.2 End-to-end evaluation

We conducted multi-table synthesis experiments on five multi-table datasets and report the averaged utility with standard deviation for all algorithms in Table 1. First, the evaluation shows that ClavaDDPM has an overall advantage against all the baseline models in terms of correlation modeling, and is surpassing the baselines by larger margins for longer-range dependencies. e.g. in Instacart 05 , our model outperforms the best baseline by 58 . 29% on 2 -hop correlations, and in Berka , our model exceeds the best baseline by 20 . 24% on 3 -hop correlations. For single-column densities and cardinality distributions, ClavaDDPM exhibits a competitive result compared to the state-of-the-art baseline models. We also evaluate ClavaDDPM against baselines on high-order single-table metrics (Appendix D.3), which shows that our model has advantages in preserving data fidelity, generating diverse data, and achieving high machine learning efficacy.

It is worth noting that ClavaDDPM, despite its complexity and capability, is more efficient and robust than some simpler baselines. PrivLava demonstrates strong performance on the California dataset

(the simplest data), but fails to converge on all the other datasets. SDV also tends to fail on complex datasets, and is limited to datasets with at most 5 tables and maximum depth of 2 [9]. Although TabDDPM shares a similar model backbone with ClavaDDPM, its synthesis fails to complete within 7 days on multiple datasets, while ClavaDDPM completes all experiments within 2 days.

5.3 Ablation study

Gaussian diffusion backbone. To decouple the effect of our Gaussian diffusion only backbone with the latent conditioning training paradigm, we also included two models in Table 1: ST-ClavaDDPM and D-ClavaDDPM, which use the Gaussian diffusion model in ClavaDDPM as backbone model, but are trained and synthesized following Single-T and Denorm. Compared to other baselines, ST-ClavaDDPM exhibits superiority in modeling both single column densities and column-pair correlations. ST-ClavaDDPM significantly outperforms its sibling ST-TabDDPM, which proves the effectiveness of using Guassian diffusion for tabular data synthesis solely. On the other hand, ST-ClavaDDPM falls short on longer-range correlations when compared to the full ClavaDDPM model. This observation provides solid evidence to the efficacy of our multi-table training paradigm.

Besides the study of the single-table backbone models, we perform a comprehensive ablation study using Berka (for it has the most complex multi-table structure) on each component of ClavaDDPM and provide empirical tuning suggestions. The full results are in Table 2.

Number of clusters k . We study the necessity of using latent cluster conditioning: (i) no conditioning with k = 1 ; (ii) many clusters with k = 1000 to approximate a direct conditioning on parent rows rather than latent variables. When k = 1 , the quality of long-range correlation degrades drastically. When k = 1000 , we still get reasonably strong performance, which showcases ClavaDDPM's robustness. Compared to the default setting ( k = 20 ), the metrics are lower in all of cardinality distribution, single column densities, and column correlations - proper latent variable learning leads to better results than direct conditioning on parent rows. We also report a new metric avg agree-rate , the average of all per-table agree rates for the labels within each foreign key group (Section 4.3.1). This measure highly depends on k , but a higher rate does not always imply a better performance (e.g., k=1 achieves perfect rates). We provide more insights on how it varies with the next parameter. We also conducted finer-grained experiments to examine the effect of k on model performance, as shown in Appendix D.2.

Parent scale λ . Varying the parent scale parameter λ changes the agree-rates as shown in Table 2, but the downstream model performance does not vary too much. This result indicates that the relation-aware clustering process is robust against such factors, and the GMM model is capable of capturing nuances in data patterns. The detailed discussion is in Appendix D.1.

Classifier gradient scale η . This parameter controls the magnitude of classifier gradients when performing guided sampling, and thus the trade-off between the sample quality and conditional sampling accuracy. Table 2 shows that, when η = 0 , which essentially disables classifier conditioning, the single column densities (1-way) are slightly higher than the default setting. However, it falls short in capturing long-range correlations. When η = 2 , the conditioning is emphasized with a higher weight, which significantly improves the modeling of multi-hop correlations compared to η = 0 case.

Comparing with no matching for multi-parent dilemma. Berka (Figure 1) suffers from the multiparent dilemma , where the Disposition table has two parent tables, Account and Client . Our abalation study switch the table matching technique to a naive merging of two synthetic table (Appendix C.2). The experiment result show that even if trained with the same hyper parameters and model structures, ClavaDDPM with matching is significantly stronger than the no-matching setup in terms of long-range correlations, with 3 -hop correlations 16 . 70% higher than no-matching.

6 Conclusion

We proposed ClavaDDPM as a solution to the intricate problem of synthetic data generation for multi-relational data. ClavaDDPM utilizes clustering on a child table to learn the latent variable that connects the table to its parents, then feeding them to the diffusion models to synthesis the tables. We presented ClavaDDPM's seamless extension to multiple parents and children cases, and established a comprehensive multi-relational benchmark for a through evaluation - introducing a new holistic multi-table metric long-range dependency . We demonstrated ClavaDDPM not only competes closely with the existing work on single-table synthesis metrics, but also it outperforms them in ranged (inter-table) dependencies. We deliberately selected the more complex public databases to exhibit ClavaDDPM's scalability, and introduce it as a confident candidate for a broader impact in industry.

We focused on foreign key constraints in this work, and made the assumption that child rows are conditionally independent given corresponding parent rows. This brings three natural follow-up research directions: i) extension to the scenarios where this prior information is not available and these relationships need to be discovered first[29], ii) further relaxing the assumptions, and iii) inspecting multi-relational data synthesis with other integrity constraints (e.g, denial constraints[15], general assertions for business rules). Furthermore, we evaluated ClavaDDPM's privacy with the common (in tabular data literature) DCR metric. Nonetheless, we think it is worthwhile to: i) evaluate the resiliency of ClavaDDPM against stronger privacy attacks[43], and ii) investigate the efficacy of boosting ClavaDDPM with privacy guarantees such as differential privacy. Similarly, the impacts of our design on fairness and bias removal, as another motivating pillar in synthetic data generation, is well worth exploring as future work. We believe the thorough multi-relational modeling formulation we presented in this work, can serve as a strong foundation to build private and fair solutions upon.

Acknowledgments

This work was supported by NSERC through a Discovery Grant, an alliance grant, the Canada CIFAR AI Chairs program. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We thank the reviewers and program chairs for their detailed comments, which greatly improved our paper.

References

A. Alaa, B. Van Breugel, E. S. Saveliev, and M. van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning , pages 290-306. PMLR, 2022.

S. A. Assefa, D. Dervovic, M. Mahfouz, R. E. Tillman, P. Reddy, and M. Veloso. Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on AI in Finance , pages 1-8, 2020.

A. Atserias, M. Grohe, and D. Marx. Size bounds and query plans for relational joins. SIAM Journal on Computing , 42(4):1737-1767, 2013.

P. Berka et al. Guide to the financial data set. PKDD2000 discovery challenge , 2000.

K. Cai, X. Xiao, and G. Cormode. Privlava: synthesizing relational data with foreign keys under differential privacy. Proceedings of the ACM on Management of Data , 1(2):1-25, 2023.

M. Center. Integrated public use microdata series, international: Version 7.3 [data set]. minneapolis, mn: Ipums, 2020.

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321-357, 2002.

T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pages 785-794, 2016.

DataCebo. Hmasynthesizer - synthetic data vault, 2023. Accessed on: May 20, 2024.

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

T. Dockhorn, T. Cao, A. Vahdat, and K. Kreis. Differentially Private Diffusion Models. Transactions on Machine Learning Research , 2023.

W. Dong and K. Yi. Residual sensitivity for differentially private multi-way joins. pages 432-444, 06 2021.

C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

J. Fonseca and F. Bacao. Tabular and latent space synthetic data generation: a literature review. Journal of Big Data , 10(1):115, 2023.

C. Ge, S. Mohapatra, X. He, and I. F. Ilyas. Kamino: constraint-aware differentially private data synthesis. Proc. VLDB Endow. , 14(10):1886-1899, jun 2021.

S. Ghalebikesabi, L. Berrada, S. Gowal, I. Ktena, R. Stanforth, J. Hayes, S. De, S. L. Smith, O. Wiles, and B. Balle. Differentially private diffusion models generate useful synthetic images. ArXiv , abs/2302.13861, 2023.

B. Ghazi, X. Hu, R. Kumar, and P. Manurangsi. Differentially private data release over multiple tables. In Proceedings of the 42nd ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems , PODS '23, page 207-219, New York, NY, USA, 2023. Association for Computing Machinery.

A. Gonzales, G. Guruswamy, and S. R. Smith. Synthetic data in health care: A narrative review. PLOS Digital Health , 2(1):1-16, 01 2023.

M. Hernandez, G. Epelde, A. Alberdi, R. Cilla, and D. Rankin. Synthetic data generation for tabular health records: A systematic review. Neurocomputing , 493:28-45, 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems , 34:12454-12465, 2021.

X. Hu, S. Sintos, J. Gao, P. K. Agarwal, and J. Yang. Computing complex temporal join queries efficiently. In Proceedings of the 2022 International Conference on Management of Data , SIGMOD '22, page 2076-2090, New York, NY, USA, 2022. Association for Computing Machinery.

jeremy stanley, M. Risdal, sharathrao, and W. Cukierski. Instacart market basket analysis, 2017.

J. Kim, C. Lee, and N. Park. Stasy: Score-based tabular data synthesis. arXiv preprint arXiv:2210.04018 , 2022.

A. Kotelnikov, D. Baranchuk, I. Rubachev, and A. Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning , pages 17564-17579. PMLR, 2023.

I. Ktena, O. Wiles, I. Albuquerque, S.-A. Rebuffi, R. Tanno, A. G. Roy, S. Azizi, D. Belgrave, P. Kohli, T. Cemgil, A. Karthikesalingam, and S. Gowal. Generative models improve fairness of medical classifiers under distribution shifts. Nature Medicine , 30(4):1166-1173, apr 2024.

N. Lawrence and A. Hyvärinen. Probabilistic non-linear principal component analysis with gaussian process latent variable models. Journal of machine learning research , 6(11), 2005.

C. Lee, J. Kim, and N. Park. Codi: co-evolving contrastive diffusion models for mixed-type tabular synthesis. ICML'23. JMLR.org, 2023.

F. Li, B. Wu, K. Yi, and Z. Zhao. Wander join: Online aggregation via random walks. SIGMOD '16, page 615-629, New York, NY, USA, 2016. Association for Computing Machinery.

P. Li and S. Chen. A review on gaussian process latent variable models. CAAI Transactions on Intelligence Technology , 1(4):366-376, 2016.

C. A. Mami, A. Coser, E. Medvet, A. T. Boudewijn, M. Volpe, M. Whitworth, B. Svara, G. Sgroi, D. Panfilo, and S. Saccani. Generating realistic synthetic relational data through graph variational autoencoders. arXiv preprint arXiv:2211.16889 , 2022.

J. Motl and O. Schulte. The ctu prague relational learning repository. arXiv preprint arXiv:1511.03086 , 2015.

H. Nickisch and C. E. Rasmussen. Gaussian mixture modeling with gaussian process latent variable models. In Joint Pattern Recognition Symposium , pages 272-282. Springer, 2010.

NIST. 2018 differential privacy synthetic data challenge, 2018. Accessed: 2024-05-17.

N. Patki, R. Wedge, and K. Veeramachaneni. The synthetic data vault. In 2016 IEEE international conference on data science and advanced analytics (DSAA) , pages 399-410. IEEE, 2016.

V. K. Potluru, D. Borrajo, A. Coletta, N. Dalmasso, Y. El-Laham, E. Fons, M. Ghassemi, S. Gopalakrishnan, V. Gosai, E. Kreaˇ ci´ c, G. Mani, S. Obitayo, D. Paramanand, N. Raman, M. Solonin, S. Sood, S. Vyetrenko, H. Zhu, M. Veloso, and T. Balch. Synthetic data applications in finance. arXiv preprint arXiv:2401.00081 , 2024.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

O. Schulte and Z. Qian. Factorbase: multi-relational structure learning with sql all the way. In International Journal of Data Science and Analytics , Int J Data Sci Anal 7, page 289-309. Springer International Publishing AG, 2019.

O. Schulte, Z. Qian, A. E. Kirkpatrick, X. Yin, and Y . Sun. Fast learning of relational dependency networks. Machine Learning , 103:377-406, 2016.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

T. Stadler, B. Oprisanu, and C. Troncoso. Synthetic data - anonymisation groundhog day. In 31st USENIX Security Symposium (USENIX Security 22) , pages 1451-1468, Boston, MA, Aug. 2022. USENIX Association.

B. van Breugel, T. Kyono, J. Berrevoets, and M. van der Schaar. Decaf: Generating fair synthetic data using causally-aware generative networks. In Advances in Neural Information Processing Systems , volume 34, pages 22221-22233. Curran Associates, Inc., 2021.

B. van Breugel, N. Seedat, F. Imrie, and M. van der Schaar. Can you rely on your model evaluation? improving model evaluation with synthetic test data. In Advances in Neural Information Processing Systems , 2023.

B. van Breugel and M. van der Schaar. Beyond privacy: Navigating the opportunities and challenges of synthetic data. arXiv preprint arXiv:2304.03722 , 2023.

L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni. Modeling tabular data using conditional gan. Advances in neural information processing systems , 32, 2019.

J. Young, P. Graham, and R. Penny. Using bayesian networks to create synthetic data. Journal of Official Statistics , 25(4):549-567, Dec 2009.

R. Zemel, R. Urtasun, and S. Fidler. Mixture of gaussians and em. https://www.cs.toronto. edu/~urtasun/courses/CSC411_Fall16/13_mog.pdf .

H. Zhang, J. Zhang, B. Srinivasan, Z. Shen, X. Qin, C. Faloutsos, H. Rangwala, and G. Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. In The twelfth International Conference on Learning Representations , 2024.

Z. Zhang, T. Wang, N. Li, J. Honorio, M. Backes, S. He, J. Chen, and Y. Zhang. PrivSyn: Differentially private data synthesis. In 30th USENIX Security Symposium (USENIX Security 21) , pages 929-946. USENIX Association, Aug. 2021.

Z. Zhao, A. Kunar, R. Birke, and L. Y. Chen. Ctab-gan: Effective table data synthesizing. In Asian Conference on Machine Learning , pages 97-112. PMLR, 2021.

S. Zheng and N. Charoenphakdee. Diffusion models for missing value imputation in tabular data. arXiv preprint arXiv:2210.17128 , 2022.

A Notation Summary

We use boldface to represent random variables. e.g. Y ∼ Y , where Y is the data of R 2 , and Y is the random variable Y being sampled from. In addition, we use subscript to represent the parent row some data or random variable refers to. e.g., x j represents the child random variable that refers to parent y j . The important notations used in the paper are summarized in Table 3.

B Algorithm Details

B.1 Diagram for two-table relational databases

Figure 2 summarizes the generative process for two-table cases.

Account(child)

Relation-Aware

3

Diffusion

Account

ID

1

ID

1

Date

Frequency

x

Clustering

(y,cy)。(

(y,Cy)

Parentrowand

C

latentvalue

Z

2

10

2

synthesis

2

15

c2

C2

Augmented

Tabtes

x

C

XT

Conditional

8

x

2

childrow

Demographic (parent)

Cluster

p(clx)

synthesis

District

ID

District

name

crimes

#of

C

Classifier

gradient

1

1.0

5

2

Conditionat

p(slc)

Group size

Conditionalgroupsizesampling

B.2 End-to-end algorithms for more tables

We detail the end-to-end algorithms for the three phases of ClavaDDPM, including (i) latent learning and table augmentation, (ii) training, and (iii) synthesis.

Latent learning and table augmentation. As shown in Algorithm 1, given a database R = { R 1 , . . . , R m } and foreign key constraint graph G , we learn the set of latent variables C i,j for every pair of parent-child ( R i → R j ) ∈ G . E and augment all the latent variables to the parent table and the child table, denoted by T j and T ′ i , respectively. We initialize each augment table with its original table (line 1). This algorithm follows a bottom-up topological order starting from the leaf child with its parent (line 2), ensuring each child table is already augmented by the time we learn the latent variable to augment its parent. For each parent-child pair R i → R j , we join T i (not T ′ i ) with R j into

a single table ( X ; Y ) (line 3) and then run the clustering algorithm using GMM and maximum voting described in Section 4.3.1. We append the corresponding clustering labels C i,j to the augmented parent table T j and augmented child table T i , respectively.

Algorithm 1 ClavaDDPM: Latent learning and table augmentation. Input: tables R = { R 1 , . . . , R m } , foreign key constraint graph G Output: latent variables { C i,j | ( R i → R j ) ∈ G . E} , augmented parent tables { T 1 , . . . , T m } , augmented child tables { T ′ 1 , . . . , T ′ m } 1: Initialize augmented tables { T 1 , . . . , T m } ← R , { T ′ 1 , . . . , T ′ m } ← R 2: for ( R i → R j ) in bottom-up topological order of G do 3: Join parent and augmented child ( X ; Y ) ← ( T i , R j ) 4: C i,j ← Clustering ( X ; Y ) ▷ Relationship-aware clustering in Section 4.3.1 5: Augment parent T j ← ( T j ; C i,j ) 6: Augment child T ′ i ← ( T ′ i ; C i,j ) . 7: end for

Training. As shown in Algorithm 2, the training phase takes in the augmented parent tables { T 1 , . . . , T m } and the foreign key constraint graph G . For each augmented table T j , we train a diffusion model p T j (lines 2-4). Then, for each parent-child pair R i → R j (lines 5-7), we train a child classifier p ϕ ( c i,j | x ) with R i 's child augment table T ′ i , where the latent column C i,j is used as labels, and all remaining columns are used as training data (including the augmented latent columns corresponding to R i 's children). Using the same table, we also estimate the foreign key group size distribution conditioned on the latent variable p ( s | c i,j ) .

Algorithm 2 ClavaDDPM: Training Input: augmented parent tables { T 1 , . . . , T m } , augmented child tables { T ′ 1 , . . . , T ′ m } , foreign key constraint graph G Output: diffusion models D , classifiers C , group size distributions S 1: Initialize D , C , S ← ∅ 2: for R j in G . R do 3: Train p T j with T j , and add to D 4: end for 5: for ( R i → R j ) in topological order of G do 6: Learn classifier p ϕ ( c i,j | x ) and p ( s | c i,j ) using with T ′ i (ignoring irrelevant latent columns) and add to C and S respectively 7: end for

Synthesis. Algorithm 3 takes in learned diffusion models D , classifiers C , group size distributions S , and the DAG representation of the database G , and outputs the synthetic database ˜ R = { ˜ R 1 , . . . , ˜ R m } . We first initialize the synthetic augmented tables to be empty (line 1). Then, for root augmented tables, since they have no parents to condition on, they can be directly synthesized from their diffusion models (line 2-4). Next, we traverse the database in topological order to synthesize the remaining augmented tables (line 5-16): If we have already synthesized ˜ T i before, which means we encounter the multi-parent dilemma, we just store the old version and continue to generate a new version (line 6-9). For each parent-child relationship R i → R j , we must have already sampled the augmented parent table ˜ T j . This is because we follow the topological order of a DAG, and all root augmented tables have been synthesized as base cases. Therefore, we can obtain the synthetic latent variables ˜ C i,j from the synthetic augmented parent ˜ T j (line 10). Then, we iterate through each synthetic latent value ˜ c i,j and perform a two-step sampling: (1) use the learned group size distribution to conditionally sample a group size ˜ s (line 12); (2) sample ˜ s rows of data conditioned on ˜ c i,j using classifier guided sampling (line 13). We repeat this process until the augmented child table ˜ T i is fully synthesized. We simply obtain synthetic tables from synthetic augmented tables by removing all synthetic latent columns (line 17-19). Finally, for all the encountered multi-parent dilemmas, we follow Section 4.3.3 to match the divergent versions.

C Experimental Details

C.1 Datasets

Here we describe the real-world datasets used in our evaluation in detail. The specifics of datasets are in Table 4.

California : The California dataset is a real-world census database ([6]) on household information. It consists of two tables in the form of a basic parent-child relationship.

Instacart 05 : The Instacart 05 is created by downsampling 5 -percent from the Kaggle competition dataset Instacart ([23]), which is a real-world transaction dataset of instacart orders. This dataset consists of 6 tables in total with a maximum depth of 3 .

Berka : The Berka dataset is a real-world financial transaction dataset ([4]), consisting of 8 tables with a maximum depth of 4 . This will be the main dataset in our work for ablation study and model anatomy.

Movie Lens : The Movie Lens dataset ([39], [32]) consists of 7 tables with a maximum depth of 2 . This dataset exhibits complex multi-parent and multi-children structures.

CCS : The CCS dataset ([32]) is a real-world transactional dataset Czech debit card company. It consists of 5 tables with a maximum depth of 2 , which exhibits complex multi-parent and multichildren patterns.

C.2 Baselines

We adopt two multi-relational synthesis models in literature as our baselines: PrivLava [5] as a representative of state-of-the-art marginal-based methods, and SDV [35] as a statistical method specially designed for multi-relational synthesis. In addition, we introduce two types of multirelational synthesis pipelines, SingleT and Denorm, as our additional baselines. For the additional baselines, we use CTGAN ([47]) and TabDDPM ([25]) as backbone models, representing the stateof-the-art tabular synthesis algorithms with GAN-based models and diffusion-based models. In the following, we describe the high-level ideas of Single-T and Denorm.

Single-T : Given a single-table backbone model, we first learn and synthesize each table individually. Then, for each parent-child table pair ( p, c ) , we assign foreign keys to the synthetic child table ˜ R c by randomly sampling group sizes in the real table R c , which enforces the synthetic group size distributions to be similar to real ones.

Denorm : For each parent-child table pair ( p, c ) , we join the table into R p,c , then use the single-table backbone model to synthesize the joint table ˜ R p,c . Finally, we split ˜ R p,c into two synthetic tables ˜ R p and ˜ R c as follows: (1) Lexicographically sort ˜ R p,c , where the parent columns are prioritized. This guarantees that similar parent records are grouped together. (2) From the real table R c , randomly sample group sizes ˜ g with replacement. Then, for each sampled ˜ g , the consecutive ˜ g rows in ˜ R p,c will be taken as a synthetic foreign key group ˜ g p,c . The child columns part of ˜ g p,c will be assigned the same foreign key and appended to the child synthetic table ˜ R c . Then, we randomly sample a parent row in ˜ g p,c and append to the parent synthetic table ˜ R p . We follow the exact same way as in ClavaDDPM to extend 2 -table Denorm to the entire database.

Random matching: We conduct ablation study by training a ClavaDDPM model with the same setup as the default setting, while instead of performing table matching to handle the multi-table dilemma, it performs a naive merging of two synthetic tables. For the diverged synthetic tables ˜ R D,A and ˜ R D,C , where ˜ R D,A is the Disposition table synthesized conditioned on the Account table, and ˜ R D,C is conditioned on the Client table, we simply keep ˜ R D,A , and randomly assign the ( D,C ) foreign keys from ˜ R D,C to ˜ R D,A .

C.3 Implementation Details

C.3.1 Classifier Training

We use an MLP for classifier with layers 128 , 256 , 512 , 1024 , 512 , 256 , 128 . The output layer size is adapted to the number of clusters k . We use learning rate of 1 e -4 , and optimize with AdamW optimizer, and use cross entropy loss as objective. The overall training paradigm follows [10], where we incorporate timestep information by encoding the timesteps into sinusoidal embeddings, which are then added to the data. For experiments on California , we train the classifier for 10000 iterations, and for all other datasets we train 20000 iterations.

C.3.2 Hyper Parameters

Baseline models. PrivLava was run under a non-private setup by setting privacy budget ϵ = 50 , and the datasets are prepossessed spesifically for PrivLava to have domain sizes less than 200 .

For all models with ClavaDDPM or TabDDPM backbones, we use the same set of hyper parameters. We set diffusion timesteps to 2000 , and use learning rate of 6 e -4 . In terms of model architecture, we use MLP with layer sizes 512 , 1024 , 1024 , 1024 , 1024 , 512 . The model architecture details are following the implementation of TabDDPM [25]. All DDPM-based models are trained 100 , 000 iterations on California dataset, and 200 , 000 on other datasets.

We conducted CTGAN experiments using the interface from SDV library, and follows the default parameters, where the learning rates for the generator and discriminator are both 2 e -4 , and is trained 300 epochs.

PrivLava's code is not publicly available, and we directly followed the authors' settings. Note that PrivLava requires a privacy budget searching process, and ϵ = 50 is the largest working privacy budget according to our experiments, where larger ϵ leads to failure. We consider this as large enough to resemble a non-private setting.

For SDV, we used the default setting of their HMASynthesizer, which by default uses a Gaussian Copula synthesizer.

ClavaDDPM settings. We list the major hyper parameters used by ClavaDDPM for each dataset in Table 5, and we provide an empirical guidance for hyper parameter tuning: it is suggested to use number of clusters k to be at least 20 , and classifier scale η to be in [0 . 5 , 2] . We empirically find ClavaDDPM consistently perform well in such a range across all datasets. Parent scale λ is a less sensitive factor, and λ = 1 is a stable starting point for tuning. In general, ClavaDDPM is robust, with a small hyper parameter space, and there is very little need for tuning.

C.3.3 Metrics

C2ST. The Classifier Two Sample Test trains a logistic regression classifier to distinguish synthetic data from real data. We consider this metric as a high-level reflection of data fidelity.

Machine Learning Efficacy. Different from prior works that evaluate MLE utilities [25, 52, 50], who work on datasets with predefined machine learning tasks, the five real-world multi-relational datasets we use do not come with a designated downstream task. In addition, the prior knowledge about which column will be used for downstream predictions will introduce significant inductive bias to the training process, especially for models capable of performing task-oriented training. To avoid such issue, we evaluate machine learning efficacy on each of the columns. To be specific, each time we select a column as target, and train an XGBoost [8] model on remaining columns. For categorical target columns, we perform regression and evaluate R 2 , and for categorical target columns we perform classification and evaluate F 1 . The overall MLE is measured by average R 2 and average F 1 across all columns.

To evaluate the single-table MLE on synthetic data generated from multi-table synthesis process, instead of performing an independent train-test split on each table, we split by foreign key relationship. e.g. for California dataset, we first perform a random 90% , 10% split on the parent table Household , and then we follow the foreign key constraints to assign corresponding child rows, i.e. Individual s to the corresponding buckets. Note that although this splitting method does not lead to the same train/test ratio on child table, we consider such sampling to be foreign key relationship preserving, which is a more important property in the context of multi-table synthesis.

D Additional Experiments

D.1 Agree rate discussion

As introduced in Section 4.3.1, within each foreign key group, we perform a majority voting to synchronize the assigned cluster label among the group. To measure the consistency of such majority voting process, we introduce the measurement of agree rate , which computes the average ratio of agreeing on the mode within each foreign key group, and the metric avg agree-rate is the average of

all per-table agree rates within a multi-table dataset.

A ( g ) represents the agree rate of some group g , where m g represents the number of records that are assigned the mode cluster within g , and the avg agree-rate

is computed as the average of the agree rates across all groups.

However, our experiment results in Table 2 indicate that the relationship-aware clustering process is robust against such factors, and the GMM model is capable of capturing nuances in data patterns. In our experiments on Berka dataset, ClavaDDPM's clustering process achieves a consistent agree rate around 81% , which is practically high enough given we have 20 clusters. Intuitively, when parent scale approaches infinity, the clustering is performed completely on parent table, which will lead to a perfect agree rate. Also note that a higher agree rate does not always imply a better performance, and the disagreement can potentially come from the intrinsic parent-child relationships. e.g. when child data is intrinsically independent of parent data, it is reasonable to have noisy learned latent variables, leading to low agree rates. However, in such cases the noisy latent variable would not degenerate model performance, because the best strategy will be a direct sampling of child table, rather than conditioning on some enforced prior distribution. In addition, as shown in Table 2, agree rates are highly affected by the number of clusters chosen, and there exists a trade-off between the granularity in clustering and consistency. In an extreme case where we have k = 1 cluster, indicating an infinitely coarse-grained latent learning, it trivially achieves perfect agree rates.

D.2 Selecting Number of Clusters k

We conducted finer-grained experiments to examine the effect of the number of clusters k on model performance, as shown in Figure 3, which offer empirical insights for selecting k . Based on the results, (1) a binary search approach could be used to efficiently find a suitable k , and (2) while it may require more computational resources, opting for a larger k is generally a safe choice.

AVG 2-way (Smoothed)

中

DataPoints

90

89.21± 1.95

88

87.43± 1.34

87.77 ± 0.887.81± 1.70

AVG 2-way (%)

86.873.51

86.40±1.56

86

84.70± 2.27

84

82 -β1.64±1.09

80

10

25

50

100

500

1000

00

D.3 High-order Single-table Evaluation

We also consider higher-order single-table evaluation metrics for the quality of some representative tables as prior work [50]: 1) α -precision and β -recall [1] to measure fidelity and diversity of synthetic data; 2) Machine Learning Efficacy (MLE) to measure the downstream-task utility; 3) Classifier Two Sample Test (C2ST) to measure if the synthetic data is distinguishable from real data by machine learning models.

We evaluated high-order single-table metrics on the California dataset across all baseline models and ClavaDDPM. Following [50], for the evaluation of MLE we perform a 90% , 10% train-test split, where the F 1 and R 2 metrics are evaluated on the 10% holdout set. Note that although PrivLava has an advantage on the California dataset when evaluated with statistical tests (Table 1), ClavaDDPM exhibits competitive, or even stronger performance than PrivLava on higher-order metrics. Especially for MLE, ClavaDDPM surpasses PrivLava by 13 . 07% in terms of average R 2 in Individual table, and also beats PrivLava on average F 1 in both tables. Also notice that the baseline ST-CLAVADDPM dominates in high-order metric evaluations, demonstrating the strength of our Gaussian diffusion-only backbone model.

ClavaDDPM achieves a second-highest β -recall on Household table and ranks first in β -recall on Individual table with large margin, gaining a 7 . 65% advantage over the best baseline without ClavaDDPM backbone. This serves as strong evidence that ClavaDDPM is not only data fidelity preserving, but is also capable of generating highly diverse data.

D.4 Privacy Sanity Check

We follow TabDDPM [25] to perform a privacy sanity check against SMOTE [7], which is an interpolation-based method that generates new data through convex combination of a real data point with its nearest neighbors. We use the median Distance to Closest Record (DCR) [52] to quantify the privacy level. We compare the median DCR, as well as DCR distributions of ClavaDDPM against SMOTE on selected tables.

As shown in table 7, ClavaDDPM although neither specialized in privacy preserving, nor in single table synthesis, it still maintains a reasonable privacy level. The charts 4 demonstrates the distributions of DCR scores, where ClavaDDPM is in blue. The overall distribution is more leaning to the right side, indicating an overall higher DCR distribution.

(a) Household

(b) Individual

(c) Transaction

(d) Order

104

smote

104

clava

smote

104

clava

smote

104

clava

smote

103

101

103

101

102

102

10?

102

101

101

101

101

109

109

100

109

00°0

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0'0

0.2

0.4

0.6

0.8

1.0

00'0

0.05

0.10

0.15

0.20

0.25

000

E Complexity Analysis

Given a multi-relational database G = ( R , E ) , with m tables, n foreign key constraints, and p rows per table. For a p -row table, we denote the time complexity of performing GMM clustering as c GMM ( p ) , training a diffusion model as c diff ( p ) , training a classifier as c class ( p ) , synthesizing as c syn ( p ) , ANN searching as c ANN ( p ) .

Phase 1: latent learning and table augmentation

Note that in practice this phase is dominated by diffusion training, primarily influenced by m .

Phase 3: synthesis

Additional step: matching

Phase 2: training

Note that the runtime in this phase is negligible compared to the earlier phases, particularly with the FAISS implementation in the non-unique matching setup.

Total

the overall runtime is dominated by Phase 2 (training) and Phase 3 (synthesis), with the critical factors being m , n , and p . The model remains robust against the number of clusters in Phase 1, as the impact on runtime is minimal due to the dominance of the later phases.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide empirical evidence in the evaluation section for the claims made in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the conclusion section (second paragraph), we discuss the limitations of our work and our interest in addressing them as future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We list all the assumptions required to derive our generative process. We do not have any proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detail the datasets, baseline algorithms, implementation details of our algorithms, and evaluation metrics in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? [Yes] Justification: We upload supplementary materials including code for reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the basic experimental setting/details, ablation study in the evaluation section, and details in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeat experiments for each configuration three times and report their averaged performance with standard deviations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include computing specifics in the evaluation section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethic https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We carefully go through the NeurIPS Code of Ethic and ensure we follow it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts of our work in the conclusion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We use public datasets for experiments.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We add citations and resources for the datasets and source code used in the evaluation.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We document our new code and and provide references for all code adopted from public resources.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of synthetic data generation in tabular formats and particularly focuses on data generation in multi-relational (multi-table) setups like relational databases. Whereas existing tabular diffusion models work only on single tables, the proposed ClavaDDPM extends the DDPM framework to support many relational tables linked via primary-foreign keys. In the hierarchy of parent-child tables, child tables contain keys from parent tables, and ClavDDPM conditions the generative process of child entries on their parent entries (grouped by the foreign key). To enable this, the model learns latent variables (expressed as GMMs) corresponding to the groups of foreign keys. Capturing the hierarchy of parent-child relations enables learning quite deep, long-range dependencies between k-hop neighbors in the relational schema. Experimentally, ClavaDDPM outperforms existing non-diffusion baselines both in terms of effectiveness and efficiency.\n\nSTRENGTHS:\n**S1.** The problem of synthetic data generation is of growing importance in the era of large, data-hungry models whose quality depends on the quality of training data. Besides, synthetic generation enables upsampling of a small custom dataset (with possible private and sensitive data) for fine-tuning backbone models. In the world of relational databases, most databases consist of multiple linked tables that have to be modeled jointly to capture data dependencies. ClavaDDPM is the first tabular DDPM that supports generation conditioned on multiple tables (organized in several hierarchy levels) and can be used on real-world datasets.\n\n**S2.** The idea of learning latent variables with classifier guidance for grouped keys is rather clever. In addition, the model supports both numerical and categorical data as well as attempts to model the generative process for tables with several parents. Conditioning on several parent tables, the authors propose soft matching of latents (conditioned on each parent separately) via approximate nearest neighbor search.\n\n**S3.** Compelling experimental results - ClavaDDPM was compared against existing non-diffusion multi-table models (PrivLava and Synthetic Dava Vault), the authors also adapted single-table baselines CTGAN and TabDDPM in several versions as baselines. Ablation studies are informative.\n\n**S4.** The paper is well-written and structured, I enjoyed reading the manuscript.\n\nWEAKNESSES:\n**W1.** The idea of long-range dependencies in relational tables has been introduced in the beginning and highlighted in the experiments, but the main part (Section 4) does not go into the details. Perhaps Section 4.2 could expand on long-range modeling mapping to the example hierarchy from Figure 1.\n\n**W2.** No discussion of the limitations, e.g., how long the training took; inference on the datasets takes several days. The checklist mentions “the second paragraph of the Conclusion” but this section in the manuscript has only one paragraph.\n\nQUESTIONS:\n**Q1.** Most of the reported numbers (total variation distance between synthetic and real data) are in the higher 95%+ range which makes it hard to distinguish the quality of new models. Are there any other metrics close to real-world tasks that could be of use? For instance, TabDDPM and others report the quality of CatBoost/XGBoost and other standard tabular ML algorithms trained on the synthetic data.\n\nComments:   \n* Line 206: Typo in LavaDDPM    \n* Captions of Tables 1 and 2 could also mention what the reported metrics are.\n\nLIMITATIONS:\nThere is no discussion of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the new ClavaDDPM approach to address the scalability and long-range dependency challenges in tabular data synthesis. ClavaDDPM uses clustering labels to model inter-table relationships, particularly focusing on foreign key constraints, and employs diffusion models' robust generation capabilities along with efficient algorithms to propagate learned latent variables across tables. Evaluations demonstrate ClavaDDPM's superior performance in capturing long-range dependencies and competitive utility metrics for single-table data.\n\nSTRENGTHS:\n$\\bullet$ The manuscript is commendably well-structured, with a logical flow that effectively guides the reader through the paper. The clarity in writing and the organization of content significantly enhance the reader's comprehension and the overall quality of the presentation.\n\n$\\bullet$ The authors have conducted a thorough experimental validation, which is a significant strength of the paper. The meticulous detailing of experimental procedures, including data handling, model training, and evaluation metrics, greatly contributes to the reproducibility and credibility of the study.\n\n$\\bullet$ The topic of the paper addresses a highly relevant issue in the field, which is a strong point. The research aligns well with current trends and challenges in the domain, ensuring that the findings will be of interest to a broad audience and have potential applications in real-world scenarios.\n\nWEAKNESSES:\n$\\bullet$ The manuscript exhibits a lack of punctuality and standardization in the presentation of mathematical formulas, leading to potential ambiguity in interpretation. Key terms are not consistently defined upon first use, and there is a misuse of symbols that may confuse readers. Additionally, the placement of table names appears to be incorrect in some instances, further detracting from the paper's clarity. It is essential to standardize the notation and ensure consistent definitions and proper placement of table names for better readability.\n\n$\\bullet$ The assumptions made in the model section require more rigorous justification. The authors should provide a more detailed explanation of why these assumptions are reasonable and how they contribute to the validity of the model. This will strengthen the theoretical foundation of the work and enhance the credibility of the proposed approach.\n\n$\\bullet$ The paper would benefit from an analysis of the model's complexity, particularly in the context of large-scale databases. Given the increasing size and complexity of real-world databases, understanding how the proposed model scales and performs under such conditions is crucial. Including a complexity analysis will provide insights into the model's practicality and efficiency. \n\n$\\bullet$ The ablation study regarding the cluster number 'k' lacks necessary granularity. A more fine-grained investigation is needed to understand the impact of varying 'k' on the model's performance and to identify the optimal settings for different scenarios. This will provide a clearer understanding of the model's behavior and its adaptability to various contexts.\n\n$\\bullet$ The section on the multi-relational synthesis problem, currently placed in the appendix, should be relocated to the main body of the paper. This will improve the flow of the paper and make it easier for readers to follow the synthesis process and understand its relevance to the overall work. Integrating this section into the main text will enhance the coherence and comprehensiveness of the paper.\n\nQUESTIONS:\nSee the above weaknesses.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ClavaDDPM to address two key deficiencies in multi-table data generation: scalability for larger datasets and capturing long-range dependencies, such as correlations between attributes across different tables. This approach utilizes cluster labels as intermediaries to model relationships between tables, paying special attention to foreign key constraints. The authors elaborated on the proposed method, made some assumptions, conducted some experiments for evaluation, and performed some analysis about the results.\n\nSTRENGTHS:\n1. The paper proposes an efficient framework to generate multi-relational data that preserves long-range dependencies between tables, and proposes relation-aware clustering for modeling parent-child constraints.\n\n2. The paper applies a matching technique based on approximate nearest neighbor search as a general solution to the problem of multi-parent relation synthesis for a child table with multiple parents.\n\nWEAKNESSES:\n1. I have some concerns about the assumptions made in the paper, which seem to be somewhat inconsistent with the scenarios in real world settings. Please refer to Questions 1 and 2 for details.\n\n2. There may be some clarifications needed in the experimental section. Meanwhile, the analysis of the experimental results does not cover some key points. Please refer to Questions 3-6 for details.\n\n3. There are some problems with the writing and formatting of this paper. Please refer to Questions 7-8 for details. Also, there are some confusing statements in the paper, which hinder the readability of the paper. Please refer to Questions 9-10 for details.\n\nQUESTIONS:\n1. The paper assumes that there are parent-child relationships between different tables and the relationships are known. I However, in real life, the relationships may be very complex and may not be known in advance. In this case, how can ClavaDDPM handle it?\n\n2. The i.i.d. assumption in Section 4.1 seems too strong, because the child rows corresponding to different primary keys may not be independent in real-world scenarios. If this is the case, does the theoretical analysis in the paper still hold?\n\n3. Please clarify why you selected CTGAN over newer or state-of-the-art models like CTAB-GAN, TabSyn, and CoDi mentioned in the related work for your experiments.\n\n4. The reason for the absence of average agree rate in the last two columns of Table 2 remains unclear. Additionally, it is not specified what NA represents. Moreover, there is a lack of clarity regarding the calculation method and formula used to determine the agree rate in this experiment.\n\n5. I feel a bit confused about why TabDDPM converges so slowly on the Instacart05 and Movie Lens datasets. The authors of TabDDPM stated in their work that TabDDPM can converge quickly with a 2080Ti GPU. In addition, I noticed that the Berka dataset seems to be larger than those two datasets. Could you kindly provide an explanation as to why TabDDPM exhibits convergence on this larger dataset, while failing to do so on the two smaller datasets?\n\n6. Are the evaluation metrics used in the experiment widely used in the field or are they proposed by the authors? I haven't seen similar metrics in related works. If they are proposed by the authors, why don't you use metrics from other papers, such as the relative error used in PrivLava [1] or more comprehensive metrics used in TabSyn [2]?\n\n7. There are some typos in the paper: “LavaDDPM” in line 206, “Dnorm” in the caption of Table 1, “The experiment result show…” in line 355.\n\n8. Page 18's experimental results table lacks a caption.\n\n9. In Section 5.2, you referred to additional results in Appendix 5.3, which is missing from the paper. Please provide the mentioned experimental results.\n\n10. In the checklist, you said that you discussed the limitations and future work in the second paragraph of the conclusion section, but there is no relevant discussion in the conclusion of the paper, not even the second paragraph, which is confusing.\n\nReference:\n\n[1]\tK. Cai, X. Xiao, and G. Cormode. Privlava: synthesizing relational data with foreign keys under differential privacy. Proceedings of the ACM on Management of Data, 1(2):1–25, 2023.\n\n[2]\tH. Zhang, J. Zhang, B. Srinivasan, Z. Shen, X. Qin, C. Faloutsos, H. Rangwala, and G. Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. arXiv preprint arXiv:2310.09656, 2023.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the challenges inherent in generating multi-relation tabular data and proposes a novel generation method based on hidden random variables.  The approach analysis correlations between primary and foreign keys between tables by predicting hidden variables associated with these keys within simultaneous tables.  These hidden random variables are inferred using Denoising Diffusion Probabilistic Models (DDPM) grounded in stochastic processes.  The method is elaborated from two-table linkage generation to multi-table linkage generation, with a step-by-step explanation of the approach.  Comprehensive experiments are conducted to demonstrate the practical effectiveness of the proposed method.\n\nSTRENGTHS:\nThis paper highlights the extensive dependencies between different tables in the process of multi-relation tabular data synthesis and introduces an innovative approach by analyzing the relationships between primary keys and foreign keys using hidden variables.  The proposed ClavaDDPM process is well-articulated and has demonstrated promising results across several datasets.  Additionally, the subsequent analysis of hidden variables enhances the credibility of this work.\n\nWEAKNESSES:\nAlthough this work attempts to elaborate on its theoretical framework with substantial content, the notation system and descriptive symbols used do not conform to standard academic mathematics conventions (see sections 3. Background and 4.1 Notations). The marks in equation are not uniform (Eq.2). Furthermore, several key formulas lack thorough proofs (for example, Formula 8). The proof of this key formula would be best placed in the appendix. Additionally, Formula 3 from a previous study lacks supporting evidence; a brief explanation of its proof process or an indication of its rationale is necessary. The validity of the method “we introduce latent random variables c such that g is independent of y conditioned on c” also requires theoretical verification.\n\nQUESTIONS:\nQuestion 1: In line 79 to 85, the author mention table with R and relation with R. Is this means that a relation in Relational Database is a table in this paper? Since the relations are model as edges of graph.\n\nQuestion 2: According to the paper content, the author mention that “the primary key of table serves as the unique identifier for each row in the table”. And in section 4.1 the rows in parent table can suitable i.i.d assumption. If every row is unique, how rows can be suitable independent and identification distributions?\n\nQuestion 3: From the perspective of learning latent variable method alone, what are the advantages of CLavaDDPM proposed in this paper compared with other studies?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-08-08"}