ClavaDDPM: Multi-relational Data Synthesis with Cluster-guided Diffusion Models

Authors: wei  pang
masoumeh  shafieinejad
lucy  liu
stephanie  hazlewood
xi  he
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:27:43

Abstract: Recent research in tabular data synthesis has focused on single tables, whereas real-world applications often involve complex data with tens or hundreds of interconnected tables. Previous approaches to synthesizing multi-relational (multi-table) data fall short in two key aspects: scalability for larger datasets and capturing long-range dependencies, such as correlations between attributes spread across different tables. Inspired by the success of diffusion models in tabular data modeling, we introduce 
 \textbf{C}luster \textbf{La}tent \textbf{Va}riable guided \textbf{D}enoising \textbf{D}iffusion \textbf{P}robabilistic \textbf{M}odels (ClavaDDPM). This novel approach leverages clustering labels as intermediaries to model relationships between tables, specifically focusing on foreign key constraints. ClavaDDPM leverages the robust generation capabilities of diffusion models while incorporating efficient algorithms to propagate the learned latent variables across tables. This enables ClavaDDPM to capture long-range dependencies effectively. 
 Extensive evaluations on multi-table datasets of varying sizes show that ClavaDDPM significantly outperforms existing methods for these long-range dependencies while remaining competitive on utility metrics for single-table data.

Main paper: 
 introduction: 
 Motivation. Synthetic data has attracted significant interest for its ability to tackle key challenges in accessing high-quality training datasets. These challenges include: i) data scarcity [14, 53] , ii) privacy [2, 19] , and iii) bias and fairness [46] . The interest in synthetic data has extended to various commercial settings as well, notably in healthcare [18] and finance [36] sectors. The synthesis of tabular data, among all data modalities, is a critical task with approximately 79% of data scientists working with it on a daily basis [45] . While the literature on tabular data synthesis has predominantly focused on single table (relation) data, datasets in real-world scenarios often comprise multiple interconnected tables and raise new challenges to traditional single-table learning [38, 3, 12, 22] . These challenges have even enforced a join-as-one approach [15, 17] , where the multi relations are first joined as a single table. However, with more than a couple of relations (let alone tens or hundreds of them as in the finance sector) this approach is neither desirable nor feasible.Challenges. Synthetic Data Vault [35] and PrivLava [5] are recent efforts to synthesize multirelational data using hierarchical and marginal-based approaches. These methods exhibit significant limitations in processing speed and scalability, both with respect to the number of tables and the domain size of table attributes, and they often lack robustness in capturing intricate dependencies. Alternatively, diffusion models have emerged as powerful tools for data synthesis, demonstrating remarkable success in various domains [37] . These models are particularly noted for their strong capabilities in controlled generation. Despite their potential, the application of diffusion models to tabular data synthesis has been limited to unconditional models [25, 50, 28, 24] , leaving a gap in effectively addressing the multi-table synthesis problem.To address these challenges, we introduce ClavaDDPM (Cluster Latent Variable guided Denoising Diffusion Probabilistic Models). Our novel approach leverages the controlled generation capabilities of diffusion models by utilizing clustering labels as intermediaries to model the relationships between tables, focusing on the foreign-key constraints between parent and child tables. This integration of classifier guidance within the diffusion framework allows ClavaDDPM to effectively capture complex multi-table dependencies, offering a significant advancement over existing methods.Contributions. In this work, we: 1) provide a complete formulation of the multi-relational modeling process, as well as the essential underlying assumptions being made, 2) propose an efficient framework to generate multi-relational data that preserves long-range dependencies between tables, 3) propose relationship-aware clustering as a proxy for modeling parent-child constraints, and apply the controlled generation capabilities of diffusion models to tabular data synthesis, 4) apply an approximate nearest neighbor search-based matching technique, as a universal solution to the multi-parent relational synthesis problem for a child table with multiple parents, 5) establish a comprehensive multi-relational benchmark, and propose long-range dependency as a new metric to measure synthetic data quality specific to multi-table cases, and 6) show that ClavaDDPM significantly outperforms existing methods for these long-range dependency metrics while remaining competitive on utility metrics for single-table data. 
related work: 
 Single-table synthesis models. Bayesian network [48] is a traditional approach for synthetic data generation for tabular data. They represent the joint probability distribution for a set of variables with graphical models. CTGAN [47] is a tabular generator that considers each categorical value as a condition. CTAB-GAN [52] includes mixed data types of continuous and categorical variables. Several studies have explored how GAN-based models can contribute to fairness and bias removal [44, 45] . In privacy, GAN-based solutions boosted with differential privacy have not been as successful as their Baysian-network-based competitors [34, 51] . Recent popular Diffusion Models, [20, 40, 42, 41] , offer a different paradigm for generative modeling. TabDDPM [25] utilizes denoising diffusion models, treating numerical and categorical data with two disjoint diffusion processes. STaSy [24] uses score-based generative modeling in its training strategy. CoDi [28] processes continuous and discrete variables separately by two co-evolved diffusion models. Unlike the previous three which perform in data space, TabSyn [50] deploys a transformer-based variational autoencoder and applies latent diffusion models. Privacy and fairness research for diffusion models are currently limited to a few studies in computer vision [26, 11, 16] .There have been few proposals for synthetic data generation for multi-relational data. A study proposed this synthesis through graph variational autoencoders [31] , the presented evaluation is nevertheless very limited. The Synthetic Data Vault [35] uses the Gaussian copula process to model the parent-child relationship. SDV iterates through each row in the table and performs a conditional primary key lookup in the entire database using the ID of that row, making a set of distributions and covariance matrices for each match. This inhibits an efficient application of SDV to the numerous tables case. PrivLava [5] , synthesizes relational data with foreign keys under differential privacy. The key idea of PrivLava is to model the data distribution using graphical models, with latent variables included to capture the inter-relational correlations caused by foreign keys.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: In the conclusion section (second paragraph), we discuss the limitations of our work and our interest in addressing them as future work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background: 
 Multi-relational databases. A multi-relational database R consists of m tables (or relations) (R 1 , . . . , R m ). Each table is a collection of rows, which are defined over a sequence of attributes. One of the attributes, let's consider the first attribute without loss of generality, is the primary key of table R, which serves as the unique identifier for each row in the table. No rows in the same table have repeated values for the primary key attribute. We use Berka database [4] as our running example in this work, as in Figure 1 . Note the Account ID, the primary key for the Account table in Berka.Given a table R j , we say a relation R i has a foreign key constraint with R j , or R i refers to R j , if R i has an attribute known as foreign key that refers to the primary key of R j : for every row r i ∈ R i , there exists a row r j ∈ R j such that r j 's primary key value equals to r i 's foreign key value. For example, the Account ID of the Loan table refers to the primary key of the Account table. If an account row is removed from the Account ID table, so would all the referring rows in the Loan table to this account, for foreign key constraint to hold. Note that the primary key of a table can consist of multiple attributes. In this paper, we focus on the case of a single attribute that is common in practice. Also note that all keys are considered row identifiers and are thus not treated or modeled alongside the actual table attributes in this work.A multi-relational database under foreign key constraints forms a directed acyclic graph (DAG),EQUATIONwith the tables R being the set of nodes, and E being the set of edges. In addition, for R i referring to R j , we also call this a parent-child relationship, where R j is the parent and R i is the child. We use the maximum depth to denote the number of nodes on the longest path in G. Figure 1 shows the corresponding graph to Berka database and its maximum depth is 4.Multi-relational synthesis problem. Given a multi-relational database R = {R 1 , . . . , R m }, we would like to generate a synthetic version R = { R1 , . . . , Rm } that has the same structure and foreign-key constraints as R and preserves attribute correlations within R, including 1) the intercolumn correlations within the same table ; 2) the intra-group correlations within the same foreign key group; 3) the inter-table correlations. The first aspect has been well defined, measured, and tackled in the literature of single-table synthesis [52, 25, 50] while the other two aspects are raised due to foreign-key constraints between tables [5] . For instance, in Berka database (Figure 1 ), the foreign key constraint between the Loan Classifier-guided DDPM. DDPM [20] uses two Markov chains, a forward chain that perturbs data to noise through a series of Gaussian transitions, and a reverse chain that converts noise back to data with the same number of steps of Gaussian transitions (Equation 2).EQUATIONPrior work [40] shows that given label y, the conditional reverse process has the formEQUATIONBy approximating log p ϕ (y | x t ) using Taylor expansion around x t = µ, the conditional reverse process (Equation 3) can be approximated with a perturbed Gaussian transition [10] EQUATIONwhere C is a constant and g = ∇ xt log (P ϕ (y | x t )) | xt=µ computed from the classifier P ϕ .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
clavaddpm: 
 Here, we elaborate on the training and synthesis process of ClavaDDPM, and each design's rationale. | j = 1, . . . , |R 2 |, where j is the index or the primary key value of the jth row, and each row follows a distribution p(y).2) The i.i.d assumption does not apply to the child table rows (x j 's) as they are constrained by their respective parent rows. Consider two loans associated with the same account id; if one's status is in debt ("C"), the other one is likely so too. To capture this dependency, we make a Bayesian modeling assumption that, although child rows associated with the same parent row are not independent, they are conditionally independent of child rows associated with other parent rows, given their respective parent. For example, consider an account table (parent) and a loan table (child). Loans related to the same account (i.e., the same parent) are not independent due to shared account-specific factors. However, loans from different accounts can be considered conditionally independent when accounting for their respective account-level information. Hence, we model X by g j | j = 1, . . . , |R 2 | , where each group g j = x i j | i = 1, . . . , g j represents a set of child table rows referring to the parent row y j .3) Without violating the assumptions made above, we further make an i.i.d assumption on (g j , y j ), which leads to an approximated distribution for the parent-child tables:EQUATIONwhere X = ∪ |R2| j=1 g j and g j = {x 1 j , . . . , x |gj | j }. This model allows us to capture the inter-table correlations (the correlation between tuples from different tables) and the intra-group correlations.Modeling. Despite the simplified formulation with several aforementioned assumptions, learning the distribution p (g j , y j ) is non-trivial. In particular, (g j , y j ) cannot be flattened into a matrix form for learning since the set structured attributes in g j , e.g., the size of a group variable g j is not fixed.A naive solution is to model a conditional distribution of the group given the parent row p (g j , y j ) = p (g j | y j ) p (y j )Direct modeling of Equation ( 6) still has the same issue as before for the foreign key group g j , which can take an arbitrary number of child rows. In particular, when modeling g j = f (x j ) for some function f , there is no trivial structured support for g j if we model for g j using only the attributes or features of the child rows. Furthermore, the conditioning space of the parent row y can be very large (e.g., Account table has a domain size of more than 11,000), which can lead to poorly learned conditional distribution if we treat y as labels in the classifier-guided DDPM. The original space of y is high-dimensional and noisy and does not guarantee any spatial proximity or smoothness. In the context of deep modeling, this drastically worsens the quality of conditional sampling.To address these two challenges, we introduce latent random variables c such that g j is independent from y j conditioned on c gEQUATIONWith this assumption, we get an indirect modeling of inter-table correlations through c:EQUATIONCompared to Equation ( 6), when c is selected to be lying on a low-dimensional, compact manifold, the latent conditional distribution p (g j | c) will be easier to model than p (g j | y j ).For each foreign key group g j , we model its size explicitly with a variable s j . By making an assumption that s j is conditionally independent from its child row variables {x 1 j , . . . , x sj j } given the latent random variable c, we essentially defined a generative process for g j : first sample its size s j ∼ s j , then sample g j child row variables. In addition, we make an i.i.d assumption on the child row variables given the latent variable. Hence, we haveEQUATIONPutting all together, we have the final formulation of the generative process for a two-table case:p (X, Y ) ≈ |R2| j=1p (g j , y j ) i.i.d assumption on g j , y j Equation ( 5)EQUATIONUnlike our naive method that uses a direct modeling of p (x j | y j ), we model p (x j | c j ), which greatly reduced the condition space, thus better capturing the inter-table correlation between X and Y . On the other hand, the intra-group correlations are intrinsically addressed, because our modeling of g j , y j and the corresponding dependency assumptions enforce that two child rows are drawn from the same distribution if and only if they belong to the same foreign key group.Based on Equation (10) , where p ϕ,ψ performs classifier guided sampling by perturbing p ϕ with the gradient of p ψ . We denote steps (7) and (8) by X ∼ p(•|C).We learn the latent variables between a parent and a child pair in a bottom-up fashion (starting from the leaf nodes in G) and pass all the latent variable values to the parent table for the next set of latent variables at higher levels. Given a parent-child pair (Y, X), the child table X also has k leaf node children, Z 1 , . . . , Z k . Let c X,Zi represent the latent variables learned on the joint space (X; Z i ). The augmented table for X is formed by appending all its latent variable values, i.e., T X = (X; C X,Z1 ; . . . ; C X,Z k ). Then, the latent variable c Y,X is learned on the joint space of (Y ; T X ) instead of (Y ; X). Therefore, our latent learning process follows a bottom-up topological order, ensuring each child table is already augmented by the time we learn the latent variable to augment its parent.The training phase and the synthesis phase are similar to the two-table case, by handling the parentchild tables in a top-down topological order using the augmented tables. We detail the end-to-end algorithms for the complex data in Appendix B. However, we would like to highlight a special case when a table X has multiple parents Y 1 , . . . , Y k . During synthesis, we will have k synthetic latent variable C1 , . . . , Ck corresponding to the k parents, and thus k copies of synthetic child tables X1 ∼ p(• | C1 ), . . . , Xk ∼ p(• | Ck ). Unifying these diverged synthetic tables presents a challenge and we present a universal solution in Section 4.3.3.Extending the model to include more tables allows for capturing longer-range dependencies, beyond just those between adjacent tables. For example, as shown in Figure 1 , the dependency between the Demographic table and the Credit Card table can also be captured and quantified. Further details are provided in Section 5.We detail how design decisions for ClavaDDPM meet our goals and align with our assumptions.Given the conditional independence between the parent row and its foreign key group (Equation ( 7)), it is important to model the latent variable c such that it can effectively capture the inter-table correlation within the same foreign key group. In ClavaDDPM, we learn c using Gaussian Mixture Models (GMM) in the weighted joint space of X and Y , denoted as H = (X; λY ), where λ is a weight scalar controlling the importance of child and parent tables when being clustered. Concretely, we consider k clusters, and model the distribution of h = (x; λy) with Gaussian distributed around its corresponding centroid c, i.e., P (h) =k c=1 P (c) P (h | c) = k c=1 π c N (h; µ c , Σ c ) .Note that diagonal GMMs are universal approximators, given enough mixtures of Gaussian distributions [49] . Therefore, we can further enforce diagonal covariance, i.e., Σ c = diag . . . , σ 2 l , . . . , which, being properly optimized, immediately satisfies our assumptions that the foreign key groups are conditionally independent of their parent rows given c. In addition, the family of Gaussian Process Latent Variable Models (GPLVM) [30, 27, 33] has been used as an embedding technique to find low-dimensional manifolds that map to a noisy, high-dimensional space. This satisfies our need to learn a stochastic map between the noisy parent space and a condensed latent space. Thus, we can achieve a better trade-off by sacrificing some information fidelity during this quantization process while making the conditional space better shaped. However, such clustering in the joint space (X; λY ) could potentially lead to inconsistency when we create the augmented table T Y = (Y ; C). Though we add a weight λ to the parent rows such that child rows with the same parent rows are likely to be assigned to the same cluster, there is still some chance that they end with different clusters. In particular, for each parent row y j ∈ Y , its child rows are assigned to different clusters. In ClavaDDPM, we impose a majority voting step to find the most popular cluster label in each foreign key group and assign it to the parent row y j . In practice, the voting agree rates tend to be high, and this can be further enforced by assigning a higher weight to the parent table (increasing λ) during GMM clustering. We evaluate the choice of λ and voting agree rates in our ablation study in Section 5.3.While alternative latent learning algorithms could potentially be applied, such as TabSyn [50] that demonstrated the utility of latent encoding of tabular data with VAE, this work focuses on demonstrating the effectiveness of a simple diagonal Gaussian Mixture Model (GMM) for ClavaDDPM. Our experiments (detailed in Section 5) reveal that ClavaDDPM with a diagonal GMM achieves state-of-the-art results while maintaining low computational overhead. We leave the exploration of more complex latent learning techniques for future work.Gaussian diffusion backbone. We consider one of the state-of-the-art diffusion models for single tabular data, TabDDPM [25] , as the backbone model. TabDDPM models numerical data with Gaussian diffusion (DDPM [20] ), and models categorical data with multinomial diffusion ( [21] ) with one-hot encoding, and carries out disjoint diffusion processes. However, the modeling of multinomial diffusion suffers significant performance overheads, and poses challenges to guided sampling. Instead, ClavaDDPM uses a single Gaussian diffusion backbone to model both numerical and categorical data in a unified space, where categorical data is mapped to the numerical space through label-encoding. To be specific, for a categorical feature with m distinct values C = {c 1 , . . . , c m }, a label encoding E : C → {0, . . . , m -1} maps each unique category c i to an assigned unique integer value. For a table row x = [x num ; • • • ; x cati ; • • • ],where x num represents all the numerical features and x cati represent a categorical feature, we obtain the unified feature byx uni = [x num ; • • • ; E (x cati ) ; • • • ].Based on this encoding, we learn p θ (y, c) on the augmented parent tableT Y = (Y ; C) through training a Gaussian diffusion model on the unified feature space (Y uni ; E (C)).Classifier guided synthesis. As defined in Equation ( 10), we model p x i j | c j by leveraging classifier-guided sampling of diffusion models, following [10] . In practice, with the sheer power of diffusion models, we jointly model p (x | c) for the entire table without distinguishing j. First, we train a Gaussian diffusion model p ϕ on child table row x, with its reverse process modeled with x t ∼ N (x t+1 ; µ ϕ t+1 , Σ ϕ t+1 ). Then, we train a classifier that classifies cluster labels based on x. The conditional reverse process can be approximated byx t | c ∼ N (x t+1 ; µ ϕ t+1 + ηΣ ϕ t+1 g ψ t+1 , Σ ϕ t+1 ), where g ψ t+1 = ∇ xt+1 log (p ϕ (c | x t+1)) and η is a scale parameter controlling the strength of conditioning. One can regard η as a hyper parameter measuring the trade-off between single-table generation quality and inter-table correlations, to be demonstrated by our ablation study in Section 5.3.Consider the case where some child table X has two parent tables Y 1 , Y 2 . Our parent-child synthesis modeling paradigm would lead to two divergent synthetic child tables X1 ∼ X | Y 1 , and X2 ∼ X | Y 2 Each synthetic table encodes its own parent-child relationship, i.e. the foreign keys.Combining X1 and X2 so that the synthetic child table contains foreign keys from both parents p 1 and p 2 is non-trivial, and we call it a multi-parent dilemma. One possible approach is to explicitly constrain the model sample space of X | Y 2 to be the synthetic data X1 , as used in PrivLava [5] . However, this approach is not applicable to diffusion models that sample from a continuous space.We provide a universal solution for all generative models. Consider some real data point x with two parent rows y j 1 and y k 2 . Ideally, some synthetic data point x following the same distribution as real data point x should be sampled from x | y j 1 , y k 2 . This can be approximated by finding the intersection of two conditional distributions x | y 1 and x | y 2 . Specifically, we estimate x by finding two synthetic data points x1 ∈ X1 and x2 ∈ X2 , such that x1 ∼ x | y j 1 and x2 ∼ x | y k 2 , and the two points are close enough. We reason as follows: although x1 was sampled from x | y 1 , as long as it is close enough to some other synthetic data point x2 sampled from x | y 2 , then x1 will also be within in the high density region of the distribution x | y 2 , indicating a high probability that x1 follows x | y 1 , y 2 . Symmetrically, the same reasoning also holds for x2 . Therefore, we can estimate the true sample data point by x = f (x 1 , x2 ) if x1 is close to x2 , where f can simply be an interpolation between two data points in practice. We call this a matching process between two divergent synthetic tables X1 and X2 , and this can be done efficiently using approximate nearest neighbor search. Although we call this a "matching", it does not require finding a one-to-one mapping. Note that this estimate can be further improved by resampling X1 and X2 and estimate X with more data points rather than just a pair, and the trade-off is a larger computational overhead, and we leave this for future research. Empirically, sampling X1 and X2 only once is already strong, and an ablation study on the effectiveness of parent matching is in Section 5.3. 
evaluation: 
 We evaluate ClavaDDPM's performance in multi-relational data synthesis, using both single-table and multi-tables utility metrics (including the new long-range dependency). We present an end-to-end comparison of ClavaDDPM to the SOTA baselines, followed by an ablation study for ClavaDDPM.Real-world datasets. We experiment with five real-world multi-relational datasets including California [6] , Instacart 05 [23] , Berka [4] , Movie Lens [39, 32] , and CCS [32] . These datasets vary in the number of tables, the maximum depth, the number of constraints, and complexity. Among all, Berka, Movie Lens, and CCS exhibits complex multi-parent and multi-children structures. We use Berka in our work for ablation study and model anatomy. Details can be found in Appendix C.1.Baselines. We adopt two multi-relational synthesis models in literature as our baselines: PrivLava [5] as a representative of state-of-the-art marginal-based methods, and SDV [35] as a statistical method specially designed for multi-relational synthesis. We also introduce two multi-relational synthesis pipelines, SingleT(ST) and Denorm(D), as our additional baselines. SingleT learns and generates each table individually, but it also assigns foreign keys to each synthetic child table accordingly to the real foreign key group size distribution such that the group size information is preserved. Denorm follows the baseline idea that joins table first, but it is hard to join all tables into a single table. Hence, Denorm first applies single-table backbone model to generate the joined table between every parent-child table pair and then split it. For these two pipelines, we use CTGAN [47] and TabDDPM [25] as the single-table backbone models, representing the SOTA tabular synthesis algorithms with GAN-based models and diffusion-based models. The details can be found in Appendix C.2.Evaluation metrics. We evaluate the quality of the synthetic data using: 1) cardinality to measure the foreign key group size distribution for the intra-group correlations; 2) column-wise density estimation (1-way) to estimate the density of every single column for all tables; 3) pair-wise column correlation (k-hop) for the correlations of columns from tables at distance k, e.g., 0-hop refers to columns within the same table and 1-hop refers to a column and another column from its parent or child table; 4) average 2-way, which computes the average of all k-hop column-pair correlations, taking into consideration of both short-range (k = 0) and longer-range (k > 0) dependencies. For each measure, we report the complement of Kolmogorov-Smirnov (KS) statistic and total variation (TV) distance 3between the real data and the synthetic data, ranging from 0 (the worst utility) to 1 (the best utility). The reported results are averaged over 3 randomly sampled synthetic data.We also consider higher-order single-table evaluation metrics for some representative tables as prior work [50] . We include their details and experiemntal results in Appendix D due to space constraints.All experiments are conducted with an NVIDIA A6000 GPU and 32 CPU cores, with a time limit of 7 days. If an algorithm fails to complete within the time limit, we report TLE (time limit exceeded). Implementation details and hyperparameter specifics are in Appendix C.3.We conducted multi-table synthesis experiments on five multi-table datasets and report the averaged utility with standard deviation for all algorithms in Table 1 . First, the evaluation shows that ClavaD-DPM has an overall advantage against all the baseline models in terms of correlation modeling, and is surpassing the baselines by larger margins for longer-range dependencies. e.g. in Instacart 05, our model outperforms the best baseline by 58.29% on 2-hop correlations, and in Berka, our model exceeds the best baseline by 20.24% on 3-hop correlations. For single-column densities and cardinality distributions, ClavaDDPM exhibits a competitive result compared to the state-of-the-art baseline models. We also evaluate ClavaDDPM against baselines on high-order single-table metrics (Appendix D.3), which shows that our model has advantages in preserving data fidelity, generating diverse data, and achieving high machine learning efficacy.It is worth noting that ClavaDDPM, despite its complexity and capability, is more efficient and robust than some simpler baselines. PrivLava demonstrates strong performance on the California dataset (the simplest data), but fails to converge on all the other datasets. SDV also tends to fail on complex datasets, and is limited to datasets with at most 5 tables and maximum depth of 2 [9] . Although TabDDPM shares a similar model backbone with ClavaDDPM, its synthesis fails to complete within 7 days on multiple datasets, while ClavaDDPM completes all experiments within 2 days.k = 20, λ = 1.5, η = 1 k = 1 k = 1000 λ = 0 λ = 10 λ = 100 η = 0 η = 2Gaussian diffusion backbone. To decouple the effect of our Gaussian diffusion only backbone with the latent conditioning training paradigm, we also included two models in Table 1 : ST-ClavaDDPM and D-ClavaDDPM, which use the Gaussian diffusion model in ClavaDDPM as backbone model, but are trained and synthesized following Single-T and Denorm. Compared to other baselines, ST-ClavaDDPM exhibits superiority in modeling both single column densities and column-pair correlations. ST-ClavaDDPM significantly outperforms its sibling ST-TabDDPM, which proves the effectiveness of using Guassian diffusion for tabular data synthesis solely. On the other hand, ST-ClavaDDPM falls short on longer-range correlations when compared to the full ClavaDDPM model. This observation provides solid evidence to the efficacy of our multi-table training paradigm.Besides the study of the single-table backbone models, we perform a comprehensive ablation study using Berka (for it has the most complex multi-table structure) on each component of ClavaDDPM and provide empirical tuning suggestions. The full results are in Table 2 .Number of clusters k. We study the necessity of using latent cluster conditioning: (i) no conditioning with k = 1; (ii) many clusters with k = 1000 to approximate a direct conditioning on parent rows rather than latent variables. When k = 1, the quality of long-range correlation degrades drastically. When k = 1000, we still get reasonably strong performance, which showcases ClavaDDPM's robustness. Compared to the default setting (k = 20), the metrics are lower in all of cardinality distribution, single column densities, and column correlations -proper latent variable learning leads to better results than direct conditioning on parent rows. We also report a new metric avg agree-rate, the average of all per-table agree rates for the labels within each foreign key group (Section 4.3.1). This measure highly depends on k, but a higher rate does not always imply a better performance (e.g., k=1 achieves perfect rates). We provide more insights on how it varies with the next parameter. We also conducted finer-grained experiments to examine the effect of k on model performance, as shown in Appendix D.2.Parent scale λ. Varying the parent scale parameter λ changes the agree-rates as shown in Table 2 , but the downstream model performance does not vary too much. This result indicates that the relation-aware clustering process is robust against such factors, and the GMM model is capable of capturing nuances in data patterns. The detailed discussion is in Appendix D.1.Classifier gradient scale η. This parameter controls the magnitude of classifier gradients when performing guided sampling, and thus the trade-off between the sample quality and conditional sampling accuracy. Table 2 shows that, when η = 0, which essentially disables classifier conditioning, the single column densities (1-way) are slightly higher than the default setting. However, it falls short in capturing long-range correlations. When η = 2, the conditioning is emphasized with a higher weight, which significantly improves the modeling of multi-hop correlations compared to η = 0 case.Comparing with no matching for multi-parent dilemma. Berka (Figure 1 ) suffers from the multiparent dilemma , where the Disposition table has two parent tables, Account and Client. Our abalation study switch the table matching technique to a naive merging of two synthetic table (Appendix C.2). The experiment result show that even if trained with the same hyper parameters and model structures, ClavaDDPM with matching is significantly stronger than the no-matching setup in terms of long-range correlations, with 3-hop correlations 16.70% higher than no-matching. 
conclusion: 
 We proposed ClavaDDPM as a solution to the intricate problem of synthetic data generation for multi-relational data. ClavaDDPM utilizes clustering on a child table to learn the latent variable that connects the table to its parents, then feeding them to the diffusion models to synthesis the tables. We presented ClavaDDPM's seamless extension to multiple parents and children cases, and established a comprehensive multi-relational benchmark for a through evaluation -introducing a new holistic multi-table metric long-range dependency. We demonstrated ClavaDDPM not only competes closely with the existing work on single-table synthesis metrics, but also it outperforms them in ranged (inter-table) dependencies. We deliberately selected the more complex public databases to exhibit ClavaDDPM's scalability, and introduce it as a confident candidate for a broader impact in industry.We focused on foreign key constraints in this work, and made the assumption that child rows are conditionally independent given corresponding parent rows. This brings three natural follow-up research directions: i) extension to the scenarios where this prior information is not available and these relationships need to be discovered first [29] , ii) further relaxing the assumptions, and iii) inspecting multi-relational data synthesis with other integrity constraints (e.g, denial constraints [15] , general assertions for business rules). Furthermore, we evaluated ClavaDDPM's privacy with the common (in tabular data literature) DCR metric. Nonetheless, we think it is worthwhile to: i) evaluate the resiliency of ClavaDDPM against stronger privacy attacks [43] , and ii) investigate the efficacy of boosting ClavaDDPM with privacy guarantees such as differential privacy. Similarly, the impacts of our design on fairness and bias removal, as another motivating pillar in synthetic data generation, is well worth exploring as future work. We believe the thorough multi-relational modeling formulation we presented in this work, can serve as a strong foundation to build private and fair solutions upon. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We include computing specifics in the evaluation section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethic https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts of our work in the conclusion section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We use public datasets for experiments.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We add citations and resources for the datasets and source code used in the evaluation.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We use boldface to represent random variables. e.g. Y ∼ Y , where Y is the data of R 2 , and Y is the random variable Y being sampled from. In addition, we use subscript to represent the parent row some data or random variable refers to. e.g., x j represents the child random variable that refers to parent y j . The important notations used in the paper are summarized in Table 3 .Relational database, relational We detail the end-to-end algorithms for the three phases of ClavaDDPM, including (i) latent learning and table augmentation, (ii) training, and (iii) synthesis.As shown in Algorithm 1, given a database R = {R 1 , . . . , R m } and foreign key constraint graph G, we learn the set of latent variables C i,j for every pair of parent-child (R i → R j ) ∈ G.E and augment all the latent variables to the parent table and the child table, denoted by T j and T ′ i , respectively. We initialize each augment table with its original table (line 1). This algorithm follows a bottom-up topological order starting from the leaf child with its parent (line 2), ensuring each child table is already augmented by the time we learn the latent variable to augment its parent. For each parent-child pair R i → R j , we join T i (not T ′ i ) with R j into a single table (X; Y ) (line 3) and then run the clustering algorithm using GMM and maximum voting described in Section 4.3.1. We append the corresponding clustering labels C i,j to the augmented parent table T j and augmented child table T i , respectively.Algorithm 1 ClavaDDPM: Latent learning and table augmentation. Augment parent T j ← (T j ; C i,j )Input: tables R = {R 1 , . . . , R m }, foreign key constraint graph G Output: latent variables {C i,j |(R i → R j ) ∈ G.E}, augmented parent tables {T 1 , . . . , T m }, augmented child tables {T ′ 1 , . . . , T ′ m } 1: Initialize augmented tables {T 1 , . . . , T m } ← R, {T ′ 1 , . . . , T ′ m } ← R 2: for (R i → R j ) in bottom-up topological order of G do6: Augment child T ′ i ← (T ′ i ; C i,j ). 7: end forTraining. As shown in Algorithm 2, the training phase takes in the augmented parent tables {T 1 , . . . , T m } and the foreign key constraint graph G. For each augmented table T j , we train a diffusion model p Tj (lines 2-4). Then, for each parent-child pair R i → R j (lines 5-7), we train a child classifier p ϕ (c i,j |x) with R i 's child augment table T ′ i , where the latent column C i,j is used as labels, and all remaining columns are used as training data (including the augmented latent columns corresponding to R i 's children). Using the same table, we also estimate the foreign key group size distribution conditioned on the latent variable p(s|c i,j ). Learn classifier p ϕ (c i,j |x) and p(s|c i,j ) using with T ′ i (ignoring irrelevant latent columns) and add to C and S respectively 7: end for Synthesis. Algorithm 3 takes in learned diffusion models D, classifiers C, group size distributions S, and the DAG representation of the database G, and outputs the synthetic database R = R1 , . . . , Rm . We first initialize the synthetic augmented tables to be empty (line 1).Then, for root augmented tables, since they have no parents to condition on, they can be directly synthesized from their diffusion models (line 2-4). Next, we traverse the database in topological order to synthesize the remaining augmented tables (line 5-16): If we have already synthesized Ti before, which means we encounter the multi-parent dilemma, we just store the old version and continue to generate a new version (line 6-9). For each parent-child relationship R i → R j , we must have already sampled the augmented parent table Tj . This is because we follow the topological order of a DAG, and all root augmented tables have been synthesized as base cases. Therefore, we can obtain the synthetic latent variables Ci,j from the synthetic augmented parent Tj (line 10). Then, we iterate through each synthetic latent value ci,j and perform a two-step sampling: (1) use the learned group size distribution to conditionally sample a group size s (line 12); (2) sample s rows of data conditioned on ci,j using classifier guided sampling (line 13). We repeat this process until the augmented child table Ti is fully synthesized. We simply obtain synthetic tables from synthetic augmented tables by removing all synthetic latent columns (line [17] [18] [19] . Finally, for all the encountered multi-parent dilemmas, we follow Section 4.3.3 to match the divergent versions. Here we describe the real-world datasets used in our evaluation in detail. The specifics of datasets are in Table 4 .The California dataset is a real-world census database ( [6] ) on household information. It consists of two tables in the form of a basic parent-child relationship.Instacart 05: The Instacart 05 is created by downsampling 5-percent from the Kaggle competition dataset Instacart ( [23] ), which is a real-world transaction dataset of instacart orders. This dataset consists of 6 tables in total with a maximum depth of 3.The Berka dataset is a real-world financial transaction dataset ( [4] ), consisting of 8 tables with a maximum depth of 4. This will be the main dataset in our work for ablation study and model anatomy.Movie Lens: The Movie Lens dataset ( [39] , [32] ) consists of 7 tables with a maximum depth of 2. This dataset exhibits complex multi-parent and multi-children structures. Table 5 : Hyper parameters of ClavaDDPM on each dataset.We conducted CTGAN experiments using the interface from SDV library, and follows the default parameters, where the learning rates for the generator and discriminator are both 2e -4, and is trained 300 epochs.PrivLava's code is not publicly available, and we directly followed the authors' settings. Note that PrivLava requires a privacy budget searching process, and ϵ = 50 is the largest working privacy budget according to our experiments, where larger ϵ leads to failure. We consider this as large enough to resemble a non-private setting.For SDV, we used the default setting of their HMASynthesizer, which by default uses a Gaussian Copula synthesizer.ClavaDDPM settings. We list the major hyper parameters used by ClavaDDPM for each dataset in Table 5 , and we provide an empirical guidance for hyper parameter tuning: it is suggested to use number of clusters k to be at least 20, and classifier scale η to be in [0.5, 2]. We empirically find ClavaDDPM consistently perform well in such a range across all datasets. Parent scale λ is a less sensitive factor, and λ = 1 is a stable starting point for tuning. In general, ClavaDDPM is robust, with a small hyper parameter space, and there is very little need for tuning.C2ST. The Classifier Two Sample Test trains a logistic regression classifier to distinguish synthetic data from real data. We consider this metric as a high-level reflection of data fidelity.Machine Learning Efficacy. Different from prior works that evaluate MLE utilities [25, 52, 50] , who work on datasets with predefined machine learning tasks, the five real-world multi-relational datasets we use do not come with a designated downstream task. In addition, the prior knowledge about which column will be used for downstream predictions will introduce significant inductive bias to the training process, especially for models capable of performing task-oriented training. To avoid such issue, we evaluate machine learning efficacy on each of the columns. To be specific, each time we select a column as target, and train an XGBoost [8] model on remaining columns. For categorical target columns, we perform regression and evaluate R 2 , and for categorical target columns we perform classification and evaluate F 1 . The overall MLE is measured by average R 2 and average F 1 across all columns.To evaluate the single-table MLE on synthetic data generated from multi-table synthesis process, instead of performing an independent train-test split on each table, we split by foreign key relationship. e.g. for California dataset, we first perform a random 90%, 10% split on the parent table Household, and then we follow the foreign key constraints to assign corresponding child rows, i.e. Individuals to the corresponding buckets. Note that although this splitting method does not lead to the same train/test ratio on child table, we consider such sampling to be foreign key relationship preserving, which is a more important property in the context of multi-table synthesis.As introduced in Section 4.3.1, within each foreign key group, we perform a majority voting to synchronize the assigned cluster label among the group. To measure the consistency of such majority voting process, we introduce the measurement of agree rate, which computes the average ratio of agreeing on the mode within each foreign key group, and the metric avg agree-rate is the average of all per-table agree rates within a multi-table dataset.EQUATIONA(g) represents the agree rate of some group g, where m g represents the number of records that are assigned the mode cluster within g, and the avg agree-rateEQUATIONis computed as the average of the agree rates across all groups.However, our experiment results in Table 2 indicate that the relationship-aware clustering process is robust against such factors, and the GMM model is capable of capturing nuances in data patterns. In our experiments on Berka dataset, ClavaDDPM's clustering process achieves a consistent agree rate around 81%, which is practically high enough given we have 20 clusters. Intuitively, when parent scale approaches infinity, the clustering is performed completely on parent table, which will lead to a perfect agree rate. Also note that a higher agree rate does not always imply a better performance, and the disagreement can potentially come from the intrinsic parent-child relationships. e.g. when child data is intrinsically independent of parent data, it is reasonable to have noisy learned latent variables, leading to low agree rates. However, in such cases the noisy latent variable would not degenerate model performance, because the best strategy will be a direct sampling of child table, rather than conditioning on some enforced prior distribution. In addition, as shown in Table 2 , agree rates are highly affected by the number of clusters chosen, and there exists a trade-off between the granularity in clustering and consistency. In an extreme case where we have k = 1 cluster, indicating an infinitely coarse-grained latent learning, it trivially achieves perfect agree rates.We conducted finer-grained experiments to examine the effect of the number of clusters k on model performance, as shown in Figure 3 , which offer empirical insights for selecting k. Based on the results, (1) a binary search approach could be used to efficiently find a suitable k, and (2) while it may require more computational resources, opting for a larger k is generally a safe choice. We also consider higher-order single-table evaluation metrics for the quality of some representative tables as prior work [50] : 1) α-precision and β-recall [1] to measure fidelity and diversity of synthetic data; 2) Machine Learning Efficacy (MLE) to measure the downstream-task utility; 3) Classifier Two Sample Test (C2ST) to measure if the synthetic data is distinguishable from real data by machine learning models. We evaluated high-order single-table metrics on the California dataset across all baseline models and ClavaDDPM. Following [50] , for the evaluation of MLE we perform a 90%, 10% train-test split, where the F 1 and R 2 metrics are evaluated on the 10% holdout set. Note that although PrivLava has an advantage on the California dataset when evaluated with statistical tests (Table 1 ), ClavaDDPM exhibits competitive, or even stronger performance than PrivLava on higher-order metrics. Especially for MLE, ClavaDDPM surpasses PrivLava by 13.07% in terms of average R 2 in Individual table, and also beats PrivLava on average F 1 in both tables. Also notice that the baseline ST-CLAVADDPM dominates in high-order metric evaluations, demonstrating the strength of our Gaussian diffusion-only backbone model.ClavaDDPM achieves a second-highest β-recall on Household table and ranks first in β-recall on Individual table with large margin, gaining a 7.65% advantage over the best baseline without ClavaDDPM backbone. This serves as strong evidence that ClavaDDPM is not only data fidelity preserving, but is also capable of generating highly diverse data.We follow TabDDPM [25] to perform a privacy sanity check against SMOTE [7] , which is an interpolation-based method that generates new data through convex combination of a real data point with its nearest neighbors. We use the median Distance to Closest Record (DCR) [52] to quantify the privacy level. We compare the median DCR, as well as DCR distributions of ClavaDDPM against SMOTE on selected tables.As shown in table 7 , ClavaDDPM although neither specialized in privacy preserving, nor in single table synthesis, it still maintains a reasonable privacy level. The charts 4 demonstrates the distributions of DCR scores, where ClavaDDPM is in blue. The overall distribution is more leaning to the right side, indicating an overall higher DCR distribution. Justification: We provide empirical evidence in the evaluation section for the claims made in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We carefully go through the NeurIPS Code of Ethic and ensure we follow it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).In the context of databases, a table is formally referred to as a relation. Throughout this work, we use these terms interchangeably.The complement to KS/TV distance between two distributions P and Q is 1.0 -DKS/TV(P ||Q). We use KS for numerical values and TV for categorical values. 


Decision: Despite the relatively low mean score of the paper, I recommend the paper for acceptance, albeit with less certainty. The reason for this decision is that one of the reviewers was very positive about the paper and provided a set of strong arguments in favor of the proposed method:

- Important problem: Synthetic data generation is growing in the era of large, data-hungry models whose quality depends on the quality of training data. Moreover, an application of this problem to relational databases.
- Novelty: ClavaDDPM is the first tabular DDPM that supports generation conditioned on multiple tables (organized in several hierarchy levels) and can be used on real-world datasets.
- Innovative: The idea of learning latent variables with classifier guidance for grouped keys is interesting. In addition, the model supports both numerical and categorical data and includes parent-child relationships. 
- Strong empirical results: ClavaDDPM was compared against existing non-diffusion multi-table models (PrivLava and Synthetic Dava Vault), the authors also adapted single-table baselines CTGAN and TabDDPM in several versions as baselines. High-quality ablation studies.
- Well-written paper.

One reviewer (bZ6q) provided a rather shallow review, likely written with the help of an LLM, and responded to the rebuttal with comments that I could not comprehend. The other negative reviewer (5kZH) ghosted the authors after the rebuttal was submitted despite several attempts to obtain a response. In my opinion, the authors provided reasonable responses in their rebuttal to said reviewer. For instance, the reviewer simply mentioned other methods to compare to without explaining why comparing to these methods adds any value to the paper. Hence, I think that these two reviews do not meet the quality standards of NeurIPS. Hence, not taking these into account, one reviewer is very positive about the paper, and one gives it a score of 5. Overall, this points to accepting the paper. I also think that his paper falls into the "new work" category (a new perspective on an important problem), which often receives harsher criticism than papers that fall into more established "mainstream" areas. For all of these reasons, I recommend to accept the paper.

Reviews: review: paper_summary: paper_summary: This work addresses the challenges inherent in generating multi-relation tabular data and proposes a novel generation method based on hidden random variables.  The approach analysis correlations between primary and foreign keys between tables by predicting hidden variables associated with these keys within simultaneous tables.  These hidden random variables are inferred using Denoising Diffusion Probabilistic Models (DDPM) grounded in stochastic processes.  The method is elaborated from two-table linkage generation to multi-table linkage generation, with a step-by-step explanation of the approach.  Comprehensive experiments are conducted to demonstrate the practical effectiveness of the proposed method.
strength_weakness: strength_weakness: Although this work attempts to elaborate on its theoretical framework with substantial content, the notation system and descriptive symbols used do not conform to standard academic mathematics conventions (see sections 3. Background and 4.1 Notations). The marks in equation are not uniform (Eq.2). Furthermore, several key formulas lack thorough proofs (for example, Formula 8). The proof of this key formula would be best placed in the appendix. Additionally, Formula 3 from a previous study lacks supporting evidence; a brief explanation of its proof process or an indication of its rationale is necessary. The validity of the method “we introduce latent random variables c such that g is independent of y conditioned on c” also requires theoretical verification.
questions: questions: Question 1: In line 79 to 85, the author mention table with R and relation with R. Is this means that a relation in Relational Database is a table in this paper? Since the relations are model as edges of graph.

Question 2: According to the paper content, the author mention that “the primary key of table serves as the unique identifier for each row in the table”. And in section 4.1 the rows in parent table can suitable i.i.d assumption. If every row is unique, how rows can be suitable independent and identification distributions?

Question 3: From the perspective of learning latent variable method alone, what are the advantages of CLavaDDPM proposed in this paper compared with other studies?
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: This paper proposes ClavaDDPM to address two key deficiencies in multi-table data generation: scalability for larger datasets and capturing long-range dependencies, such as correlations between attributes across different tables. This approach utilizes cluster labels as intermediaries to model relationships between tables, paying special attention to foreign key constraints. The authors elaborated on the proposed method, made some assumptions, conducted some experiments for evaluation, and performed some analysis about the results.
strength_weakness: strength_weakness: 1. I have some concerns about the assumptions made in the paper, which seem to be somewhat inconsistent with the scenarios in real world settings. Please refer to Questions 1 and 2 for details.

2. There may be some clarifications needed in the experimental section. Meanwhile, the analysis of the experimental results does not cover some key points. Please refer to Questions 3-6 for details.

3. There are some problems with the writing and formatting of this paper. Please refer to Questions 7-8 for details. Also, there are some confusing statements in the paper, which hinder the readability of the paper. Please refer to Questions 9-10 for details.
questions: questions: 1. The paper assumes that there are parent-child relationships between different tables and the relationships are known. I However, in real life, the relationships may be very complex and may not be known in advance. In this case, how can ClavaDDPM handle it?

2. The i.i.d. assumption in Section 4.1 seems too strong, because the child rows corresponding to different primary keys may not be independent in real-world scenarios. If this is the case, does the theoretical analysis in the paper still hold?

3. Please clarify why you selected CTGAN over newer or state-of-the-art models like CTAB-GAN, TabSyn, and CoDi mentioned in the related work for your experiments.

4. The reason for the absence of average agree rate in the last two columns of Table 2 remains unclear. Additionally, it is not specified what NA represents. Moreover, there is a lack of clarity regarding the calculation method and formula used to determine the agree rate in this experiment.

5. I feel a bit confused about why TabDDPM converges so slowly on the Instacart05 and Movie Lens datasets. The authors of TabDDPM stated in their work that TabDDPM can converge quickly with a 2080Ti GPU. In addition, I noticed that the Berka dataset seems to be larger than those two datasets. Could you kindly provide an explanation as to why TabDDPM exhibits convergence on this larger dataset, while failing to do so on the two smaller datasets?

6. Are the evaluation metrics used in the experiment widely used in the field or are they proposed by the authors? I haven't seen similar metrics in related works. If they are proposed by the authors, why don't you use metrics from other papers, such as the relative error used in PrivLava [1] or more comprehensive metrics used in TabSyn [2]?

7. There are some typos in the paper: “LavaDDPM” in line 206, “Dnorm” in the caption of Table 1, “The experiment result show…” in line 355.

8. Page 18's experimental results table lacks a caption.

9. In Section 5.2, you referred to additional results in Appendix 5.3, which is missing from the paper. Please provide the mentioned experimental results.

10. In the checklist, you said that you discussed the limitations and future work in the second paragraph of the conclusion section, but there is no relevant discussion in the conclusion of the paper, not even the second paragraph, which is confusing.

Reference:

[1]	K. Cai, X. Xiao, and G. Cormode. Privlava: synthesizing relational data with foreign keys under differential privacy. Proceedings of the ACM on Management of Data, 1(2):1–25, 2023.

[2]	H. Zhang, J. Zhang, B. Srinivasan, Z. Shen, X. Qin, C. Faloutsos, H. Rangwala, and G. Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. arXiv preprint arXiv:2310.09656, 2023.
limitations: limitations: n/a
paper_summary: paper_summary: This paper proposes the new ClavaDDPM approach to address the scalability and long-range dependency challenges in tabular data synthesis. ClavaDDPM uses clustering labels to model inter-table relationships, particularly focusing on foreign key constraints, and employs diffusion models' robust generation capabilities along with efficient algorithms to propagate learned latent variables across tables. Evaluations demonstrate ClavaDDPM's superior performance in capturing long-range dependencies and competitive utility metrics for single-table data.
strength_weakness: strength_weakness: $\bullet$ The manuscript exhibits a lack of punctuality and standardization in the presentation of mathematical formulas, leading to potential ambiguity in interpretation. Key terms are not consistently defined upon first use, and there is a misuse of symbols that may confuse readers. Additionally, the placement of table names appears to be incorrect in some instances, further detracting from the paper's clarity. It is essential to standardize the notation and ensure consistent definitions and proper placement of table names for better readability.

$\bullet$ The assumptions made in the model section require more rigorous justification. The authors should provide a more detailed explanation of why these assumptions are reasonable and how they contribute to the validity of the model. This will strengthen the theoretical foundation of the work and enhance the credibility of the proposed approach.

$\bullet$ The paper would benefit from an analysis of the model's complexity, particularly in the context of large-scale databases. Given the increasing size and complexity of real-world databases, understanding how the proposed model scales and performs under such conditions is crucial. Including a complexity analysis will provide insights into the model's practicality and efficiency. 

$\bullet$ The ablation study regarding the cluster number 'k' lacks necessary granularity. A more fine-grained investigation is needed to understand the impact of varying 'k' on the model's performance and to identify the optimal settings for different scenarios. This will provide a clearer understanding of the model's behavior and its adaptability to various contexts.

$\bullet$ The section on the multi-relational synthesis problem, currently placed in the appendix, should be relocated to the main body of the paper. This will improve the flow of the paper and make it easier for readers to follow the synthesis process and understand its relevance to the overall work. Integrating this section into the main text will enhance the coherence and comprehensiveness of the paper.
questions: questions: See the above weaknesses.
limitations: limitations: No.
paper_summary: paper_summary: The paper tackles the problem of synthetic data generation in tabular formats and particularly focuses on data generation in multi-relational (multi-table) setups like relational databases. Whereas existing tabular diffusion models work only on single tables, the proposed ClavaDDPM extends the DDPM framework to support many relational tables linked via primary-foreign keys. In the hierarchy of parent-child tables, child tables contain keys from parent tables, and ClavDDPM conditions the generative process of child entries on their parent entries (grouped by the foreign key). To enable this, the model learns latent variables (expressed as GMMs) corresponding to the groups of foreign keys. Capturing the hierarchy of parent-child relations enables learning quite deep, long-range dependencies between k-hop neighbors in the relational schema. Experimentally, ClavaDDPM outperforms existing non-diffusion baselines both in terms of effectiveness and efficiency.
strength_weakness: strength_weakness: **W1.** The idea of long-range dependencies in relational tables has been introduced in the beginning and highlighted in the experiments, but the main part (Section 4) does not go into the details. Perhaps Section 4.2 could expand on long-range modeling mapping to the example hierarchy from Figure 1.

**W2.** No discussion of the limitations, e.g., how long the training took; inference on the datasets takes several days. The checklist mentions “the second paragraph of the Conclusion” but this section in the manuscript has only one paragraph.
questions: questions: **Q1.** Most of the reported numbers (total variation distance between synthetic and real data) are in the higher 95%+ range which makes it hard to distinguish the quality of new models. Are there any other metrics close to real-world tasks that could be of use? For instance, TabDDPM and others report the quality of CatBoost/XGBoost and other standard tabular ML algorithms trained on the synthetic data.

Comments:   
* Line 206: Typo in LavaDDPM    
* Captions of Tables 1 and 2 could also mention what the reported metrics are.
limitations: limitations: There is no discussion of the limitations.

Comments: title: Additional comment on question 9
References
Rebuttal to the Weakness
Rebuttal to the Questions
Please read and respond to the rebuttal
Reference List and Table 2
comment: We would like to additionally address question 9, where the mentioned experiment results are actually located in Appendix D.1. It was a typo for reference and we will fix it in the revision.
Thank you for your response and we appreciate your comments that help refine our paper.

$\quad$

> Reference to Appendix A Notation Summary

Thank you for your feedback. To ensure clarity and consistency, we have included a paragraph at the beginning of Section 4.1 that introduces the relevant notations, such as parent/child tables, the entire database, and the distinctions between data and random variables. This section is complemented by a running example using a subset of the Berka dataset. We appreciate your suggestion and will add a reference to Appendix A for further clarification.

$\quad$

> Issues with Equations 5-9

We would greatly appreciate it if you could point out any specific missing symbols or inconsistencies in Equations 5-9. This would help us further improve the clarity and accuracy of our work.

$\quad$

>Visualization of Label Clustering and Table Relationships

Thank you for your suggestions. We will consider adding some visualizations in the appendix to enhance understanding in the revised version.

$\quad$

>Clarification on Mathematical Deduction

The mathematical derivation spans from Equation 5 to Equation 10, where we derive a probability expression for the parent-child scenario under the assumptions we have applied. The final expression (Equation 10) illustrates the distribution modeled by each component in the ClavaDDPM framework. We believe this strengthens the theoretical foundations of our work, as previous approaches in tabular data synthesis, such as TabDDPM and TabSyn, do not explicitly provide the probability assumptions they rely on or clarify the distributions they model, instead solving the problem in a purely end-to-end manner.

- Equation 5 is a direct result of the i.i.d. assumption on $(g_j, y_j)$.
- Equation 6 is based on conditional probability.
- Equation 7 introduces an independence assumption we are making.
- Equation 8 is a series of conditional probability expansions of Equation 6, given Equation 7.
- Equation 9 expands the conditional foreign key group distribution by introducing the group size variable, which directly extends a part of Equation 8.
- Equation 10 presents the final expression of the parent-child distribution we are modeling, incorporating the independence assumptions introduced in our work.

This final expression involves three probability distributions, each representing a learnable component of the ClavaDDPM framework:
- $p(y, c)$ is the augmented parent distribution, modeled by training a parent diffusion model.
- $p(s|c)$ is the conditional group size distribution, calculated through frequency counting.
- $p(x|c)$ is the conditional child distribution, modeled using classifier-guided diffusion.

We are happy to provide further details and welcome any specific recommendations to improve clarity.
Thank you for your detailed responses and efforts to address my concerns. I appreciate the clarifications provided, particularly regarding the assumptions in your approach. However, after careful consideration, I have decided to maintain my original score. I believe there are still several areas that require further attention:

Reference to Appendix A Notation Summary: I suggest adding explicit references to the notation summary provided in Appendix A within the main text. This will help readers navigate the mathematical formulations more effectively and avoid potential confusion.

Issues with Equations 5-9: I noticed that there are some missing symbols in Equations 5 through 9, which result in these equations not being properly formatted. Addressing these inconsistencies is crucial for the clarity and accuracy of your mathematical presentation.

Visualization of Label Clustering and Table Relationships: It would be beneficial to include some visualizations that illustrate the relationship between label clustering and table structures. Such visual results would provide additional empirical evidence and enhance the understanding of your approach.

Clarification on Mathematical Deduction: You mentioned that you provided a mathematical deduction to strengthen the theoretical foundation of your approach. Could you clarify where and how this deduction is presented in the paper? A clearer explanation or more detailed description would be helpful in understanding the rigor of your theoretical contributions.

I hope these suggestions help further refine your work. Thank you again for your dedication and for considering my feedback. I look forward to seeing the final version of your paper.
Dear reviewer,

Please respond to the rebuttal and provide substantive arguments and, in particular, more concrete strengths and weaknesses of the paper. Unless you update your line of argument, I will not be able to take your review into account in the decision making process.
Dear reviewer,

Please respond to the rebuttal as is expected of you as a NeurIPS reviewer asap! Thanks
Thank you for your time and effort in reviewing our work as well as the rebuttal. We greatly appreciate your constructive feedback and the positive score you've given, which encourages us to further refine the research.
Thank you, after reading the responses to this and other reviews my concerns are resolved and I remain positive about the work.
[1] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.

[2] A. Kotelnikov, D. Baranchuk, I. Rubachev, and A. Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pages 17564–17579. PMLR, 2023.

[3] H. Zhang, J. Zhang, B. Srinivasan, Z. Shen, X. Qin, C. Faloutsos, H. Rangwala, and G. Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. arXiv preprint arXiv:2310.09656, 2023.

[4] K. Cai, X. Xiao, and G. Cormode. Privlava: synthesizing relational data with foreign keys under differential privacy. Proceedings of the ACM on Management of Data, 1(2):1–25, 2023.

[5] Razavi, Ali, Aaron Van den Oord, and Oriol Vinyals. "Generating diverse high-fidelity images with vq-vae-2." Advances in neural information processing systems 32 (2019).
>Although this work attempts to elaborate on its theoretical framework with substantial content, the notation system and descriptive symbols used do not conform to standard academic mathematics conventions (see sections 3. Background and 4.1 Notations).

**R: Non-standard notations**

Thank you for pointing this out. We acknowledge that our notation differs slightly from the standard. In probability theory, capital letters typically denote random variables, while lower case letters represent their actual values. We modified this convention to distinguish between row data and table data. For example, we use $x$ to denote a row of a child table and $X$ to denote the entire child table. To avoid conflicts, we use bold letters $\mathbf{x}$ and $\mathbf{X}$ to represent the corresponding random variables.
To clarify this further, we will describe our design choices in detail and ensure consistency throughout the manuscript in the revised version.

$\quad$

>The marks in equation are not uniform (Eq.2)

**R: Non-uniform marks**

We obtained Equation 2 from the Background section of TabDDPM[2]. We would greatly appreciate it if you could point out the specific issues with the notation or formality, and we are more than happy to make any necessary improvements.

$\quad$

>Furthermore, several key formulas lack thorough proofs (for example, Formula 8). The proof of this key formula would be best placed in the appendix.

**R: Proofs for formula**

We agree with your observation that this is a key formula in our work. The derivation is based on Equation 7. Due to space constraints, we provided only a brief one-line expansion in the main paper. We appreciate your suggestion and will include a detailed proof in the appendix.

$\quad$

>Formula 3 from a previous study lacks supporting evidence; a brief explanation of its proof process or an indication of its rationale is necessary.

**R: Formula 3**

Formula 3 is directly adopted from previous work [1], which was derived using Bayes' theorem. Due to space limitations, we were unable to provide a more detailed explanation in the main paper and included only the result. However, we appreciate your point and will add a more detailed background section in the appendix of the revised version. Additionally, we will briefly mention the Bayesian modeling rationale in the main paper.

$\quad$

>The validity of the method “we introduce latent random variables c such that g is independent of y conditioned on c” also requires theoretical verification.

**R: Validity of assumption**

This is a valid concern. In fact, ClavaDDPM adopts a Bayesian modeling paradigm, where we make certain Bayesian assumptions and let the trained models enforce those assumptions. Therefore, we chose GMM with diagonal covariance, which by design learns latent variables $c$ that enforce conditional independence between $g$ and $y$ (with covariance being zero). We will include a more detailed demonstration of this aspect in the appendix to better address the theoretical foundations.
**A1: Terminologies of table**

Yes, a relation is a table. In this paper, we primarily adhere to the terminology used in relational database literature, where a “table” is formally referred to as a “relation”, and multi-relational synthesis corresponds to multi-table synthesis. Consequently, our notation slightly differs from previous works that only consider single tables, such as TabDDPM[2] and TabSyn[3]. To be clear, we used “tables” and “relations” interchangeably in the paper. Specifically, we model “relations” or “tables” as nodes and “foreign key constraints” as edges, consistent with relational database terminology. We apologize for any confusion and will clarify this further in the revised version.

$\quad$

**A2:I.i.d. assumptions**

We fully understand your concern. In our work, the term “unique identifier” refers to a unique ID for a row  and is not related to the data distribution itself. Below is an example table:

|Primary key(id)|Gender|Country|
|-|-|-|
|2|M|US|
|3|F|UK|

In this table, the primary key (unique identifier) is simply an ID, while the actual data distribution involves attributes like (gender, country). By i.i.d., we mean that the distribution of $(gender, country)$ is independent. When modeling, the primary key is not included as part of the data but is used solely for enforcing foreign key constraints. For example, if a child table row has a foreign key value of $3$, it refers to the row $(F, UK)$ in the parent table. The value $3$ is not passed into the diffusion model and is assigned additionally.

Similarly, previous works in tabular data synthesis, such as SDV, PrivLava, and TabDDPM, treat keys as identifiers rather than data, with key values not being passed into the model. We will clarify this in our revised version to prevent further confusion.

$\quad$

**A3: Latent variable learning**

In the context of tabular data synthesis, PrivLava[4] is the only prior work that has learned latent variables (excluding the works that directly adopt latent generative models, which do not explicitly learn latent variables). Compared to PrivLava, our approach is agnostic to data domain size and has faster convergence. PrivLava's latent learning is marginal-based, making it applicable only to predefined fixed domains (they primarily support integers). Additionally, their use of the EM algorithm is slow and prone to convergence issues in practice. Our experiments in Section 5, Table 1, demonstrate that their method fails to converge on many real-world datasets.

Importantly, our method (diagonal covariance GMM) enforces our independence assumption and aligns with our theoretical analysis. In Section 4.1, Equation 7, we assume that child table groups $g$ are independent of parent row $y$, conditioned on latent variable $c$. Thus, we learn latent variable $c$ such that its distribution supports this assumption. As mentioned in Section 4.3.1, we use a GMM with diagonal covariances in the joint space of $(X, Y)$. This ensures that, when the GMM is properly trained, the covariance between $X$ and $Y$ is zero in each Gaussian cluster with centroid $c$. Consequently, conditioned on $c$, $g$ and $y$ will tend to be independent, as $g$ is a collection of $X$.

While more advanced latent learning methods, such as VQVAE[5], could perform latent learning and quantization simultaneously, we considered them during development but identified potential trade-offs in terms of runtime. Currently, the runtime of the latent learning phase is negligible compared to other phases, and VQVAE lacks synergy with our theoretical assumptions. Nevertheless, as noted at the end of Section 4.3.1, we are open to exploring the impact of other latent learning methods in future work.
Dear reviewers,

First of all, thank you for your service to the ML community. Writing high-quality reviews and engaging with authors' responses is essential to a healthy ML research community. 

It is essential that you read the rebuttals and provide a response and/or follow-up questions within the next few days. This will allow the authors sufficient time to react. While a detailed response addressing all points is not necessary, at a minimum, you should indicate you have read and considered the review and whether you will maintain or revise your score. Please also take the time to read the other reviews. 

I want to thank you again, and I look forward to following the discussions here.
**Reference**

[1] K. Cai, X. Xiao, and G. Cormode. Privlava: synthesizing relational data with foreign keys under differential privacy. Proceedings of the ACM on Management of Data, 1(2):1–25, 2023.

[2]N. Patki, R. Wedge, and K. Veeramachaneni. The synthetic data vault. In 2016 IEEE international conference on data science and advanced analytics (DSAA), pages 399–410. IEEE, 2016.

[3] F. Li, B. Wu, K. Yi, and Z. Zhao. Wander join: Online aggregation via random walks. In Proceedings of the 2016 International Conference on Management of Data, pages 615–629,2016.

[4] C. Ge, S. Mohapatra, X. He, and I. F. Ilyas. Kamino: constraint-aware differentially private data synthesis. Proc. VLDB Endow., 14(10):1886–1899, jun 2021.

[5] T. Stadler, B. Oprisanu, and C. Troncoso. Synthetic data – anonymisation groundhog day. In 31st USENIX Security Symposium (USENIX Security 22), pages 1451–1468, Boston, MA, Aug. 2022. USENIX Association.

[6] A. Kotelnikov, D. Baranchuk, I. Rubachev, and A. Babenko. Tabddpm: Modellingtabular data with diffusion models. In International Conference on MachineLearning, pages 17564–17579. PMLR, 2023.

[7] H. Zhang, J. Zhang, B. Srinivasan, Z. Shen, X. Qin, C. Faloutsos, H. Rangwala, and G. Karypis. Mixed-type tabular data synthesis with score-based diffusion in latent space. arXiv preprint arXiv:2310.09656, 2023.

$\quad$

**Table 2: ClavaDDPM vs ClavaTabsyn**

|Berka|Overall|Col Shape|0-hop|Cardinality|1-hop|
|-|-|-|-|-|-|
|ClavaTabsyn|93.39|88.65|92.83|93.47|98.59|
|ClavaDDPM|96.29|94.56|94.12|97.55|98.93|