GOGGLE: Generative Modelling for Tabular Data by Learning Relational Structure
Authors: Tennison Liu
Zhaozhi Qian
Jeroen Berrevoets
Mihaela van der Schaar
Venue: ICLR 2023 poster (2023)
Keywords: tabular data
synthetic data
generative model

Abstract: Deep generative models learn highly complex and non-linear representations to generate realistic synthetic data. While they have achieved notable success in computer vision and natural language processing, similar advances have been less demonstrable in the tabular domain. This is partially because generative modelling of tabular data entails a particular set of challenges, including heterogeneous relationships, limited number of samples, and difficulties in incorporating prior knowledge. Additionally, unlike their counterparts in image and sequence domain, deep generative models for tabular data almost exclusively employ fully-connected layers, which encode weak inductive biases about relationships between inputs. Real-world data generating processes can often be represented using relational structures, which encode sparse, heterogeneous relationships between variables. In this work, we learn and exploit relational structure underlying tabular data to better model variable dependence, and as a natural means to introduce regularization on relationships and include prior knowledge. Specifically, we introduce GOGGLE, an end-to-end message passing scheme that jointly learns the relational structure and corresponding functional relationships as the basis of generating synthetic samples. Using real-world datasets, we provide empirical evidence that the proposed method is effective in generating realistic synthetic data and exploiting domain knowledge for downstream tasks. 

Published as a conference paper at ICLR 2023
GOGGLE: GENERATIVE MODELLING FOR TABULAR DATA BY LEARNING RELATIONAL STRUCTURE
Tennison Liu University of Cambridge t1 t15220cam ac.uk
Zhaozhi Qian University of Cambridge zq224 cam ac. uk
Jeroen Berrevoets
University of Cambridge jb23 4 cam ac uk
Mihaela van der Schaar University of Cambridge Alan Turing Institute mv4720 C cam. ac uk
ABSTRACT
Deep generative models learn highly complex and non-linear representations generate realistic synthetic data. While they have achieved notable success in computer vision and natural language processing, similar advances have been less demonstrable in the tabular domain. This is partially because generative modelling tabular data entails particular set of challenges, including heterogeneous relationships, limited number of samples, and difficulties in incorporating prior knowledge. Additionally, unlike their counterparts in image and sequence domain, deep generative models for tabular data almost exclusively employ fully-connected layers, which encode weak inductive biases about relationships between inputs. Real-world data generating processes can often be represented using relational structures, which encode sparse, heterogencous relationships between variables. In this work, we learn and exploit relational structure underlying tabular data (where typical dimensionality d < 100) to better model variable dependence, and as natural means to introduce regularization on relationships and include prior knowledge. Specifically, we introduce GOGGLE, an end-to-ent message passing scheme that jointly learns the relational structure and corresponding functional relationships as the basis of generating synthetic samples. Using real-world datasets, we provide empirical evidence that the proposed method is effective in generating realistie synthetic data and exploiting domain knowledge for downstream tasks.
INTRODUCTION
Learning generative models for synthetic data is an important area in machine learning with many applications. For example, synthetic data be used to simulate settings where real data is scarce unavailable [7, 10]. support better supervised learning by increasing quality in datasets [8], improving robustness and predictive performance [69, 60]. and promoting fairness [72]. Additionally, synthetic data is increasingly being used to overcome usage restrictions while preserving privacy [32, 78, 54].
Deep generative models have achieved notable success in approximating complicated, high- dimensional distributions as encountered in computer vision, natural language processing, and more [46]. A key contributor to this success is that learning architectures can easily exploit relational inductive bias that enhance the learning of joint distributions. Informally, relational inductive biases encode assumptions about the relational structure, which describes variables, and their relationships [3]. For example, image variables (pixels) have high covariance within a local region and relational rules that are invariant across
which are exploited by kernels convolutional neural networks (CNN) to better model image distributions [42]. Similarly, sequence variables are highly dependent on sequentiality and relational rules are invariant across time-steps- - recurrence relations leveraged by recurrent neural networks (RNNs) to capture distributions over time [27].
In this work, we hope to exploit similar relational inductive biases to better model real-world tabular datasets (where typical dimensionality d 100). However, while images and sequences

---

Published as a conference paper at ICLR 2023
homogeneous data formats with known relational structure, tabular data commonly contain more heterogeneous relationships (e.g. variables are only correlated with a small subset of other variables), where the exact relational structure is obscured by obvious relational structure, deep generative models almost exclusively employ multilayer perceptrons (MIP) learn representations tabular data. This is less than ideal MLPs encode virtually no relational
knowledge [62]. Without an
indeed, all variables can interact to determine any other variable's b value.
However, this all-to-all relational structure is often unnecessary, the data generating process (DGP) of tabular data is better described using sparse relational structures [39, 4J. Variable dependencies can be more accurately captured by considering them as edges and learning representations over the resulting relational structure. We hypothesize that generative models exploiting the relational structure can more adequately address certain challenges that arise during modelling distributions for tabular data, including heterogeneous relationships between variables, smaller datasets, which are more prone to overfitting, and the lack of mechanism to incorporate prior knowledge that can improve modelling performance.
Contributions. We introduce Generative MOdellinG with Graph LEarning (GOGGLE), an end-to-end framework that learns an approximate relational structure as the foundation of generative modelling. More specifically, we devise a general message passing scheme that models tabular data by jointly learning (1) the relational structure and (2) the corresponding functional relationships (dependencies) in the learned structure. To the best of our knowledge, this is the first work to jointly learn the relational structure and the parameters of generative model to model tabular data. Additionally, we propose regularization on variable dependencies to reduce model overfitting on smaller tabular datasets and propose simple mechanism to include prior knowledge into the generative process.
We demonstrate the advantages of our approach in series of experiments on multiple real-world datasets. We employ both qualitative and quantitative approaches to demonstrate that GOGGLE achieves consistent improvements over state-of-the-an benchmarks in generating synthetic data and exploiting prior knowledge for better downstream performance.
2 CHALLENGES IN TABULAR DATA GENERATION
While deep generative models have seen notable success in image and sequence domains, tabular data ubiquitous in many salient applications. including medicine, finance, and economies. Generative modelling for tabular data presents a distinct set of challenges, which are largely open research questions. Here, we highlight them in turn:
1. Complicated relational structure. Tabular data commonly contain heterogencous relational structures, including sparse dependencies (variables only dependent on a small subset of other vari- ables), and heterogeneous functional relationships (dependencies) between variables [62]. Unlike in images and sequences, where the relational structures (locality and sequentiality respectively) are better understood (and arguably generalizable), variable dependencies in tabular datasets are domain specific and rarely known.
2. Overlitting and memorization. Modern deep generative models are over- over-paramen parameterized, thus requiring large datasets to learn the underlying distribution without overfitting [1]. This is espe- cially demanding tabular datasets that smaller and where collection is difficult/expensive. Real-world DGPs are sparse in structure, and we address overfitting concerns by enforcing sparsity in variable dependencies, thus achieving regularization effect by restricting the hypothesis space. Additionally, as variables can only be generated using their neighborhoods, the model is incentivized to find informative neighbors.
3. Domain knowledge, In many fields, such as medicine or social sciences, we have rich domain knowledge on dependence between variables, sparsity, or importance of specific variables (i.e. de- gree of connectivity [74]). Incorporating prior knowledge is especially useful in practical settings where we may not have large datasets but can obtain expert knowledge to aid model learning. As far we know, this is capability that is currently lacking in tabular deep generative models. Our generative process takes into account the relational structure, allowing diverse range of (partial) domain knowledge to be incorporated.
distinction. We emphasize that the goal of our work is not probabilistic structure discovery, which aims to discover the unique probabilistic graph from observed data [15, 85). As there is
2

---

Published as a conference paper at ICLR 2023
Table 1: Overview of generative models in the tabular domain. Comparisons are made on underlying model class and deep learning module, generative model distribution pe, and the following desiderata: model is capable of (1) generating in-distribution samples. (2) regularizing variable dependencies, (3) incorporating prior knowledge.
Model Classl Module
Model
Generative model Pe
(2)
Non-neural methods
BN
BN / None
I pa [Pa(z.)
V
MX
MM None
2k=1 ( o2:0)
v
x
Neural methods
Copul sANN
GAN / MLP
x
1ableGAN
GAN/ / MLP
J (= p(=) Pz
/
x
x
CTGAN
GAN MLP
x
TVAE
VAE/ MLP
x
GOGGLE
VAB/ MPNN
II: (0)
v
generally set of plausible graphs that have equal likelihood on the training data, such methods make assumptions about the graph type, variable distribution, and functional relations, which are necessary to disambiguate a unique graph. In contrast, the aim of GOGGLE is to learn approximate relational structure to guide generative modelling (see Appendix D).
RELATED WORKS
Traditionally, generative modelling was performed through explicit parameterisation and estimation of a joint probability distributions. Examples of this include Bayesian networks (BN) [82, 33], mixture models (MM) [56, 29] and copulas (CP) [68, 49]. These models are limited in more complex distributions, where it is difficult to learn high- high-dimensional dependencies correctly.
Recent advances in deep generative modelling (including VAEs [35], GANs [23], normalizing flows (NFs) [55] have seen models that can generate realistic synthetic data from complex, high-dimensional distributions. Specifically for tabular synthetic data, [76] introduced CTGAN and TVAE, which are GAN-based and VAE-based models respectively that tackled many practical issues of modelling tabular data, including multimodality and mixed data types. Similarly, TTableGAN [47] employed GAN framework with an auxiliary classifier to predict the label of generated sample. medGAN [10] and ehrGAN [7] are more specialist methods developed for healthcare specifically. Instead of generating synthetic data, neural methods have also been employed to perform data imputation [22, 80). Perhaps most similar to our work, [37] (and its tabular variant [75]) learn causal graph in the representation space to generate synthetic data, but assume access to the true causal graph.
The aforementioned methods resort to MLPs to model complex dependencies in an all-to-all fashion. Additionally, they employ regularization the weight space (e.g. L2 regularization [58] and Dropout [666) to reduce overfitting. These effects, as we will discover in 85, are unlikely to be optimal for generative performance. Our work is also related to the field of relational inference, which seeks to infer relationships between objects from observation data alone. Representative works include [36], which seeks to infer interactions between objects in interacting systems, and [24, 25) that infers relationships from omic interactions. In these works, correctly recovered relational structure is the object of inference. This stands in stark contrast to our work, where a partially correct structure is satisfactory for our purposes. Indeed, as we shall show later, even learning partially correct structure can greatly improve synthetic data performance (see Appendix D for further discussion). The key contribution of this work is in learning the relational structure jointly with deep representation learning modules to improve generative modelling. We provide an overview in Table 1, evaluating related methods based on their ability to address the previously described challenges.
4 GOGGLE: GENERATIVE MODELLING WITH GRAPH LEARNING
4.1 PROBLEM FORMULATION
Generative modelling. Generative modelling for tabular data attempts to uncover probability distribution px over X E X' C Rd! We have access to a training dataset D, which consists of N ii.d. samples ~ px. The goal is to learn the parameters 0 of generative model such that the
Generative modilling for hime--iia datrsets, thrse encsuntered in genoenics (where di typically 1000) outside the scope this work.
3

---

Published as a conference paper at ICLR 2023
Training
Inference
Initialise GΦ
<<(x))
Fa(e: Gp)
Upelato Graph Go
Trxined Graph 6o
Loss Function
AXUP
NNAP
K&dl
Update &o 8' (F(((() +
Traindd Model Fa
Dounds of Monsage Passing
2
Loss Function
R(Gp) Knowledge
4()
Initialise Fg
Figure 1: Key components of GOGGLE Framework. ! Simultancous learning of relational structure Go and F6 s.t. generative process respects relational structure. 2 Injection of prior knowledge and regularization on variable dependence. Synthetic sample generated using
Fo (z; Go) 2 pz.
model distribution pe is close to px- Most deep generative models employ set of noise variables Z E Z R", which follows tractable distribution pz E Pz (e.g. Gaussian distribution). Instead of learning px directly, the model learns mapping has the same distribution as X. The mapping go can either surjective function (as in GANs [23]), a bijective function (as in normalizing flows [55]), or a stochastic mapping (as in VAEs [35]) [45].
2 + X where the transformed variable ga (z)
Relational structures as graphs. graph G (X, E) is a representation the relational structure, where X denotes random variables and E' edges that encode relationships between variables. The most general type of graph is a mixed graphs, which contain both directed and undirected edges, where an undirected edge e.g. X; Xx, is encoded as two directed edges X; - X, and X, + X; Importantly, a graph admits a sparse and compact representation, which underlies many real-world generative processes [6, 41, 20]. Specifically, we say that variable X; is only dependent on other variables with directed edges into X,, (i)) = {X, : (X; + X₂) E E}. light of simplicity, we will refer to (() as the neighborhood of X,. Formally, by a specific functional relationship: X, (i()()),e)) E,). where E is a noise variable.
state that each variable is determined
4.2 OVERVIEW
We hypothesize that sparse dependencies in a tabular dataset can be more accurately captured through relational structure. To that end, we propose a novel framework of generative modelling that is guided by an underlying relational structure. While tabular generative models are conventionally constructed with MIPs, we introduce message passing scheme that operates on learned graph, better capturing heterogeneous variable relationships, and allowing regularization and prior enowledge to be injected through the adjacency matrix. As the underlying structure is not known a priori (or partially observed at best) in realistic settings, we design an end-to-end generative model to learn the relational structure simultaneously with the corresponding functional relationships.
Figure provides a schematic overview GOGGLE. The four key components are: A learnable relational structure, which is parameterized by graph and represented using a weighted adjacency matrix G& that indicates dependence between variables; & generative model (- that maps from noise vector z pz such that the transformed variable (2; G&) has the same distribution as X. The way the relational structure and the generative model interacts is that the generative model only allows the of relations deemed important by the learned relational structure G influence the generation of a variable. Additionally, there are the & loss function C(-) and regularization term RRG,)) used train the model and encode prior knowledge, respectively.
4.3 LEARNING THE RELATIONAL STRUCTURE
Learning the relational structure entails exploring possible adjacency matrices based on relationships between variables. In the absence of informative prior knowledge, we make the key (but minimal) assumption of sparsity (viz. Assumption 1), that the relational structure is sparser than a fully-
4

---

Published as a conference paper at ICLR 2023
connected graph. Intuitively, one can consider MLPs that represent all-to-all relationships as being described by a fully-connecte relational structure. The sparsity assumption encourages learning of a sparse structure that also determines the most informative neighborhood that generates each variable with the highest likelihood. When we have some prior knowledge about variable dependencies, we will incorporate it through the adjacency matrix and regularization term in the loss function.
Assumption (Graph sparsity), We assume there exists a graph G such that the adjacency matrix A = A(G) {0, 1} dxd has at least one instance where Aij Aji 0, indicating variables X; and X3 are not connected.
In our graphs, nodes are random variables and edges denote dependence between them. We represent an undirected edge X; - Xj as two directed edges, one from X; + Xj, and one from X; - X, Specifically, we learn the entries in the unnormalized adjacency matrix, Go E Raaxd which is then normalized through an element-wise sigmoid(; c(-) function to obtain G E [0, [0,1]axd. Intuitively, Goli.j] represents the strength of the dependence between X; + Xy.
4.4 LEARNING THE FUNCTIONAL. RELATIONSHIPS
The generative process must be consistent with the relational dependencies specified by the learned relational structure Ga- We learn a generative model Fo 2 + X' by learning the mapping from a tractable noise distribution (that can be easily sampled from) to the data distribution. For the ease of exposition, we can informally say that the generative model describes the set of functional relationships between variables F'g = (fi, fa), where each f; describes how X; depends on its neighborhood (1) in Gé.
Learning Fg leads to two challenges. Computational complexity: G is continuously updated during training, possibly resulting in different dependencies (different set of d functional relationships) that has to be learned, limiting the scalability of the approach. P Cycles in the graph: Consider undirected edge between two variables X, Xj- In this example, X; is in the neighborhood of Xz, and the generation of X, will have to depend on the values of Xy. The reverse is also true. This highlights that (-(-) should be able work with potential cycles in the Gs to ensure the functional relationships in the model are faithful to the learned graph.
Learning challenges.
A flexible parameterization. To address these challenges, we employ message passing neural network (MPNN) as a flexible and expressive parameterisation of the functional relationships model [21]. This scheme addresses the aforementioned challenges by naturally handling cycles in the graph and relaxing the computational burden of learning functional relations by applying common information propagation procedure. The proposed message passing scheme performs I. rounds of
message passing and generates each variable 2; from an initial embedding h, which is constructed using the noise term 24. Each round [L] of message passing is defined in terms of a message function a (!)) (-), an aggregation function @ (500)) (-) and an update function (-).
During each round I, a message m ! is constructed for each of the variables in the neighborhood Yj (i),, using the message function o!))
m ( (1) h1g (n)
Then, the aggregation function (9) combines all incoming messages from the neighborhood through weighing each individual message by the learned weight in Gø:
h Win a (IGy.m)) (1) lj VJEN(())
The variable embeddings h are then updated using the aggregated neighborhood messages and its previous embedding h: through the update function ("))
(I-1)
h = 70 (Roio) n--1))
Each round of message passing exploits the relationships and sparsity in the learned structure by generating each variable using solely information from variables it depends on.
One could also interpect Ga (f probability edge existing from X, X, and obtuin bintry adjocency by sampling, where
pormmeteos learnal through Clumbel-Softuux trick [3I].
5

---

Published as a conference paper at ICLR 2023
Generative Model (2; GΦ)
Learned Graph Gg
fo,
A
MPNN Round
< lx)
I Houud
piu (N 1)
fog
Iuput Fentures
Emmbaiding
@0o
Reconatrcction
Latent Voctor
Mossage Passing Round
Figure 2: Generative architecture. The generative model is consistent with the learned relational structure by generating each variable using its neighborhood. GOGGLE adopts a VAE architecture with MPNN to gradually generate synthetic data through I. rounds of message passing.
To obtain the initial node embeddings 1,0), we add the variable index as one-hot en- coded vector the latent variable, which is then transformed using an embedding function, i.e. h. (0) = fo, (concat(z; 11:)) Here, 1; is the one-hot vector for the variable i, concat [-] is the concatenation function and fa, (-) is the embedding function implemented using single-layer MLP. This transformation encodes meaningful variable-specific information in messages and embed- dings. Using this scheme, the learning problem is drastically simplified from learning functional relationships in each round to just two message passing functions.
Lastly, σ()) and 700) are arbitrary, learnable functions, and @ is permutation invaariant function, V E I.). The node embeddings after the last round are taken to be the generation, 4t ( The number of rounds T, of message passing, the choice of Y C and the construction of the initial embeddings ( are design choices that should be made to suit each task specifically. Additionally, the scheme can be synchromous, which is useful when all the variables are being generated jointly, or asynchronous, which is more suitable when the variables are being generated sequentially.
including mean, max pooling, or sum. In other words, Fg = {(x(0) (()) (
4.5 PUTTING IT TOGETHER
We adopt a VAE style architecture [35] and perform amortized inference on the initial noise vectors." Specifically, the noise vector sampled from an encoder using the reparameterization trick 2 0 (0, I). The functional relationships model Fo(-) plays the role of the decoder to generate -= Fo(z; G ~ p(a[z). The exact design of the method is visually illustrated in Figure
Loss function and regularization. The generative model and learnable graph are jointly trained by combining the classic ELBO loss function with a graph regularization term R(Gg). The training objective is described below, where À is hyperparameter controlling the regularization strength.
J(0 Φ) = ELBO X; ()) + XR(G₂)
[x()))) [In po (x Go)]
DrL D XR(Ga)
Likelihood
KL
Regularisation
A key advantage of GOGGLE is that can casily incorporate prior knowledge about the graph through the regularization term RGG)) .This term will encourage the adjacency matrix to have certain characteristics. In general, sparsity prior R(G,) ||Grllp will reward sparse graphs, and partial knowledge of feature relationships can be encoded through the prior R(G,) ) = I[& - Go lp- where ||. |lr denotes the p-norm. We provide more thorough discussion on the types of prior knowledge that can be incorporated in Appendix B.
Training and generation. We implement message construction and embedding update func- tions using single-layer ReLU-acttivted MLP. Specifically, (()) ReLU (w xxf(-1) and (1())
(0) Ret21 (w ccomeut Maa A. "I). Here, Wm and W,, are weight matrices of their respective MI.Ps. Additionally, we aggregate messages using the mean, which can be viewed as taking a weighted average over messages from neighborhood.
We note while consider VAB architecture, appropriate generative architecture functions (eg. GANs, nummalizing flows) should designed with application in mind and uuch extensions are ficure works.

---

Published as a conference paper at ICLR 2023
We use standard normal prior for pz NO, I). The adjacency matrix is initialized as fully- connected graph, where edge weights are gradually refined during learning. In practice, we apply hard threshold on the learned adjacency matrices, where entries < 0.1 are zeroed out (similar to ReI.U activation). In the absence of prior knowledge, we impose a sparsity penalty R(Ga) = ||Ga] Once the training is complete, synthetic sample be generated by first sampling noise vector pz and passing it through the functional relationships model: I = Fo(2; Go).
2
4.6 A REMARK ON DATA AUGMENTATION
Moreover, and this might be of independent interest, we show that by using the blueprint presented, we can also construct generative model that can perform data augmentation. We consider data augmentation as the ability to conditionally generate synthetic data when conditioned on any variable to be a specific value, i.c. ã Fø(z, X; z4; Gg). To do so, assume the DGP can be represented by a directed, acyclic graph (DAG), meaning that variables are generated sequentially (asynchronous message passing), following topological order &G, obtained from the learned graph. Specifically,
the generation of each variable is described through a structural equation model: ã; = f( 24) [50]. During generation, can condition specific variables on particular value to allow sampling from the conditional distribution. We elaborate more on this method in Appendix A.
5 EXPERIMENTS
The core claim in this work is that exploring relational inductive biases can better capture the sparsity and heterogeneous relationships in tabular data, and consequentially lead to enhanced learning of generative models. We quantitatively evaluate aspects of our method to support this claim:
1. Synthetic data quality: How good is the synthetic data? $5.1 quantitatively evaluates the characteristics of the synthetic data with respect to a variety of state-of-the-art benchmarks.
Prior knowledge: Does prior knowledge improve performance? $5.2 evaluates whether prior knowledge about variable dependence can improve generation performance.
Gains: Why does work? $5.3 investigates the dynamics of relational structure learning and generative model, and to what extent the relational structure contributes to performance gains.
In the interest of limited space, we attach additional results in Appendix C. Specifically, we include: 4. Additional datasets: evaluating synthetic data performance on 6 more datasets; 5. t-SNE and graph visualizations: I-SNE visualization on synthetic datasets to qualitatively investigate quality [73]; we also examine the graphs learned in experiments; 6. Sensitivity analysis: to better evaluate the performance our method on different dataset sizes and number of features; Data augmentation:
assesses an alternative implementation of our method to perform data augmentation.
Benchmarks. We compare against state-of-the-ar tabular synthetic data models, including Bayesian Networks (BN) [51] and GAN-based models: CTGAN [76] and TableGAN [47]; VAE-based models:
TVAE [49]; and normalising flows NFLOW [55].
Following the experiment design in recent works [49, 76], we employ 8 real-world datasets from the UCI repository [16], and 2 datasets from the BN repository [39]. We employ datasets with different number of samples (ranging from 569 to 581012) and different feature counts (ranging from 12 to 168) to gain a better understanding of our method's performance profile. We provide additional information about benchmarks, datasets, hyperparameters, and evaluation methods in Appendix B. For all results, we report mean + std averaged over 10 runs. Our code is provided on GitHub."
5.1 EVALUATION OF SYNTHETIC DATA
To assess the quality of synthetic data, we observe three desiderata (similar to [79, 76)): qual- ity myy-saplles should be realistic, cover the data distribution, and be generalized. We evaluate using the three-dimensional metric (a-precision, 3-recall, authenticity) proposed in [1], reporting their average. iiteectionssmplle
should be indistinguishable from the real data. We report the AU- ROC performance of three post-hoc classifiers to distinguish real and generated samples. Finally, P Serity--samplss should be just useful the real data on predictive tasks (i.e. rain-on--yntheeic
ppt thi. con't ison't fa/coBeLiy het past /// wtom frender acchwe Teb//0o00
7

---

Published as a conference paper at ICLR 2023
Table 2: Quality, detection, and utility of synthetic data. Bold indicates the best performance.
Dataset
Adult
Breast
Covertype
Credit
0.04
0.59 0.02
0.59 = 0.02
0.61 0.03
MX
0.58 0.00
0.48 = 0.03
0.52 = 0.06
0.37 + 0.03
Quality (?) (higher is betler)
CTEAN
0.37 : 0.01
0.58 = 0.02
0.59 = 0.04
0.61 + 0.07
TableCAN
0.59 0.00
0.35 1: 0.06
0.63 1 0.04
0.61 1 0.06
TVAE
0.03
0.60 = 0.07
0.53 0.03
0.02
0.03
0.58 0.02
0.38 0.03
GOGGTRR
0.62 - 0.02
0.61 + 0.01 0.01
0.59 + 0.06
0.61 + 0.07
SN
0.69 0.03
0.69 6.02 6.02
0.66 = 0.03
0.70
0.04
0.70 0.01
0.80
0.75
Detection (L)
CTEAN
0.72 = 0.03
0.71 1 0.04
better)
TablaCAN
0.79 1
TVAE
0.04
0.72
0.72 0.05
0.77 0.03
NFLOW
0.78 0.00
0.T2
0.79 0.03
0.74
GOGGTR
0.68 0.004
0.68 0.03
0.72
0.69
3N
-0.01
0.00
0.06 0.00
-0.06
MX
-0.06 0.001
-0.08
-0.12
00.10 0.02
Utility (
CTEAN
-0.05 0.00
-0.06
-0.06 0.01
-0.05 0.03
(higher is hetter)
Tabl Tab1
0.04
-0.31 0.08
0.18 0.04
-0.21 0.05
NFLOW
0.04 0.01
-0.07 0.01
0.08 1 0.02
0.13 1. 0.03
GOGSLE
0.01 0.00
0.04 0.01
0.05 = 0.02
0.04 0.01
test-on-real) [19]. Utility is evaluated based average AUROC change of three downstream prediction models trained on synthetic data relative to real data.
As indicated in Table 2, GOGGLE consistently generates higher quality and lower detection synthetic data relative to benchmarks. This im- provement especially noticeable on smaller datasets (i.e. breast and credit). In this regime, conventional generative models will overfit to the limited Dirain, whereas GOGGLE exploits sparsity to achieve better generalization perfor- mance. We note that BN emerge as our strongest competitors in this setting (also noted in [76]),
Table 3: Average rank of models. GOGGLE con- sistently achieves superior synthetic data quality.
Quality (f) Detection ()) Utility (f)
CTGAN
4.9
4.1
4.1
TableGAN
4.8
4.8
IVAE
3.6
4.1
3.1
EN NFLOW
1.7
2.1
2.3
2.9
3.5
3.2
GOGGLE
1.1
1.4
1.3
frequently out-performnng deep generative models. The logical hypothesis is that BN generalize well to smaller datasets due to their stricter assumptions on the underlying graph, adding strong regularization effect. However, on larger-scale datasets, learning a high-quality BN more diffi- cult. GCGGLE, however, maintains superior performance on larger, more complex datasets our assumptions are less restrictive. On the utility aspect, indicates superior predictive performance on datasets. Remarkable, the predictive scores GOGGLE's synthetic data are very close those real data. We evaluate on additional datasets Appendix C.I, and reported the average rank of each method across all 10 datasets in Table 3.
5.2 GENERATION WITH PRIOR KNOWLEDGE
Next, we investigate whether incorporating partial knowledge through the relational structure can lead to enhanced performance. Specifically, we regularize the learned graph G with partial prior knowledge Go. i.e. Rg = IIG - Goll1- This form of incorporating prior knowledge reflects that we are not completely confident in our prior knowledge.
We evaluate on ECOLI an MAGIC, where we know the true underlying graph a-prior. Specifically, we investigate three settings: zero knowledge GOGGLEd, where no domain knowledge used, partial knowledge GOGGLE50. where 50% of edges are known (by randomly sub-sampling 50% of the edges), and complete knowledge GOGGLE100 where the complete graph known. We are interested in evaluating in each setting when different amounts of data are available to train model. We visually depict the effect of prior knowledge on synthetic data quality in Figure 3.
The DGPs for both datasets (known a-prior) are Gaussian Bayesian Networks, matching exactly the assumptions the BN baseline, giving BN the upper hand as they have the correct model specification. GOGGLE, even with partial knowledge, achieves significant performance gain when prior knowledge is employed. The performance gains from prior knowledge are especially noticeable in lower data regimes, i.e. when Drrain 500 samples. This highlights a key advantage of
8

---

Published as a conference paper at ICLR 2023
Table 4: Ablation study. Bold indicates the best performance.
Adult
Credit
Quality (1)
Detection (L)
Utiiy (1)
Quality (1)
Delection (1)
Utiiiyy (1)
GOGGLE DR
0.41 + 0.07
0.81 + 0.05
-0.54 = 0.08
0.46 = 0.03
0.71 # 0.03
-0.03 0.01
SOGELB-COV
0.06
0.74 0.05
-0.07
0.53 0.04
0.67 0.07
0.01
GOOGLE-aN
0.56 1: 0.04
0.69 1: 0.05
-0.05 1 0.01
0.41 1 0.09
0.67 1 0.04
-0.06 0.00
GOSGL DENSS
0.49 0.09
0.89 +.00
-0.14
0.56 + 0.03
0.62 + 0.08
-0.07
GOGGLE-NKI
0.53 1: 0.07
0.72 4: 0.08
00.15 1 0.03
0.60
4 0.04
0.65
1 0.05
-0.03 0.000
GOGGLE
0.62 + 0.02
0.69
+ 0.03
0.01
= 0.00
0.60
= 0.06
0.70
0.002
-0.01:0.00
GOGGLE,
GOGGLEw
GOGGLE.
CTGAN
NFLOW
Quality (f) (higher is bettee)
Detection
Utility (f)
(kower is better)
(higher is bester)
01E9
0.55
11:2
-0.3
0.55
291
301
1062
2000
Datases Size
Ditisee Sizee
Dataees Size
Figure 3: Prior knowledge and generation. ECOLI (top row) and MAGIC (bottom row).
model-to the best of our knowledge, it is the first generative model that can leverage prior knowledge to generate more realistic synthetic samples.
5.3 ABLATION STUDY
GOGGLE is designed with the joint learning of both graph and functional relationships model. Having empirically demonstrated strong overall results, an immediate question is how important the dynamics of the two parts are for performance. Specifically, we consider the performance gain due to joint learning over learning the two parts separately. This includes the where graph is first learned separately: (1) sparse graph initialized using a sparse Erdos-Renyi random graph, with 10% connectivity (GOGGLE-ER [18], (2) initialized using corre-oorrelttion matrix between variables (GOCGLE-COV), (3) learned using the PC algorithm (GOGGLE-EN) [64]. We also consider the case (4) when graph is not learned but initialized as a fully connected dense graph (GOGGLE-DENSE); and (5) when the functional relationships model is replaced by predictor that generates each variable using its neighborhood (GOGGLE-NMP) We perform an ablation study to report the performance achieved by each of these settings in Table 4.
We empirically observe that, the joint learning of both components is crucial to achieve consistently good performance. More specifically, we note that GOGGLE-ER achieves the worst performance across all settings. This is expected as the graph randomly initialized. GOGGLE-COO learns an undirected graph, and only considers first order dependencies, leading to worse performance, while GOGGLE-BI learns a directed graph, which is too restrictive an inductive bias incorrect. GOGGLE-DENSS employs fully connected graph, and is equivalent to MLP-based generative models. While it achieved lower detection on Credit, it is prone to overfit, resulting in lower quality and utility scores. Lastly, GOGGLE.-NMP generates each variable using only its immediate neighbors and ignores information from variables more than Z-hop away, leading to worse performance.
6 DISCUSSION
In summary, we proposed GOGGLE, a novel tabular data generative model that jointly learns the relational structure and functional relationships through message passing scheme to better capture the sparsity and heterogeneous relationships in tabular data. The explicit use of relational structure to guide generation allows prior knowledge and regularization on variable dependencies to be directly modelled. Limitations and future work. In this work, we focus on tabular data regime, where < 100, future works can focus on higher-dimensional tabular data (e.g- genomics). To scale to higher-dimensions, we suggest extending the relational structure learning to the representation space for better generative models [43].

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank the anonymous ICLR reviewers as well as members of the van der Schaar lab for many insightful comments and suggestions. Tennison Liu would like to thank AstraZeneca for their sponsorship and support. Jeroen Berrevoets thanks W.D. Armstrong Trust for their support. This work is also supported by the National Science Foundation (NSF, grant number 1722516) and the Office of Naval Research (ONR).
ETHICS AND REPRODUCIBILITY STATEMENT
Ethics. This paper addresses fundamental and well-established problem in MI.. Synthetic data has the potential for social good, by allowing unfettered access to otherwise private data, an aspect which should be explored in future works. However, we provide word of caution as synthetic data will inherit potential bias (if any) in the original dataset and inclusion of incorrect prior knowledge can lead to unintended downstream effects.
Reproducibility. We detailed exact implementation details, including dataset preprocessing. im- plementation of benchmark methods, architecture design, hyperparameter tuning, and evalua- tion methods in Section 4, Section 5, and Appendix B. All datasets used in this work can be downloaded from the UCI repository [16] and we provide the code to produce our results hawllluuu///aaaaaa /GOGGE and the wider lab repository https:
at https::////t /ttp:::s//gttuu.b../ wttp:///.o...mmteonooooo/icoccee //g.ahaa..cm//aaadeeeeeaaaaabbb
REFERENCES
[1] Alaa, A. M., van Breugel, B., Saveliev, E., and van der Schaar, M. (2021). How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. arXiv preprint
[2] Banerjee, O., El Ghaoui, L., and d' Aspremont, A. (2008). Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data. The Journal of Machine Learning Research, 9:485-516.
[3] Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez- Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., al. (2018). Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: (8:::8006000 arXiv:/806. 01261.
[4] Berrevoets, J.. Kacprzyk, K., Qian, Z.. and van der Schaar, M. (2022). Navigating causal deep learning. arXiv preprint arXiv:2212.00911
[5] Blackard, J. A. and Dean, D. J. (1999). Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and electronics in agriculture, 13151:131-15.
[6] Bullmore, E. T. and Bassett, D. S. (2011). Brain graphs: graphical models of the human brain connectome. Annual review of clinical psychology. 7:113-140.
[7] Che, Z., Cheng, Y., Zhai, S., Sun, Z., and Liu, Y. (2017). Boosting deep learning risk prediction with generative adversarial networks for electronic health records. In 2017 IEEE International Conference on Data Mining (ICDM), pages 787-792. IEEE.
[8] Chen, R. J., Lu, M. Y., Chen, T. Y., Williamson, D. F., and Mahmood, (2021). Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering, 5(6):493-497.
[9] Chickering, D. M. (2002). Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov)5007554.
[10] Choi, E., Biswal, S., Malin, B., Duke, J., Stewart, W. F. and Sun, J. (2017). Generating multi-label discrete patient records using generative adversarial networks. In Machine learning for healthcare conference, pages 385-305. PMLR.
10

---

Published as a conference paper at ICLR 2023
[11] Chow, C. and Liu, C. (1968). Approximating discrete probability distributions with dependence trees. IEEE transactions on Information Theory, 14(3):462 467.
[12] Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, (2009). Modeling wine preferences
by data mining from physicochemical properties. Decision support systems, 47(4):547-553.
[13] de Jongh, M. and Druzdzel, M. (2009). A comparison of structural distance measures for causal bayesian network models. Recent Advances in Intelligent Information Systems, Challenging Problems of Science, Computer Science series, pages 443-456.
[14) DeVries, T. and Taylor, G. W. (2017). Dataset augmentation feature space. arXiv preprint arXiv:/702.05.0
[15] Drton, M. and Maathuis, M. H. (2017). Structure learning in graphical modeling. Annual Review of Statistics and Its Application, 4:365-393.
[16] Dua, D. and Graff, C. (2017). UCI machine learning repository.
[17] Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. (2019). Neural spline flows. Advances in neural information processing systems, 32.
[18] Erdös, P., Rényi, A., al. (1960). On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 5(1):17-60.
[19] Esteban, C., Hyland, S. L., and Rätsch, G. (2017). Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint 1rX6iv1706.022: 02663.
[20] Farasat, A., Nikolaev, A., Srihari, S. N., and Blair, R. H. (2015). Probabilistic graphical models in modern social network analysis. Social Network Analysis and Mining, 5(1):1-18.
[21] Gilmer, J., Schoenholz, S. S., Riley, F., Vinyals, O., and Dahl, G. E. (2017). Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR.
[22] Gondara, I. and Wang, K. (2018). Mida: Multiple imputation using denoising autoencoders. In Pacific-Asia conference on knowledge discovery and data mining, pages 260-272. Springer.
[23] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley. D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27.
[24] Hajiramezanali, E., Hasanzadeh, A., Duffield, N., Narayanan, K., and Qian, X. (2020). Bayrel: Bayesian relational learning for multi-omics data integration. Advances Neural Information Processing Systems, 33:19251-19263.
[25] Hasanzadeh, A., Hajiramezanali, E., Duffield, N., and Qian, X. (2022). Morel: Multi-omics relational leamingg arXiv preprint rxXiv:2003.08149
[26] Higuera, C., Gardiner, K. J., and Cios, K. J. (2015). Self-Organizing Feature Maps Identify Proteins Critical to Learning in a Mouse Model of Down Syndrome. PLOS ONE, 1066)e0129126. Publisher: Public Library of Science.
[27] Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 18):17351
[28] Hofmann, H. (1994). Statlog (german credit data) data set. UCI Repository of Machine Learning Databases, 53.
[29] Hu, J., Reiter, J. P., and Wang, Q (2018). Dirichlet process mixture models for modeling and generating synthetic versions of nested categorical data. Bayesian Analysis, 13(1):1832200
[30] Jahrer, M. (2019). Porto seguro's safe driver prediction.
[31] Jang, E., Gu, S., and Poole. B. (2016). Categorical reparameterization with gumbel- softmax. arXiv preprint 16/
11

---

Published as a conference paper at ICLR 2023
[32] Jordon, J., Yoon, J., and Van Der Schaar, M. (2018). Pate-gan: Generating synthetic data with differential privacy guarantees. In International conference on learning representations.
[33] Kaur, D., Sobiesk, M., Patil, S., Liu, J., Bhagat, P., Gupta, A., and Markuzon, N. (2021). Application of bayesian networks to generate synthetic health data. Journal of the American Medical Informatics Association, 28(4))801-81.
[34] Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:/412.600
[35] Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. 1rXiv:1312.6114
arXiv preprint
[36] Kipf, T., Fetaya, E., Wang, K.-C., Welling, M., and Zemel, R. (2018). Neural relational inference for interacting systems. In International Conference on Machine Learning, pages 2688-2697, PMLR.
[37] Kocaoglu, M., Snyder, C., Dimakis, A. G., and Vishwanath, S. (2017). Causalgan: Learning causal implicit generative models with adversarial training. arXiv preprint 1rXiv:1709. 709.02023.
[38] Kohavi, R. et al. (1996). Scaling up the accuracy of naive-bayes classifiers: A decision-tres tree hybrid. In Kdd, volume 96, pages 202-207.
[39] Koller, D. and Friedman, N. (2009). Probabilistic graphical models: principles and techniques. MIT press.
[40] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.
[41] Kwon, D.-H. and Bessler, D. A. (2011). Graphical methods, inductive causal inference, and econometrics: literature review. Computational Economics, 38(1):85-106
[42] LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E.,, Hubbard, W., and Jackel, D. (1989). Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551.
[43] Li, X., Chen, Z., Poon, L. K., and Zhang, N. L. (2018). Learning latent superstructures in variational autoencoders for deep multidimensional clustering. arXiv preprint arXivv:803 05206.
[44] Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs and variable selection with the lasso. The annals of statistics, 34(3):1436-1444
[45] Nielsen, D., Jaini, P., Hoogeboom, E., Winther, O., and Welling, M. (2020). Survae flows: Surjections bridge the gap between vaes and flows. Advances in Neural Information Processing Systems, 13:12685-12666.
[46] Nikolenko, S. I. (2021). Synthetic data for deep learning, volume 174. Springer.
[47] Park, N., Mohammadi, M., Gorde, K., Jajodia, S., Park, H., and Kim, (2018). Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment. 11(10):1071- 1083.
[48] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L.., and Lerer, A. (2017). Automatic differentiation in pytorch.
[49] Patki, N., Wedge, R., and Veeramachaneni, K. (2016). The synthetic data vault. In 2016 IEEE International Conference Data Science and Advanced Analytics (DSAA), pages 399-410, IEEE.
[50] Pearl, J. (2009). Causality. Cambridge university press.
[51] Pearl, (2011). Bayesian networks.
[52] Peters, and Bühlmann, (2015). Structural intervention distance for evaluating causal graphs. Neural computation, 27(3)771-779..
12

---

Published as a conference paper at ICLR 2023
[53] Plataniotis, K. N. and Hatzinakos, D. (2017). Gaussian mixtures and their applications to signal processing. Advanced signal processing handbook, pages 19-124.
[54] Qian, Z., Cebere, B.-C., and van der Schaar, M. (2023). Syntheity: facilitating innovative use cases of synthetic data in different data modalities. arXiv preprint arXiv:2331.00577.
[55] Rezende, D. and Mohamed, S. (2015). Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR.
[56] Roberts, S. J., Husmeier, D., Rezek, I,, and Penny, W. (1998). Bayesian approaches gaussian mixture modeling, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(1 1142.
1):1133- 133-
[57] Schäfer, J. and Strimmer, K. (2005). A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical applications in genetics and molecular biology, 4(1).
[58] Schmidhuber, (2015). Deep learning in neural networks: An overview. Neural networks, 61:85-117.
[59] Scutari, M., Howell, P., Balding, J. and Mackay, (2014). Multiple quantitative trait analysis using bayesian networks. Genetics, 198(1):12--137.
[60] Sehwag, V., Mahloujifar, S., Handina, T., Dai, S., Xiang, C., Chiang, M., and Mittal, (2021). Robust learning meets generative models: Can proxy distributions improve adversarial robustness? arXiv preprint arXiv:2104.09422
[61] Shimizu, S., Inazumi. T., Sogawa, Y., Hyvärinen, A., Kawahara, Y., Washio, T., Hoyer, P. O., and Bollen, K. (2011). Directlingam: A direct method for learning a linear non con-gausiian structural equation model. The Journal of Machine Learning Research, 12:1225-122 1248.
[62] Shwartz-Ziv, Ziv, R. and Armon, A. (2022). Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90 90.
[63] Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25.
[64] Spirtes, P. and Glymour, C. (1991). An algorithm for fast recovery of sparse causal graphs. Social science computer review, 91):622-7.
[65] Spirtes, P., Glymour, C. N., Scheines, R., and Heckerman, D. (2000). Causation, prediction, and search. MIT press.
[66] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple way prevent neural networks from overfitting. The journal of machine learning research, ((1):1929-1958
[67] Street, W. N., Wolberg, W. H., and Mangasarian, O. L. (1993). Nuclear feature extraction for breast tumor diagnosis. In Biomedical image processing and biomedical visualization, volume 1905, pages 861-870. SPIE.
[68] Sun, Y., Cuesta-Infante, A., and Veeramachaneni, K. (2019). Learning vine copula models for synthetic data generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5049-5057.
[69] Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., and Birchfield, S. (2018). Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 969-9777
[70] Tsamardinos, L., Brown, L. E., and Aliferis, F. (2006). The max-min hill-climbing bayesian network structure learning algorithm. Machine learning, 65(1)31177.
13

---

Published as a conference paper at ICLR 2023
[71] Ucar, T., Hajiramezanali, E., and Edwards, L. (2021). Subtab: Subsetting features of tabular data for self-supervised representation learning. Advances in Neural Information Processing Systems, 14:18853-18863
[72] van Breugel, B., Kyono, T., Berrevoets, J., and van der Schaar, M. (2021). Decaf: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems, 34.
[73] Van der Maaten, L. and Hinton, G. (2008). Visualizing data using t-sne. Journal of machine learning research, 9(1).
[74] Waples, R. S. and Gaggiotti, O. (2006). Invited review: What is population? empirical evaluation of some genetic methods for identifying the number of gene pools and their degree of connectivity. Molecular ecology, 15(6):1419-143) 1439.
[75] Wen, B., Colon, L. O., Subbalakshmi, K., and Chandramouli, R. (2021). Causal-tgan: Generat- ing tabular data using causal generative adversarial networks. arXiv preprint arXiv:2104. 106.0
[76] Xu, I.., Skoularidou, M., Cuesta-Infante, A., and Veeramachaneni, K. (2019). Modeling tabular data using conditional gan. Advances Neural Information Processing Systems, 32.
[77] Xu, L. and Veeramachaneni, K. (2018). Synthesizing tabular data using generative adversarial networks. arXiv preprint arXiv: 1et1.11111...4.
[78] Yoon, Drumright, L. N., and Van Der Schaar, M. (2020a). Anonymization through data synthesis using generative adversarial networks (ads-gan). IEEE journal of biomedical and health informaties, 24(8):2378 2388.
[79] Yoon, J., Jarrett, D., and Van der Schaar, M. (2019). Time-series generative adversarial networks. Advances in Neural Information Processing Systems, 32.
[80] Yoon, J., Jordon, J., and Schaar, M. (2018). Gain: Missing data imputation using generative adversarial nets. International conference on machine learning, pages 5689-5698. PMLR.
[81] Yoon, J., Zhang, Y.. Jordon, J., and van der Schaar, M. (2020b). Vime: Extending the success of self-and -upeesupee learning to tabular domain. Advances in Neural Information Processing Systems, 33:1 1033-11043.
[82] Young, J., Graham, P., and Penny, R. (2009). Using bayesian networks to create synthetic data. Journal of Official Statistics, 25(4):549.
[83] Zhang. H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. In International Conference Learning Representations.
[84] Zheng, X., Aragam, B., Ravikumar, K., and Xing, E. P. (2018). Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31.
(85) Zhou, Y. (2011). Structure learning of probabilistic graphical models: comprehensive survey. arXiv preprint arXiv:1111.6925
14

---

Published as a conference paper at ICLR 2023
A DATA AUGMENTATION MODEL
The data augmentation model conditionally generates augmented data, i.e. अ = Fo 7 = Go).
To allow conditioning on any variable during the generation stage, we assume a directed acyclic graph (DAG). This means that variables are generated sequentially, following topological order TG. which can be described through SEM [50]:
á = (i)),,,
In contrast to the synthetic data model, the generation of each variable is conditioned the predicted value of its parents. 5 Each variable transformed through a specific embedding function h fo, (ā:) V i E [d], where h/ Rd and implement fo, using one layer MI.P This is followed by one round of message passing, where the embedding update function is implemented using single-layer MLP. We aggregate parent embeddings using the mean, equivalent taking a weighted average:
m 0g (-) n
(()) = ()) =
mean (G) (2) , ,VJEN()))
2; 78 (-) tanh (:(1) x (x())
The output after message passing is taken to be the prediction the current variable ă; This model is trained using the GAN adversarial loss (eq. (5)). Additionally, introduced in [84] to force a DAG to be recovered:
the continuous DAG penalty
max min Elog (ZFo + log(1 D(X)) +X(tr(exp(Ga o G d) F6 D
(5)
GAN Loss
Regularisation
To generate augmented data, we randomly sample a subset of the variables, and for each of which, we sample a conditioning value uniformly from the support of that variable, 2 (min(X,). maxi(x) We then generate augmented data by conditioning chosen variables on sampled values, & XX = a) Fa(z, X, = a; G [50]. We note that this conditional generation corresponds to sampling oul-of-dixtributtion, as we are sampling from different distribution that is defined on the same support as the training.
B IMPLEMENTATION DETAILS
B. MODELS AND EVALUATION
Models. All models implemented in PyTorch [48]. The data is split 60-20-20 into train, validation and test sets and reported results are averaged over 10 runs. Training performed using the Adam [34]. All experiments
an NVIDIA Tesla K40C GPU, taking less than hour complete.
Hyperparameters. For all methods compared, we consider hyperparameters include batch size (64, 128), learning rate (1e-3, 1e--3,5e-33 5, 1e-2). We include weight decay of 1e-3 [58]. Hyper- parameters are searched using Bayesian Optimization [63]. Each model is allowed a computation budget of 10 sweeps in Bayesian Optimization, where the search objective the reconstruction loss on the validation set. We describe model-specific hyperparameters in the exact implementation details below. For the graph sparsity term, we consider regularization penalty À E {1c-3, 3, 1c-2, Ic 1c- For f0.1,0.5 1.0]. All models are trained for maximum of 1000 epochs, with early stopping no improvements on the validation set for 50 epochs.
the KL divergence penalty, we consider
We partition the observed dataset D into training set Derain and test set Drest, and train generative models on Drrain- Using the trained models, we generate a synthetic data set Dayns which has the same number of samples Dreat- We then evaluate the aforementioned desiderata on (Drest, D sym) To evaluate quality, we employ the three-dimensional metric, a-precision, 3-recall, and authenticity, which assesses whether the samples are realistic, diverse enough to cover the
Evaluation.
((() Pa(f), the neighberhood strictly parent variables due the DAG
15

---

Published as a conference paper at ICLR 2023
variabilityy in real data, and generalization performance respectively [1]. Each metric evaluates different aspect of synthetic data quality, and we average the three metrics to obtain holistic score. Detection evaluated by training post-hoe classifiers distinguish samples from the original and generated datasets. Specifically, we train a two-layer MI., XGB classifier and GMM classifier and average the classification AUROC. To evaluate utilityy we report average performance achieved by three downstream prediction models (linear model, two-layer MIP, and XGB model) trained on Daya and evaluated on Drest. We report the change in AUROC of models trained on D Payn and those trained on Dtrain-
B.2 SYNTHETIC DATA BENCHMARKS
In this subsection, we provide further details on the benchmarks we compare against: including
Bayesian networks (BN) [51], GAN-based [76]. TableGAN [47] and VAE-based [76, 49] and normalizing flow NFLOW [55]. Additionally, we also consider data augmentation methods: Gaussian noise InNoi se [40]; Mi xÜp [83] and SwapNoise [30]. The baselines are implemented using the open source package synthcity [54]."
[53]. We train a Gaussian mixture model, where the number of components E {5, 10, 15), and each component has its own general covariance matrix. We use Dnpectation-Maximizatre to fit the parameters of the model, with a stopping condition 1e-3 or max iteration of 200 iterations. Lastly, initialize the model using f--maas clustering.
BN [51]. We train BN in two stages, where the first stage learns the network structure and the second stage performs learning based on the returned DAG. We use the PC algorithm to learn the DAG [64]. Once a DAG is returned, the conditional probabilities are learned through maximum likelihood estimation, where continuous variables assumed to come from linear Gaussian conditional probability distribution (CPD) and discrete variables from a discrete CPD.
MLP with two ReLU-activated hidden layers to implement the generator. Similarly, we employ an MLP with two Rel_t-activeted hidden layers to implement the discriminator. The hyperparameters
CTGAN [76], TableGAN [47]. For CTGAN,
tuned according the recommended settings in [49]. TableGAN is implemented using Deep Convolution GAN with recommended settings in [47], where the generator has three deconvolutional layers, and the discriminator has three convolutional layers.
TVAE [49]. The VAE-based model is implemented with an encoder with two ReLU-activated layers. The decoder similarly has two hidden layers. We use a 32 dimensional latent space that is normally distributed and a standard normal prior.
NFLOW [55]. We implement the normalizing flows using the rational-quadratic transform introduced in [17]. Specifically, is implemented using an MLP with 2, 228-dimensional hidden layers and permutation operations. A standard normal base distribution is employed, and the flow is run with 500 steps.
B.3 DATA AUGMENTATION BENCHMARKS
InNoise [40], MixUp [83], SwapNoise [30]. For InNoise, se, we add zero-centered Gaussian noise ~ (00, )) to the inputs, where we consider a E (0.01, For SwapNo se, we randomly swap 10% of elements between two inputs. MixUp is implemented by randomly combining two samples Az' + (1 A)r", where x' xx ~ Dirain and À Beta(0.2.. 0.2).
DAFS [14]. We train an autoencoder, where the encoder and decoder are both implemented as MLPs with two ReLU-activated hidden layers. We take the feature vector at the output of the encoder c; and randomly apply one of three possible operations (1) add Gaussian noise E ~ (0, o2), (2) interpolation
X(c3 G) + C; or (3) extrapolation A(e: <g) G; where À = 0.5 as suggested by the authors. Augmented samples x' are then obtained by passing the altered feature vector through the decoder.
16

---

Published as a conference paper at ICLR 2023
Table Experimental datasets. Description of experimental datasets.
Number of instances
Number of
Dataset
Description
features
Adult [38]
Census data
48842
15
Breast [67]
Breast cancer
569
32
Covertype [5]
Forest cover
581012
Credit [28]
Credit risk
1000
20
ECOLI [57]
Functional genomics
2000
46
MAGIC-IRRI [59]
Plant genetics
2000
64
Red [12]
Wine quality
1599
White [12]
Wine quality
4898
Mice [26]
Protcin expression
1080
82
Musk [16]
Musk molecules
6598
168
B.4 DATASETS
We use 10 datasets in total, including UCI datasets [16]," specifically Adult, Breast,
Covertype, Credit, White, Red, Mice, Musk and 2 Bayesian Network repository datasets [39], specifically ECOLT and MAGIC-TRRT. A summary of the datasets, including dataset descrip- tion, the dimensionality, and number of samples, is presented in Table 5.
B.5 PRIOR KNOWLEDGE
The use of an explicit graph to guide generation allows for a variety of prior knowledge to be incorporated through the adjacency matrix. Here, we describe few options of incorporating domain expertise:
Sparsity. The learned graph be sparse such that variables only depend small subset of other variables. Mathematically. R(Gg) = |Goolpp where || 1Ip denotes the Lp matrix norm. Dependence. Partial knowledge about the dependencies between features can be encoded through a graph prior Go. i.e., R(G,) ||G Gol p.
Graph types. Graphs of specific types can be learned. For example, if an undirected graph is assumed, we can employ symmetric prior a G Gl'lp, ll, Alternatively, we can use the DAG penalty
[84] to encourage learning directed, acyclic graph (DAG), R(Gs) tr(exp(G Go) D), where tr()) is the matrix trace and is the number of variables.
Connectivity. Encourage different patterns of connectivity through penalty degree of each variable D& Dollp- where E Rd the degree of each variable.
C ADDITIONAL EXPERIMENTS
In this section, we provide additional results to comprehensively evaluate our proposed methods, specifically:
1. Additional datasets: $C.1 evaluates synthetic data performance on 4 additional datasets. Visualizations:CC.. visualizes t-SNE projections on original and synthetic datasets to qualitatively investigate quality and examines learned adjacency matrices.
Sensitivity: $C.4 investigates performance sensitivities according data size and feature counts.
4. Data augmentation: $C.5 describes the best model performance after data augmentation.
C.I ADDITIONAL RESULTS
Additional datasets. We assess the quality of synthetic dataset using the same desiderata introduced in 1, namely quality, detection, and utiliry. We six additional datasets, ECOLI, MAGIC-IRRI. Red, White, Mi ce, and Musk. The results are reported in Table 6.
Htrps://anchie.... los wan.iiiiiinllitttssst..g pip
hupp: cww..enppppptitrryy
17

---

Published as a conference paper at ICLR 2023
Table Quality, detection, and utility of synthetic data. Bold indicates the best performance.
Thitzort
ECOLI
MAGIC-IRRI
Red
White
Mice
Musk
0.67
0.67
6.04
0.68
0
= 0.06
Quality (1)
(higher
TableGaN
0.38
0.38
0.06
0.08
0.33 0.07
0.33
0.46 0.05
0.07
0.53
0.53
0.02
0.04
0.41
0.4G
0.04
0.03
0.56 + 0.03
0.04
is better)
TVAE
0.45
0.08
0.61 0.02
0.07
0.04
0.58 + 0.05
0.57
0.05
0.57 - 0.04
X25CM
0.36
0.03
0.07
60C6 .P
0.67 0.05
0.63 0.09
0.63 = 0.07
0.62 2 0.08
0.50
0.05
0.61 + 0.02
BBI
0.39 0.07
0.40
0.08
0.71 = 0.06
0.60 + 0.03
0.73 +: 0.05
0.80 = 0.05
Detection (L)
CTGAM
0.74
0.73
0.06
0.70
0.03
(lower is betoe))
Tabl
TVAE
0.74 + 0.02
0.74 0.02
0.09 0.05
0.73
0.72 = 0.07
0.04
0.74 + 0.04
0.75 0.09
0.71 0.06
0.90 0.06
0.77 + 0.06
- 0.05
X250M
0.70 0.03
0.70 + 0.08
0.74 = 0.05
0.73 + 0.03
0.71 0.03
0.73 + 0.05
0.60
0.69
0.00
0.12
BRI
0.01 0.00
0.05 + 0.00
-U #..01
-0.11=0.04
-0.02 0.00
-..19+0.006
Ulility (1)
higher better)
TablwGAN
CTTAAN
-0.1840.06
-0.20+0.03
0.0640.01
0.00 0.00
-0.1340.01
-0.10+0.05
0.01 = 0.00
-0.05=0.01
0.00
-0.11=0.02
-0.1700.11
-0.11=0.02
-0.15=0.04
-0.08=0.03
-0.09=0.02
0..1
-0.30=0.03
-0.08
W71CMM
0.0550.01
00022:0.00
-0.06=0.01
-0.14-0.05
-0.08=0.02
-0.14
GOEGLE
0.02:00:00
0.01 0.00
0.01 = 0.00
-0.08 0.001
-0.10=0.02
-0.11
We note that BN achieves the best performance ECOLI and MAGIC-IRRI, which is reasonable as those datasets are generated according to known Bayesian network, and BN models have a natural advantage. On those two datasets, GOGGLE is able consistently outperform other deep generative models. On Red and White, GOGGLE achieves superior performance against other benchmarks. On the contrary, BN, our closest competitor, achieve worse performance as the underlying DAG assumptions become too restrictive. Additionally, we highlight that models trained on synthetic data generated by GOGGLE consistently achieves similar performance those trained real datasets, indicating strong data utility.
C.2 VISUALIZATION OF SYNTHETIC DATA RESULTS
In Figures 4 to we observe that synthetic data generated by GOGGLE exhibit markedly better overlap with the original dataset than other benchmarks using t-SNE for visualization. We note that the GAN-based models, specifically CTGAN and TableGAM exhibit mode collapse behaviour and the TVAE and NF NFW can fail to match the underlying distribution (on ECOLI and Breast, respectively).
C.3 ANALYSIS OF RELATIONAL STRUCTURES
Qualitative analysis. We visualize the learned graphs on Credit and Breast in Figure 9. For the purposes of our qualitative analysis, we deliberately increase the weighting of the graph sparsity regularization A. The Breast dataset [16] contains numeric features extracted from images of a breast mass. We note that the targer variable (diagnosis tumor) has a high degree of connectivity, and dependent various physical properties of the tumor, including mean perimeter and mean compuctness. We similarly observe informative variables identified in the Credit t dataset [16], where the account balance depends on occupation, credit amount, and length of current employment. Additionally, we plot the adjacency matrix of trained models in Figure 8, where sparsity regularization term was applied to all models encourage sparsely connected graphs.
Quantitative analysis. We previously claimed that we are only interested in an approximately correct relational structure, which our evaluations in Table and Figure found is sufficient improve synthetic data quality. However, are these structures indeed approximately correct? That is the question we aim to address here. We compute the structural hamming distance (SHD) [13], which computes graph distances between predicted and ground truth graphs by the number of insertions, deletions or flips to transform one graph another. We compare the relational structure learned by GOGGLE against 1) the graph learned by a Bayesian Network (BN), 2) Erdos-Rent Rensy random graph generated with edge probability matching empirical edge probability p in the ground-trus -ruth graph (ER (p)). 3) thresholded correlation graph (CORR).
We describe results in Table 7. We note that EN model is the pseudo-orace as the true DGP for ECOLI and MAGIC are indeed Bayesian Networks with linear Gaussian functional relationships (matching the model specifications of BN). ER (p) serves as a dummy bascline as the predicted graph is randomly guessed. We note that the correlation graphs (CORR, with threshold at 0.5) learn many
18

---

Published as a conference paper at ICLR 2023
Table 7: Structural hamming distance. Distance from learned graph to ground truth graph.
Datasets
(p)
CORR
GOGGLE
ECOLI
61 2
131 - 6
204
110 5
MAGIC
82 + 3
192 + 10
103
95 = 2
Ta
camp-
camp-1
campl
camp-
cong 1
camp-1
(a)
(b) CTGAN
(c) TableGAN
(d) TVAE
(c) NELOW
(f) GOGGLE
Figure 4: t-SNE projection Breast dataset.
comp-1
comp-1
comp-1
comp-1
(a) BN
(b) CTGAN
(c) TableGAN
(d) TVAE
(c) NELOW
(f) GOGGLE
Figure 5: t-SNE projection on Red dataset.
campls
campl1
camp1
Campl
Campd
(a) BN
(b) CTGAN
(c) TableGAN
(d) TVAE
(e) NFLOW
( GOGGLE
Figure 6: i-SNE projection on ECOLI dataset.
camp1
(a) BN
(b) CTGAN
(c) TableGAN
(d) TVAE
(c) NFLOW
(f) GOGGLE
Figure 7: t-SNE projection on White dataset.
(a) Adults
(b) Breast
(c) Credit
(d) ECOLI
(e) MAGIC
(1) White
Figure 8: Learned adjacency matrices.
spurious, yet uninformative edges as it is solely driven by association relationships. We note that the graphs learned by GOGGLE, sparse, and informative.
19

---

Published as a conference paper at ICLR 2023
Tean aoort
(a) Credit
Breast
Figure 9: Learned graphs.
Table 8: Data augmentation. AUROC on Dreat models trained on augmented data. Bold indicates
the best performance.
Adolt
Covertype
Credit
Beseline
0.70 = 0.09
0.96 + 0.01
0.59 + D.13
0.65 0.02
InMoise
0.08 0.05
0.97
U.D1
0.61 + 0.10
0.64 0.03
Mixtip
Swaphicise
0.01
0.58 0.88
0.63
ESAug
0.71 0.06
0.96 +: 0.01
0.80 + 0.13
0.67 0.02
GOCOLE-SD
0.70 0.05
0.97 1 0.01
0.80 0.10
0.86 0.01
GOGeLR
0.08
0.00
0.10
0.67
C.4 SENSITIVITY ANALYSIS
Lastly, we are interested in identifying settings where GOGGLE excel as a generative model. Specif- ically, are interested in understanding sensitivities of model performance with respect to the effects of feature counts and number of samples in the dataset, We compare our model against the benchmarks Figure 10. Here, the datasets are shown on the x-axis and are sorted in order of increasing feature count, and increasing number of samples (see Table 5 for more on datasets). We note that the advantage of GOGGLE is more noticeable when there are less number of samples (i.e.. on Breast, Credit and Red). In the regime with larger number of samples, all models exhibit similar performance, although GOGGLE still achieves performance improvements. Furthermore, models achieve similar performance when the number of features is low. However, when the number of features increases, the performance of GAN-based models deteriorate. This is interesting, and a potential logical explanation is that they overfitting to the training data.
C.5 EVALUATION OF AUGMENTED DATA
We compare popular tabular data augmentation methods by inspecting downstream model perfor- mance. Specifically, we generate augmented data Dang from Dirain (and has the same number of samples), train predictive models on the combined D Dcomt (Dtrains Daug and evaluate perfor- mance on Drest- We train four downstream prediction models. including linear model, two-layer MLP, RF classifier, and XGB model and report the averaged performance achieved by the four models. We perform data augmentation on GOGGLE by randomly selecting variables to condition on and sampling uniformly from the marginal support of the variable. In Table 8, we observe that augmented data generated by GOGGLE leads to improved generalization performance across all datasets.
D CONNECTION TO RELATED WORKS
There are several parallels between our works and several related research fields, namely probabilistic graph discovery, relational learning, and self- supervied learning (SSL). These fields reflected by the common approach in exploiting relational structure underlying tabular data. Here, we discuss commonalities and differences in depth.
20

---

Published as a conference paper at ICLR 2023
BN
CTGAN
TABLEGAN
TVAE
+
NFLOW
GOGGLE
0.8
0.0
0.8
0.6-
0.6
-0.2
0.4
0.4
004
Red
White Adule
Credit
Bronet
ECOLI
Coverty MAGIC-IRAI
Red White Adule
Cruiit
Breant
ECOLI Coverty MAGICAIRI
Red White
Adule
Credit
Braast
ECOLI
(a) Feature Count
BN
CTGAN
TABLEGAN
TVAE
NFLOW
GOGGLE
0.8
0.0
0.6
0.6
-02-
0.4
0.4
-0.4
Browt Gredit
Red ECOLI MAGIC-RRI IRiii
White
Adult Covettpe
Bresst Credit
Red ECOL! MAGIC-
White
Adule Coventyi
Breatt Credit
Red ECOLI MAGIC MACIC-IRGI IRRII
White
Adult Curentypo
(b) Number Samples
Figure 10: Sensitivity analysis. Evaluating synthetic data based on (left) quality (1), (middle) detection (L), and (right) utility (t). Datasets are sorted according to (a) increasing feature counts, and (b) increasing number of samples.
Probabilistic graph discovery. Probabilistic graph discovery aims to recover the true probabilistic graphical model (PGM) underlying observed data [85, 15J. We propose generative model for tabular data that learns and leverages an underlying graph improve the performance Importantly, our model does not recover the true PGM from data if the data is indeed generated by PGM (i.e. it does not perform PGM structural discovery). Additionally, the message passing computation is not an instance of or an approximation to probabilistic inference routine. Specifically, we take advantage of the sparse and compact representations of graphical models to learn better generative models, incorporate prior knowledge, and perform conditional generation. We do so by incorporating a graph as explicit structure into the generative process. We summarize the key distinctions between work and the probabilistic structure learning literature.
data synthesis.
The graph learned in GOGGLE encodes conditional dependence structure between variables (global Markov property), in the same sense that PGMs reflect allowed conditional dependencies. The key distinction between our approach and probabilistic structural learners is that we only require an approximate structure. In contrast, structural learners aim recover unique graphs that close the true DGP. This learned graph is used to answer probabilistic inference queries, c.g. P(Y | = x). which requires the graph to correct. In order recover unique graph, structural learns generally assume certain graph, or distributions, or relationships between variables.
Our objective is to learn an approximate graph that models associational dependence and can guide generation. Therefore, we do not need to make similar assumptions that unnecessarily restrict the class of learnable distributions and can lead to miss-specified model. Additionally, we emphasize that our proposed method is not designed to perform sampling-basee probabilistic inference. Due to the different objectives, we make minimal assumptions on the graph type, variable distribution, and functional relations.
Relational learning. The goal of relational inference is to infer relationships between objects from observational data alone (interacting objects in NRI [36], and molecular interactions in multi-omics integration in BayRel [24] and MoReL [25]). The output of the learning method is a probabilistic relational graph, which is evaluated based on correctness of the recovered graph. The biggest difference of our work is that GOGGLE is fine if the structure is only partially correct, or wrong, which is in stark contrast any literature structure discovery. The relational structure in our case
21

---

Published as a conference paper at ICLR 2023
Table 9: Comparison to related works. Commonalities and differences between our work, relational inference, and probabilistic structural discovery.
Probabilistic Graph Discovery
GOGGLE
Commonality Objective
Edges reffect allowed conditional dependencies between variables
Recover unique probabilistic graph
Learn approximate graph describing dependencies between variables
underlying observed data
Evaluation Metric
Quality of discovered graph: graph
Quality of synthetic data: quality, detection, and utility [1]
distance measure [52, 61]. edge
classification metric
Application
Probabilistic inference P(Y X = x)
Generate conditional synthetic data ~ (X)
Specific graph types (i.e. directed or undirected)
Arbitrary graph types (i.e. mixed, directed, or undirected)
Distributional assumptions on variables (e.g. Gaussian)
No assumptions variable distribution
Assumptions
Assumptions on functional relationships
No assumptions on functional relationships model
between variables (c.g. Linear with
Gaussian additive noise)
Representative Works
UGM: Chow-Lit algorithm [11]. graphical LASSO [2], neighborhood
Deep generative: GAN-based [76]. VAB-based [77]. Non-neural: BN,
selection [44]. DGM: moore-based [9],
mixture models, copula
constraint- based [65], hybrid [70]
acts as an inductive bias encouraging a sparse set of informative neighbors to be found, thus better learning the distribution of tabular data, and plays a regularization effect on spurious relationships. As we show in Table 4 and Figure 3, these partially correct structures that we learn can consistently improve generative synthetic data performance.
SSL. methods for tabular data are similarly driven by uncovering relational information between features to learn good representations. However, this is often done implicitly, as in VIME [81], which applies masking operator to encourage the representation learning module to learn inter-featur relationships. Similarly, SubTab [71] employs a contrastive loss by generating views on subsets of features, implicitly encouraging learned representations to extract mutual information. Our work is different in explicitly incorporating a relational mechanism into the generative process, explicitly encouraging informative relations to be found, in addition to providing a flexible mechanism for prior knowledge and regularization. We discuss specific differences in objectives, methods and evaluation criteria below:
Table 10: Comparison to related works. Commonalities and differences between our work, relational inference, and self-supervised learning (SSL).
Relational Inference
SSL
GOGGLE
Commonality Learning output (goal)
Exploit relational structure between variables
The relational graph G
Representation vector h
Data distribution p(X)
Learning method
Probabilistic inference over relational graph p(G]X)
Learning encoder function
Joint iearning of the relation graph G and the distribution p(X) that is compatible with C Quality of synthetic data: quality, detection, utility
h = f(x) through contrastive loss or pretext
generation
Evaluation
Edge prediction metric; graph distance metric
Quality of representations, e.g. performance in
downstream tasks
Representative works
NRI [36], BayReL [24]. MoReL [25]
VIME [81], SubTab [71]
CT-GAN [77]. TVAE [76]
22


Reviews:
[{"review_id": "m5EnEXarNn", "reviewer": "Reviewer_bxj6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to use relational inference for the generative model in a tabular setting. The proposed method jointly learns the relation between features in tabular data and parameters of generative models.\n\nThe paper is fairly well structured, apart from missing some of the related works and weak experiments.\n\nMy main concern with this paper is the lack of novelty. One of the main known issues in relational learning (even out of tabular setting) is computational complexity and dealing with high dimensional data, however, the proposed method could not address this issue and only the experiments are conducted on tiny feature sizes which is not an issue. This is not consistent with real-world high-dimensional tabular data.\n\nApart from that, it is not clear to me why they have added the prior knowledge through regularization and not embedded it as a graph similar to MoReL. Incorporating as a regularizer could be done in simpler models as well and there is no comparison in the experiments. ", "summary_of_the_review": "I believe the paper is not novel and could not address the main issue of this domain. The experiments also cannot support the claims. ", "strength_and_weaknesses": "**Pros:**\n\n- Generative modeling for tabular data is a hard and interesting problem.\n\n**Cons:**\n\n- The novelty is limited.\n\n- The authors missed some related papers, especially in the methodological parts, including NRI [ICML 2018], MoReL [ICLR 2022], BayReL [ NeurIPS 2020], and many other related relational inference papers in other domains.\n\n- Some of the self-supervised learning models for tabular data can be used as generative models, including VIME [NeurIPS 2020], SubTab [NeurIPS 2021], and many other available methods which are based on self-supervised methods.\n\n- The learned structure is not well explored in experiments. How it will look like and how is different from the ground truth?\n\n- The datasets are not high dimensional. \n\n- The authors did not report the performance of the raw data model without generated dataset.\n\n- The authors missed some of the basic regression methods and is not clear how the reported ones are tuned. \n\n- The authors reported an average of three different methods. I would like to see the performance of each separately, and also LR and simple methods should be included. \n\n- The AUROC metrics is not sufficient. I would like to see the ROC plot as well as other metrics such as recall, FDR, F1, etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lmFhFIVDSk_", "reviewer": "Reviewer_k5vp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces a new generative model for tabular data. It proposes to learn both a relational structure between variables and also a data generating process conditioned on that structure. This formulation has not been presented before for neural methods and is a significant contribution. The benefits of this method are that it can incorporate prior information via an adjacency matrix, leverage the benefits of neural networks for generation in the large data regime, and is more amenable to certain types of regularization. These benefits are quantified empirically on a few tabular datasets where the proposed method seems to be the most consistent performer and usually outperforms other baselines.", "summary_of_the_review": "Overall, the proposed method in this paper seems very promising - a model-based tabular generation scheme that can leverage prior knowledge, that is more amenable to regularization in the small data regime, and that has solid performance empirically across a number of datasets. However, the paper does not meet two of these major marks:\n1) In terms of leveraging prior information, it seems Goggle was always outperformed by either BN or CTGAN in this setting.\n2) Details of hyperparameter tuning are not provided. Specifically, it seems like an equal computational budget may not have been given to each method. Thus, it is hard as a reviewer to deduce whether performance gains come from the superiority of the method or from more extensive parameter tuning.\nNonetheless, the effort put forth in the paper is solid and the main ideas presented would make a valuable contribution. I am marking my score a little lower due to the two major drawbacks above, but I am happy to increase the score during & after discussion with the authors, depending on their response.\n\n********* UPDATE AFTER REBUTTAL ****************\n\nThe authors have resolved my main concerns regarding hyperparameter tuning and have adjusted the language in the prior information section to scope down the claims (as well as adding some new experiments). Additionally, the authors have added a clarification that their method has only been validated on relatively low-dimensional datasets, which their experiments support. For this reason, i am increasing my score as I believe this is a solid contribution to the community.", "strength_and_weaknesses": "Strengths:\n- Empirically, the proposed method is usually the best performer across 10 benchmarks.\n- The model-based approach to learning dependencies and graph-based approach to generative modeling is unique and a valuable contribution for the community. I could foresee future work building on this and finding better ways to incorporate prior information.\n- Strong empirical evaluations, ablations, and benchmarking efforts to compare to baselines overall.\n\nWeaknesses:\n- The story on leveraging prior information is not convincingly demonstrated through experiments. Looking at Figure 3, Goggle never seems to outperform BN, and in some cases is handily beat by either BN or CTGAN. Given that leveraging priors is a major strength of the proposed method, the empirical results here are not consistent with that story. Also, the language in that section \"it is encouraging to see that our method achieves comparable performance to BN methods\" is misleading and should be altered.\n- Placement of results tables. Though the method is evaluated on 10 datasets, only results on 4 datasets are presented in the main text. From an outside perspective, it would appear as if these datasets may have been cherrypicked, especially since relative performance numbers on the other 6 datasets in the appendix are not as high.\n- Baselines comparison. Was equal hyperparameter tuning and equal computational budget given to each of the baselines? It does not seem specified in the appendix.\n- Comparison to a MM (mixture models) baseline is missing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K_KlHzNFOY", "reviewer": "Reviewer_Hvkk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new generative model for tabular data, as a VAE whose decoder structure is built around GraphNN layers.\n\nThe rationale for this structure is given in terms of inductive biases: on structured data like images or text, the successful models are built around architectures that take advantage of good inductive biases derived from the structure of the data. Tabular data lacks this general structure, and instead generally has a problem-dependent structure. Models like MLP don't have any of those inductive biases, explaining why they don't perform so well on tabular data.\n\nThe paper thus introduces GOGGLE, a VAE-like generative model whose decoder half is built using GraphNN layers that act akin a message-passing scheme on top of a graph defined over the features of the data. Similar to a probabilistic graphical model (while *not* being one), this structure reflects the dependencies between the variables as learned by the model. The paper proposed several regularization schemes over the learned adjacency matrix of this graph to reflect different inductive biases that can be injected in the model such as encouraging sparsity or injecting a partially known dependency structure. While the constructed graph is *not* a proper PGM, it structures the way in which the generative model processes the data.\n\nThe paper  provides experimental validation on several datasets and against multiple other models from the literature tailored to tabular data. Experiments show that GOGGLE reaches better performance across the board and following multiple metrics, including the quality of generated samples to train a downstream classifier on synthetic data. GOGGLE is notably competitive with Bayesian Networks, which remained the state of the art on many tabular problems, suggesting that the injected graphical structure is indeed core to performance on these problems.\n\nThis is further confirmed by an ablation study, which shows that the quality of the graph underlying the GraphNN decoder has a significant effect on the performance of GOGGLE, and that learning it jointly with the rest of the model using the proposed regularization is notably better than trying to infer it beforehand using another method.", "summary_of_the_review": "This is a good paper that provides a meaningful contribution to the question of generative modelling on tabular data.", "strength_and_weaknesses": "**Strengths:**\n- The proposed architecture is well justified and well detailed. The goals and non-goals of the model are clearly stated (notably appendix D is a very welcome comparison).\n- Experiments are detailed and extensive, and the ablation study clearly illustrates the impact of the core contribution (the GraphNN structure of the decoder).\n- The introduced model seems to have a good potential for meaningful impact on the problem of synthetic tabular data, as expressed by the performance impact, in particular in terms of *utility* (using the generated synthetic data for downstream tasks).\n\n**Weaknesses:**\n- The probabilistic part of the decoder is hardly discussed, while potentially having a big impact on the performance of the model. Appendix explains that a MSE loss is used for continuous variables, which corresponds to an isotropic Gaussian noise with fixed variance. See [*Simple and Effective VAE Training with Calibrated Decoders*, Rybkin et al, 2021](https://proceedings.mlr.press/v139/rybkin21a.html) for a discussion of why this is an important factor over the model performance. That paper focuses on image data, but the core argument translates to tabular data. I believe learning a per-feature noise scale for the Gaussian observation model may further improve the performance of the proposed model.\n- The is no discussion of the impact of $L$, the number of GraphNN layers stacked in the decoder. I suspect this parameter might depend a lot on the dataset at hand, and in particular of how sparse or strong the correlations between the variables are?\n- The discussion suggests extending the approach to learning graphs in the latent representation rather than between the observed variable. There is actually already some work in this vein existing (see for example [*Learning Latent Superstructures in Variational Autoencoders for Deep Multidimensional Clustering*, Li et al, 2019](https://arxiv.org/abs/1803.05206)) which would be worth discussing in the related works section of the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster