Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces a new generative model for tabular data. It proposes to learn both a relational structure between variables and also a data generating process conditioned on that structure. This formulation has not been presented before for neural methods and is a significant contribution. The benefits of this method are that it can incorporate prior information via an adjacency matrix, leverage the benefits of neural networks for generation in the large data regime, and is more amenable to certain types of regularization. These benefits are quantified empirically on a few tabular datasets where the proposed method seems to be the most consistent performer and usually outperforms other baselines.
strength_and_weaknesses: Strengths:
- Empirically, the proposed method is usually the best performer across 10 benchmarks.
- The model-based approach to learning dependencies and graph-based approach to generative modeling is unique and a valuable contribution for the community. I could foresee future work building on this and finding better ways to incorporate prior information.
- Strong empirical evaluations, ablations, and benchmarking efforts to compare to baselines overall.

Weaknesses:
- The story on leveraging prior information is not convincingly demonstrated through experiments. Looking at Figure 3, Goggle never seems to outperform BN, and in some cases is handily beat by either BN or CTGAN. Given that leveraging priors is a major strength of the proposed method, the empirical results here are not consistent with that story. Also, the language in that section "it is encouraging to see that our method achieves comparable performance to BN methods" is misleading and should be altered.
- Placement of results tables. Though the method is evaluated on 10 datasets, only results on 4 datasets are presented in the main text. From an outside perspective, it would appear as if these datasets may have been cherrypicked, especially since relative performance numbers on the other 6 datasets in the appendix are not as high.
- Baselines comparison. Was equal hyperparameter tuning and equal computational budget given to each of the baselines? It does not seem specified in the appendix.
- Comparison to a MM (mixture models) baseline is missing.
clarity,_quality,_novelty_and_reproducibility: The paper is overall clearly written and is easy to understand.
The quality of the proposed method seems reasonable; although it is not always the best, it most often is the best.
The proposed method for a model-based way to learn a generative model for tabular data is novel.
The work is NOT reproducible as code is not supplied (though there are some details in the appendix, I do not consider this reproducible unless code is provided).

A couple minor points to improve for paper clarify:
- I did not understand the point about synchronous vs asynchronous generation. There needs to be more explanation here, or it should go in the appendix (given that it seems the point was not revisited in the main text).
- How many samples are generated from Goggle for downstream evaluation for assessing Utility? Is it the same number of samples as there were original real samples? If so, it seems surprising to me that training on Goggle synthetic data is better than training on real data - this would be worth highlighting more!
summary_of_the_review: Overall, the proposed method in this paper seems very promising - a model-based tabular generation scheme that can leverage prior knowledge, that is more amenable to regularization in the small data regime, and that has solid performance empirically across a number of datasets. However, the paper does not meet two of these major marks:
1) In terms of leveraging prior information, it seems Goggle was always outperformed by either BN or CTGAN in this setting.
2) Details of hyperparameter tuning are not provided. Specifically, it seems like an equal computational budget may not have been given to each method. Thus, it is hard as a reviewer to deduce whether performance gains come from the superiority of the method or from more extensive parameter tuning.
Nonetheless, the effort put forth in the paper is solid and the main ideas presented would make a valuable contribution. I am marking my score a little lower due to the two major drawbacks above, but I am happy to increase the score during & after discussion with the authors, depending on their response.

********* UPDATE AFTER REBUTTAL ****************

The authors have resolved my main concerns regarding hyperparameter tuning and have adjusted the language in the prior information section to scope down the claims (as well as adding some new experiments). Additionally, the authors have added a clarification that their method has only been validated on relatively low-dimensional datasets, which their experiments support. For this reason, i am increasing my score as I believe this is a solid contribution to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to use relational inference for the generative model in a tabular setting. The proposed method jointly learns the relation between features in tabular data and parameters of generative models.

The paper is fairly well structured, apart from missing some of the related works and weak experiments.

My main concern with this paper is the lack of novelty. One of the main known issues in relational learning (even out of tabular setting) is computational complexity and dealing with high dimensional data, however, the proposed method could not address this issue and only the experiments are conducted on tiny feature sizes which is not an issue. This is not consistent with real-world high-dimensional tabular data.

Apart from that, it is not clear to me why they have added the prior knowledge through regularization and not embedded it as a graph similar to MoReL. Incorporating as a regularizer could be done in simpler models as well and there is no comparison in the experiments. 
strength_and_weaknesses: **Pros:**

- Generative modeling for tabular data is a hard and interesting problem.

**Cons:**

- The novelty is limited.

- The authors missed some related papers, especially in the methodological parts, including NRI [ICML 2018], MoReL [ICLR 2022], BayReL [ NeurIPS 2020], and many other related relational inference papers in other domains.

- Some of the self-supervised learning models for tabular data can be used as generative models, including VIME [NeurIPS 2020], SubTab [NeurIPS 2021], and many other available methods which are based on self-supervised methods.

- The learned structure is not well explored in experiments. How it will look like and how is different from the ground truth?

- The datasets are not high dimensional. 

- The authors did not report the performance of the raw data model without generated dataset.

- The authors missed some of the basic regression methods and is not clear how the reported ones are tuned. 

- The authors reported an average of three different methods. I would like to see the performance of each separately, and also LR and simple methods should be included. 

- The AUROC metrics is not sufficient. I would like to see the ROC plot as well as other metrics such as recall, FDR, F1, etc.
clarity,_quality,_novelty_and_reproducibility: Novelty is limited. And the reproducibility of the baselines is not obvious. 
summary_of_the_review: I believe the paper is not novel and could not address the main issue of this domain. The experiments also cannot support the claims.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new generative model for tabular data, as a VAE whose decoder structure is built around GraphNN layers.

The rationale for this structure is given in terms of inductive biases: on structured data like images or text, the successful models are built around architectures that take advantage of good inductive biases derived from the structure of the data. Tabular data lacks this general structure, and instead generally has a problem-dependent structure. Models like MLP don't have any of those inductive biases, explaining why they don't perform so well on tabular data.

The paper thus introduces GOGGLE, a VAE-like generative model whose decoder half is built using GraphNN layers that act akin a message-passing scheme on top of a graph defined over the features of the data. Similar to a probabilistic graphical model (while *not* being one), this structure reflects the dependencies between the variables as learned by the model. The paper proposed several regularization schemes over the learned adjacency matrix of this graph to reflect different inductive biases that can be injected in the model such as encouraging sparsity or injecting a partially known dependency structure. While the constructed graph is *not* a proper PGM, it structures the way in which the generative model processes the data.

The paper  provides experimental validation on several datasets and against multiple other models from the literature tailored to tabular data. Experiments show that GOGGLE reaches better performance across the board and following multiple metrics, including the quality of generated samples to train a downstream classifier on synthetic data. GOGGLE is notably competitive with Bayesian Networks, which remained the state of the art on many tabular problems, suggesting that the injected graphical structure is indeed core to performance on these problems.

This is further confirmed by an ablation study, which shows that the quality of the graph underlying the GraphNN decoder has a significant effect on the performance of GOGGLE, and that learning it jointly with the rest of the model using the proposed regularization is notably better than trying to infer it beforehand using another method.
strength_and_weaknesses: **Strengths:**
- The proposed architecture is well justified and well detailed. The goals and non-goals of the model are clearly stated (notably appendix D is a very welcome comparison).
- Experiments are detailed and extensive, and the ablation study clearly illustrates the impact of the core contribution (the GraphNN structure of the decoder).
- The introduced model seems to have a good potential for meaningful impact on the problem of synthetic tabular data, as expressed by the performance impact, in particular in terms of *utility* (using the generated synthetic data for downstream tasks).

**Weaknesses:**
- The probabilistic part of the decoder is hardly discussed, while potentially having a big impact on the performance of the model. Appendix explains that a MSE loss is used for continuous variables, which corresponds to an isotropic Gaussian noise with fixed variance. See [*Simple and Effective VAE Training with Calibrated Decoders*, Rybkin et al, 2021](https://proceedings.mlr.press/v139/rybkin21a.html) for a discussion of why this is an important factor over the model performance. That paper focuses on image data, but the core argument translates to tabular data. I believe learning a per-feature noise scale for the Gaussian observation model may further improve the performance of the proposed model.
- The is no discussion of the impact of $L$, the number of GraphNN layers stacked in the decoder. I suspect this parameter might depend a lot on the dataset at hand, and in particular of how sparse or strong the correlations between the variables are?
- The discussion suggests extending the approach to learning graphs in the latent representation rather than between the observed variable. There is actually already some work in this vein existing (see for example [*Learning Latent Superstructures in Variational Autoencoders for Deep Multidimensional Clustering*, Li et al, 2019](https://arxiv.org/abs/1803.05206)) which would be worth discussing in the related works section of the paper.
clarity,_quality,_novelty_and_reproducibility: This is a very clear paper, the contribution seems novel and important.

Extensive description of the model and experimental setting is given in the paper and appendix. One notable exception is the value used for the $L$ hyperparameter in the various experiments that I could not find. The author plan to release the full source code if the paper is accepted.
summary_of_the_review: This is a good paper that provides a meaningful contribution to the question of generative modelling on tabular data.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper adds to the emerging literature on deep networks for tabular data. This paper focuses on generative models, with the goal being specifically to generate synthetic data that has similar properties to a real tabular dataset. The paper proposes a VAE-based approach, in which the decoder features a graph neural network, with a node for every variable.

There was significant disagreement regarding the paper. The AC and the reviewers engaged in an active discussion, including an in-person meeting, where we discussed the points from the contrasting reviews and the author response. This meeting clarified the issues, but did not yield consensus. For this reason, I carefully reviewed the paper, reviews, and author response in order to make my recommendation.

The reviewers agreed that the architecture is conceptually interesting. The reviewers praised the breadth of the experimental evaluation, particularly in the choice of baselines. 

One reviewer cited the “limited novelty” --- I would agree with this, if it is taken to refer to the statement that essentially what the paper is doing is modifying the decoder of a VAE from an MLP to a GNN --- this does not seem a large change, so one could argue that this paper is technically fairly similar to existing work. However, a priori, it would not be clear to me that a GNN would be beneficial for this task, maybe a MLP is good enough. Therefore, I believe this experimental demonstration makes a valuable contribution, especially when (as here) the graph structure is learned as well. 

About the evaluation metrics: I agree with reviewer bxj6 that it does not seem meaningful to average alpha-precision, beta-recall, and authenticity from Alaa et al. Nor does it seem right to present these as orthogonal dimensions to the detection AUC. This can be seen from the authors’ definition of quality vs detection (“samples should be realistic” vs “samples should be indistinguishable from the real data”). How are these concepts different? Surely if a sample is realistic, it is difficult to distinguish from the real data, and vice versa? I agree with the reviewer that it would be better to present the three quality methods separately. I strongly encourage the authors to do so. I think the other two metrics in this paper are reasonable, even if I am not sure if the detection metric provides additional information beyond that in alpha-precision and beta-recall.

Overall though, my opinion is that this shortcoming of the paper is not serious enough to seriously jeopardize the main findings of the paper, so I recommend that this paper be accepted.

 
Minor comments:

Thanks for adding the mixture model (MM) baseline. I believe that you need to add it to the list of “benchmarks” in Section 5. Also, to be fair, there should be an option for the number of mixture components that causes the number of parameters (i.e., means, variances, mixture weights) to roughly match those of the neural networks. Has that been done?

justification_for_why_not_higher_score: A broader set of datasets (e.g. higher dimensional) might help to make this paper of interest to a wider subset of the community.
justification_for_why_not_lower_score: The experimental comparison of an MLP decoder vs a GNN encoder makes a potentially useful contribution.
Decision: Accept: poster