Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an algorithm of synthetic data generation for many-to-many dataset. In particular, the authors decompose the overall distribution into three components and model the relationship in each component independently. Furthermore, the authors also present a private version of the algorithm and conduct experiments on real-world datasets.
strength_and_weaknesses: Strength:

Developing differentially private SDG is an important and well-motivated topic. The idea to decompose the joint distribution into multiple components and treat each component separately is interesting. 

Weakness:

1. The privatization step in section 4.3 is straightforward and may not be scalable. It looks like in the case of prefix embeddings, there are no learnable parameters for $P(U|L)$ and $P(V|L, U)$, and the privatization process reduces into a direct application of Laplace mechanism to the BJD matrix. Moreover, the noise added to each entry of the BJD matrix scales with the number of nodes, which can be prohibitively large for complicated graphs with plenty of nodes.  Finally, the authors mention that we can use DP-SGD to train the embedding. If that is the case, it would be better if the authors can present the training objective and algorithm details in the paper.
2. The descriptions of the distribution model in section 4.2 are not clear. In particular, the authors only provide very general introduction to different node embeddings β or aggregators γ. It would be better if the authors can present more details about the how these embeddings and aggregators work. 
3. In the experiment, the authors mention that “we use node2vec to extract node embeddings”. I am wondering how the authors privatize this embedding learning step to prevent the privacy leakage.

Question:
1. This is a follow-up question about the privatization step in experiments. In section 4.3, the authors mention that "otherwise we use DP-SGD to train the embedding". I notice that the author do not mention any training details concerning DP-SGD in experiment (section 5), so my question is that whether the authors implement DP-SGD in any of the experiments. If yes, it would be great if the authors can provide more relevant implementation details.


clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. As mentioned above, there are some statements in the paper that need more clarification. I have some concerns about the novelty since the privatization part is quite straightforward and there is not any theoretical result to provide any privacy or utility guarantee.
summary_of_the_review: The paper presents a synthetic data generation algorithm for many-to-many datasets. The idea is well-motivated and the authors do a good job in presenting the algorithms. However, I am not sure the novelty is enough since the privatization part in sec 4.3 is a direct application of Laplace mechanism for pre-fixed embedding and the authors only briefly mention about training embedding privately with DP-SGD without providing any further details. Therefore, the paper can be improved if the authors can
1. Add more details about the embedding learning with DP-SGD
2. Provide theoretical guarantee (privacy or utility) of the proposed algorithm.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper consider the problem of synthetic data generation by fitting a generative model to the dataset and then drawing samples from it. The authors consider tabular datasets with many to many relationships.
The key idea that the authors propose is to decompose the likelihood in a certain way and then model each of the factors using appropriate techniques such as sampling structured graphs and constructing graph embeddings. The paper also presents a differentially private extension of the above technique. The authors conduct experiments on a benchmark dataset to adjudge the efficacy of this modeling using appropriate metrics.
strength_and_weaknesses: Strengths:

* The problem is natural since a lot of real world datasets are tabular and have many to many relationships. A generative model which preserves these properties is important for downstream usecases such as synthetic data generation. 
* The paper is well-written; it provides a systematic investigation of approaches to the problem discussing why (extensions of) prior work seem insufficient and how the authors remedy these via proposed approach. Overall, I think the write-up is generally accessible to the wider ICLR audience. There are parts where it gets a little heavy on citing and drawing from prior work (Section 4) without enough details (more on this later). 

Weaknesses:

* I think a simple small example of a (made-up) tabular dataset with many-to-many relationships will help the readers. Admittedly, I am not very familiar with this area, and so it took me some time to understand the intended use-case and how the abstraction captures the setting.

* Clarification on modelling assumptions: The proposed approach is to fit a generative model is on a given (single) dataset, which in the simplest setting is a bi-partite graph. In order for the model to learn from this single graph, there need to be (some kind of) sufficient independence between the vertices/edges which provide an overall statistical leverage. The authors do not present a formal statistical model of the underlying graph, however, when fitting the generative model, the factorization implicitly uses certain statistical assumptions (which a-priori, I am not sure if it holds generally); in particular Eqns, (1) and (2).
A formal model would help ascertain the validity of such a decomposition.
Also, a discussion on why these independence assumptions are valid in typical usecases would be helpful.

* The structure of the paper essentially presents a decomposition of the problem into many sub-problems (which is novel, as far as I understand) and then the authors draw from prior work to to solve the sub-problem. 
My issue is that the presented description of the technique is minimal and, in places, relegated to a citation. A reader who is not familiar with prior works doesn't really understand the details. Also, it gives the impression that the authors are essentially combining different techniques together.

* (Minor) When discussing various possible factorization of the joint density model, the author argue why modelling of certain factors is not a good idea. While this is good in the sense that it showcases that the authors considered alternatives before finalizing the presented approach, some of the negative aspects are rather "hand-wavy" -- saying that "model can be computational challenge to implement" is perhaps not the strongest defense against using that method.

* A differentially private extension is based on running DP-SGD on the loss obtained after parametrizing the generative model (NueralM2M). The authors say that it has worse performance than the other method. DP-SGD has a clipping parameter and the empirical performance typically is sensitive to its choice. I am wondering if the authors tried to tune this parameter and yet observe this worse performance. Some details would be helpful here.

* Some elaboration om what it means by "faithful" in the context of 
"faithful generative models" would be useful.

clarity,_quality,_novelty_and_reproducibility: I think the paper is mostly well-written and polished. There are parts which require knowledge of prior work to understand it better and more details can help here. According to the authors, modeling of tabular data with many to many relationships has been missing from literature and this work tries to address it. 
summary_of_the_review: The paper presents a systematic investigation of generative modeling of tabular data with many to many relationships, which seems like a natural and interesting problem. The proposed apporach, while natural, requires non-trivial application of prior techniques (such as random graph generation). The apporach is evaluated on a benchmark datasets and the metrics showcase improvements over a prior approach. I am mostly positive about the paper but have some basic questions (see above) which I hope the authors response will adress.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Consider the problem of synthetically generating data based on real data with multiple tables and links between tables, such as a movie recommendation dataset, and if possible, in a differentially private way.  This paper suggests a new approach for doing so and applies it to real data.
strength_and_weaknesses: Strengths
+ Solves a novel problem of generating multiple-table data with rich dependency structure.
+ Does a good job of relating the work to previous approaches and showing the novelty of the approach.

Weaknesses/Questions
- The evaluation criteria are not well explained.  Why does high similarity of mutual information indicate high-quality synthetic data?  Are the metrics used in this paper novel, or have they been previously used in the literature. [after rebuttal: addressed]
- The paper could be improved if some other evaluations founds in the literature were incorporated.  If one were to train a recommender system on the synthetic data, does it find similar features to a recommender system trained on the real data? [after rebuttal: addressed]
- Is there any way to empirically evaluate the quality of the differential privacy in hiding individuals? [after rebuttal: addressed]
- I'm not sure that "faithfulness" is really a property of this method, since it doesn't check the conditional independencies in the real data. However, I think the paper uses the term "faithfulness" to capture the fact that the model doesn't underfit the real data by assuming a constrained dependency structure.  Perhaps a better way to describe this property of the method is that it is potentially consistent for arbitrary dependency structures? [after rebuttal: addressed]

Additional questions/comments after rebuttal:
- How do prove the statement in Appendix C that n_u and n_u' are conditionally independent for sufficiently rich \beta?  Is the proof also in Wagstaff et al. 2019?
- Typo in Appendix D 'Detailed Eexplanation'

Note: I am not familiar with the differential privacy literature, so I defer to other reviewers in the evaluation of the claims of differential privacy.


clarity,_quality,_novelty_and_reproducibility: Quality:
High.  The paper explains the problem well and uses well-motivated methods to solve the problem.  Furthermore, the paper takes some effort to apply a previous approach to the problem in order to obtain a benchmark.  The evaluation appears reasonable, except the metrics are not well-motivated.

Clarity:
The paper is clearly written.

Originality:
The method solves a novel problem.
summary_of_the_review: As the paper convincingly solves a novel and interesting problem, and after rebuttal, explains the evaluation criteria and shows improved results over previous methods, I recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies synthetic data generation for tabular datasets with many-to-many relationships among them. The main novelty in this paper is modeling the many-to-many relationships via a new factorization model and proposing a flexible end-to-end framework for synthetic generation based on this new model. The framework is also extended to ensure a formal privacy guarantee via adopting the notion of approximate differential privacy. 

All reviewers agreed that the contributions are strong enough to merit acceptance and will be useful to the community, especially given the importance and the practical significance of the problem of synthetic data generation. The authors have been very responsive to the reviewers' initial concerns and made necessary adjustments and clarifications to improve the readability of their paper and to emphasize their contributions. 
justification_for_why_not_higher_score: The contributions are above the bar of acceptance but the level of impact is not very high to merit an oral or spotlight slot. 
justification_for_why_not_lower_score: The paper makes enough contributions to merit acceptance.
Decision: Accept: poster