Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the synthesis of privacy-preserving datasets from an interesting angle: It claims knowledge about the causal relationship of the data attributes can simultaneously amplify privacy and improve accuracy. 
To shed light on this surprising result, the paper analyzes in which condition a generative model equipped with a causal graph decrease the sensitivity of the data release task, thus increasing accuracy. Surprisingly this also reflects in better model performance against membership inference attacks. 
The paper then proposes a VAE-based mechanism to generate causally consistent datasets and provide results over several benchmarks.
strength_and_weaknesses: **Presentation**
The paper is generally well-written; There are a few typos and errors but are not affecting the overall readability. 
In terms of claims, I would refrain, in the Introduction, to say with such certainty that Membership inference attacks are breaking synthetic data generation models. This is not only an overstretch but is also dangerous as readers may mistrust synthetic data generators, which, in my opinion, provide one of the best alternatives for privately releasing datasets. 

**Related work**
The similarity of the proposed method with PrivBayes [1] should be discussed. E.g., where and why, in the proposed approach, the causal representation is different from what done by PrivBayes? 

**Technical Questions**
- Can you clarify when the sensitivity of (1) will be smaller than (3) w.r.t. the properties of the causal relationships? Does this hold only when there exists at least one variable with no dependency on y? (see proof of Lemma 1).
- On page 4, under paragraph "why does causality provide any privacy benefit?" the paper claims that "the model will be more stable". Can the authors elaborate further? What assumptions are taken in terms of causal graph dependencies? See also the question I ask above.
- Why are the values of the MI attacks so high? 
- The results also do not seem to show that MI attacks can be better overcome (thus better privacy protection) in the causal setting considered. Can you elaborate on your claims? 

[1] PrivBayes: Private Data Release via Bayesian Networks. ACM Transactions on Database Systems 42(4):1-41, 2017
clarity,_quality,_novelty_and_reproducibility: - The paper is relatively well-written. Additional clarifications on some general aspects and assumptions would have been appreciated. 
- I'd like to authors to stress further how this proposal departs from other generative models that exploit some prior distributions. 


Minor Errors
- [page 1] The de-facto mechanism --> The de-facto standard mechanism 
- [page 1] a more border -> a broader 
- [page 1] to validate the theoretical results -> why do you need to validate your theory? Use another word.
- [page 4] the the true -> the true

summary_of_the_review: The paper studies a very important problem: the release of privacy-preserving data through synthetic generators. 
The claims that privacy and accuracy can be both amplified by using additional side information are counter-intuitive to me and require further explanation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tries to measure whether adding causal information to a differentiably private generative model allows for a more favorable utility-privacy tradeoff curve.
strength_and_weaknesses: STRENGTHS
- Rather than just relying on the privacy budget (epsilon) to quantify privacy, this work uses  susceptibility to a membership inference attack. It is very valuable to explore quantifiable notions of privacy based on risk (eg in order to compare privacy risks between DP-based methods and non-DP methods).

WEAKNESSES
- I am not persuaded that this was a good measure of "utility." Page 5 describes that the metric involves randomly selecting 20 attributes to be predicted, rather than picking actually useful targets. This is especially surprising given this work's emphasis on the importance of causal and meaningful relationships in data generation. I find the approach concerning because it is essentially performing an *attribute* inference attack (ie if I know someone's age, gender, etc, can I predict their HIV status using the synthetic data?). Without more details & careful thought, this metric could arguably be another measurement of risk/privacy instead of utility. 
- On multiple occasions, the work explicitly stresses that its contribution is meant to be severable from the construction of a given causal graph. However, I'm not sure such a separation can be so clean. The field of causal inference suffers in reputation (and ultimately empirical performance) because although "the causal graph can be constructed using domain expertise" in theory... in practice, trying to construct a meaningful graph often leads to over-promise/under-deliver. The work could benefit from showing the CG for the 3-node experiment.
clarity,_quality,_novelty_and_reproducibility: Clarity: The clarity of this work can be improved. The datasets (page 8) and experiments (page 7) are not described until the nearly the end of the paper, which makes it very confusing to read the results on page 6. There were, however, some helpful parts, such as the description of the notation on page 3 (eg upper-case variables = sets, etc).

Quality: This paper seems to still be a bit of a work-in-progress. It shows promising directions, particularly w.r.t. evaluating privacy extrinsically without just reporting the privacy budget. However, the experiments and presented results are a bit scattered in terms of the picture that is trying to be painted.

Novelty: This work is moderately novel. It aims to use causal information to structure a differentiably private generative model. All of the techniques are reasonably well-used, but combining them has some degree of novelty.

Reproducibility: There is no code available. This work would likely be difficult to reproduce without additional resources and guidance.
summary_of_the_review: This paper seems to still be a bit of a work-in-progress. It shows promising directions, particularly w.r.t. evaluating privacy extrinsically without just reporting the privacy budget. However, the experiments and presented results are a bit scattered in terms of the picture that is trying to be painted. The two largest areas of improvement I would suggest are: 1) either reworking the notion of utility or at least further justifying why it is appropriate; and 2) re-organizing some of the sections of writing so that the experiments flow more meaningfully and coherently.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the effect of adding causality to differentially private generative models. A theorem is proved showing that the sensitivity of a causally informed mechanism is lower than a non-causal mechanism. Experiments show that using causal information with the same privacy budget results in models which perform better on downstream tasks than generative models which do not use causal information.
strength_and_weaknesses: Strengths
- The paper address a topic of relevance to multiple research communities 
- The paper includes a nice mix of theory and empirical results
- The paper is generally well written

Weaknesses
- There appear to be gaps between the claims made in the paper and the results which makes the significance unclear (see below)
- Some results are not well explained
- Some assumptions may not align to realistic settings
- It is not clear how robust some of the empirical results are
clarity,_quality,_novelty_and_reproducibility: Causality is central to the claim made by the paper, but it is not clear from the results (theoretical or empirical) than the improved privacy/utility tradeoff is due to causal information or rather from just being able to factorize the distribution correctly / having a sparser model. The paper even indicates at the end of page 4 that this is why performance is better, but then does not address whether performance would be the same or different if you simply used a different (incorrect) causal model from the same Markov equivalence class. Thus, it is not clear the claims about causality are correct.

If the impact is due not to causality, but rather having a sparser model, then the results are maybe not surprising (you’ve simply eliminated some unassociated variables that would add noise) and hence can get a better privacy/utility tradeoff. In most cases, however, you would not have this sparse model for free, but would need to learn it and dedicate some of your privacy budget to learning the correct Markov equivalence class. The paper discusses this, but most of the empirical results assume some knowledge of the underlying graph.

The empirical results also leave some questions unanswered. In many cases using the synthetic data (even in some cases with DP) actually consistently increases performance on the downstream task even over the real data, which is surprising since the generative model is not doing any augmentation aimed at increasing performance on the specific task. This is not explained in the text and the performance differences are listed without confidence intervals so it’s unclear how robust these results are.
summary_of_the_review: The paper addresses an interesting problem and includes a nice mix of theoretical and empirical results, but there appear to be some gaps between the theory and claims and the robustness of the empirical results are unclear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper claims new empirical and theoretical results on the stability of causality constraints.  Reviewers generally liked the research direction but find some of the claims unclear. Specifically, the theoretical part and experimental part of the paper seem to be only connected on the very high level.  The theoretical part focuses on analyzing the stability of the fitted parameters with or without a causal graph, when the dataset is generated according to a causal graph.  The connection to the experiments, in which the authors used the standard DP-SGD for training a VAE, is unclear.   It is also unclear how the stability of the fitted parameters implies a "privacy amplification by causality". No claims about differential privacy were given, nor how the parameters of DP improve with causality. 

Overall, there is not enough support among reviewers to have a more in-depth discussion to investigate the possibility of acceptance.  
justification_for_why_not_higher_score: The paper does not fall into the ballpark of a borderline paper, thus triaged.
Decision: Reject