Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission is about causal structure learning and how some simulated directed acyclic graphs (that are used for benchmarking structure learning approaches, and that are generally used for causal representations) can have properties (in particular, marginal variances and also covariance patterns) that could render some structure learning algorithms more efficient, even though that might be not necessarily replicable for real-world data; that is highlighted a concern.

The authors describe a measure, varsortability, that relates to a relationship between marginal variances and the causal structure in a causal graph. They discuss how identifiability might be affected by that. The submission shows how, and explains potentially why, the varsortability of causal graphs and the data simulated/generated from them might interplay with the performance of some structure learning algorithms (in particular, continuous structure learning methods), and how that performance might be explained and depend (potentially, inadvertently) up to some significant extent by the varsortability (in particular, for simulated DAGs). The submission covers that through some theoretical analysis and empirical studies. Because marginal variances and varsortability depend on the choice of the scales of variables, the submission raises concerns about the usefulness/interpretation of existing benchmarks (particularly if no experiments with data standardisation are performed) and their results for existing structure learning algorithms. The submission also presents a baseline structure learning method that relies on marginal variances; that method can be used as a baseline for benchmarking. The theoretical results in the submission is for linear additive noise models; the empirical experiments cover both some linear and non-linear settings. The empirical experiments (and results) cover simulated and also some real-world data. In addition to discussing the points that relate to marginal variances, the submission also has a part that relates to covariance patterns and how those might influence the benchmarking in a similar way. The paper finishes with a conclusion and discussion, including some recommendations/comments as part of that.

main_review: The paper is written very well. I believe it is a very important paper. The paper is significant because it raises valid concerns about existing benchmarking (and results received with them) in a constructive way (see the summary section of the review for more details on the contribution). Thanks extremely much to the authors for their work on this.

Many related work papers are referred to in the submission. The submission naturally relies on previous work. Based on that coverage, it is assumed that the idea and implementation, as a whole, are original.

Some clarity points:

* The sentence on lines 53-56: it might be helpful to clarify for a reader whether this statement (“... cannot rely …”) is covered in those references or it is a finding/point of this submission.

* Line 90: “inadvertent” (also similarly in line 319 and “inadvertently” in line 1) - are they inadvertent or potentially “advertent”/natural (and the submission also refers to this question in lines 375-377)? (As far as the variables have the same meaning and it is natural for them to have the same scale?)

* Some clarification/more details might be helpful for lines 169-177:
** Line 172: “... obtaining varsortability for each individual …” - is something like “a varsortability component/part” meant? (since a pair of nodes is just a part of a graph)
** Regarding sentences “Even … quickly” and “The further … to the causal order” - I assume the authors might be (partially) relying on the Appendix B regarding those statements. However, those statements are not trivial, to the best of my understanding, and a proof (or at least a sketch) might be essential for them. (For example, can there be some cancellation of noise on a causal path along a causal order? (As the authors mention as well in lines 212-214.))
** Philosophically, how can we be certain that those properties are not replicated in the real world (for at least some data)?

* Line 179: “If varsortability v = 1, the causal structure is identifiable” - this is a strong statement. I see how it could be potentially (sometimes?) identifiable (from a “partial” causal order), but I don’t see why the causal structure would be necessarily identifiable in general, especially from noisy data. A proof, a sketch and/or a reference would be very helpful.

* Lines 193-194: if we don’t know the true data scale (and maybe there is no _true_ scale) but variables have the same meaning (e.g. seconds or meters), would a scale matter (at least for ANMs) as far as the scale that is used the same (e.g. seconds and seconds but not hours)?

* Figure 1: “The performance of _sortnregress_ … matches that of …” - it is not an exact match. (And someone could argue NOTEARS performs somewhat better than sortnregress in some cases, although it is not clear whether there are significant differences.) It might be worth highlighting that.

* Line 358: “later” - just to check, later to this submission’s first publication (e.g. on arXiv)?

Some other/minor points:

* It might be helpful to define “DAG” in the abstract.

* Line 9: “where varsortability may be absent” - can a measure be absent? (Is it meant that varsortability is low or equal to 0?)

* Line 10: “no longer identify” - do they rather just (sometimes) identify it, generally, much worse? (They still might perform somewhat better than the “randomregress”, e.g. see Figure 1, top-right. Before, they also did not identify it perfectly.)

* Lines 14-15: “[existing?] generic benchmarks”

* Line 43: it might be helpful to define the “MSE”. (It is appreciated that there is some definition after the line 139.)

* Line 94: “i.i.d.” presumably refer to any x^(i) overall (i can be any from 1 to n) and not to dimensions - it might be not clear from the first read although that is probably obvious in this context.

* Line 101: “if and only if” - a reference to a proof might be helpful, even if trivial.

* The equation after line 159: it might be helpful (although probably trivial) to describe what does “i -> j \in E^k” mean.

* It might be helpful to cover what happens with the varsortability if e.g. a graph does not have any edges.

* More derivation details for section 3.4 might be helpful (e.g. in an appendix section).

* Can varsortability be measured (approximately, using empirical variances) for the real dataset (as in the appendix G)?

* Line 328, “from finite-sample to the population setting” - just to check, is it a reference to sections “D.1 Infinite Sample” and “D.2 Finite Sample” in the appendix (or to some other section(s))?

* Line 360: “score-equivalend” -> “score-equivalent” ?

* Line 371: “is needed to .” - potentially, some missing (or extra) words?

* Note that the appendix was looked at (for some parts) but was not carefully checked.

*

Given some of the clarity points, although I think this work is very important, I am sorry, sadly I don’t feel comfortable at this point to recommend its acceptance. There is a chance though that: (a) I did misunderstand some points; (b) some points are obvious and they just were not mentioned by the authors; (c) or/and some points are valid and they can be addressed by the camera-ready version submission if the paper is accepted. Given the authors’ rebuttal, to which I am looking forward, there is a chance that the score will be adjusted. Given the importance of the paper, it probably would be very beneficial for the field if it is accepted to facilitate the discussion on this important topic.

All parts of the review are provided to the best of my understanding and knowledge. Mistakes/misunderstandings in the review are possible. I kindly ask to point me to them, if any, if possible, please.

*

* A discussion point (of the review): The paper topic relates to an important question on whether scales are important for structure learning tasks (and how should benchmarks be constructed; and what should metrics be). If some variables have the same meaning, does it make sense (and is it essential for full possible “identifiability”) to use their scales? (For example, if two variables concern time, is it natural to use seconds and seconds (without any standardisation)? And also, potentially, generally to use a consistent metric system, assuming there are some natural connections between different base units, or/and they can be naturally expressed through one another?)
** (It also brings an interesting question on how a structure learning algorithm can try to incorporate information about different units that can’t be necessarily naturally expressed through each other (e.g. numbers of people and currency (money) units) using some meaningful statistical information that describes some relationships between them (e.g. an average income or some average expense per a person); but it is out of the scope of this review.)

*

**For the update, please, see my message dated the 2nd of September 2021 (GMT).**

limitations_and_societal_impact: * There is a question: to what extent is high varsortability natural in the real world? Is it generally natural? (E.g. for variables where scales can be chosen.) Authors acknowledge this question too (e.g. in lines 375-378). I wonder whether it should be covered a bit more earlier in the paper as well (e.g. in the abstract or on the first pages)?
** If that is the case (that it is natural), then would it be appropriate to use algorithms that rely on scales in practice (as far as scales are (somewhat) consistent)? Even though (some of) their performance is explained by varsortability, is it fine (and desired?)?

* (I assume it is a general issue in the field, and that is not trivial to resolve:) There seems to be a limited set of real-world, expert-annotated datasets.
** It is also interesting to note, based on the authors’ note about that, that on the one real-world dataset that was used in the paper, for SID the algorithms perform generally quite similarly (the appendix, line 761) and, if I understand correctly, not very well (although NOTEARS on the raw data seems to be performing relatively well on SID, and sortnregress too).

*

Societal impact:

There are generally potential huge positive impacts of the paper, and that is really appreciated: the submission raises questions and contains work that are important for the field and its applications (and that is expected to lead to positive social impact). The authors make a statement regarding that in the paper checklist.

(Further study is required to understand better e.g. whether high varsortability is natural for some real-world data (given some assumptions about scale relationships); the authors make a similar statement in lines 375-378 regarding additive noise models. Note that those further studies would affect further evaluation of different (existing and new) structure learning algorithms (and development of new ones), and further analysis and development of benchmarks and metrics. An obvious potential negative social impact might be if e.g. (a hypothetical scenario) varsortability of real-world datasets would be for some reason generally originally underestimated and if that would for some reason lead to bias towards some algorithms (e.g. if they don’t perform well on standardised data) until later it would be established that high varsortability is actually a natural property (for some real-world data) and identifiability significantly depends on it (again, that is just a hypothetical scenario).)
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper shows that one needs to be cautious when rescaling your data prior to using a causal structure learning algorithms. The authors introduce the concept of varsortability that measures the agreement in how much the marginal variance tends to increase along the causal order. They show that standardization of your data can hurt performance in identifying the DAG (or its equivalence class) which can be explained by varsortability. The authors claim that this concept of varsortability also explains why even after standardization certain continuous structure learning algorithm perform well. The authors focuses on additive noise models and perform an extensive benchmark.

main_review: The paper is very well written and a pleasure to read. The authors shed some new light on current causal structure learning algorithms by introducing the intuitive concept of varsortability. They show that varsortability is high in common simulation schemes used for benchmarking causal structure learning algorithms. Interestingly, they show that high varsortability may still be exploited after data standardization and may explain why certain continuous structure learning algorithms work so well. They motivate this possible explanation by matching the performance of their algorithm sortnregres to other algorithms. These results, however, do not imply, or proof, that varsortability is a "causal" explanation of why other continuous learning algorithms work so well in the presence of high varsortability. In Section 3.4 the authors give a heuristic argument why this is the case, but the paper would be much stronger if the authors could actually proof this, and state precisely under which circumstances this would hold. Part of the arguments of this section could I think be improved upon. For example, the authors state that:
- line 239: "We expect the residuals after the first gradient step ... to be larger ..."
- line 254: "b) why performance changes once data is standardized ... causal order."
without further explanation. It would be helpful if the authors could elaborate more on this. Nevertheless, the concept of varsortability and the benchmarking results will still be a valuable contribution to NeurIPS.

Minor comments:
- line 217-221 "In general, ... causal order.": What motivates these claims? I think a reference would be appropriate.
- line 226-229: "We do not expect combinatorial ... to be dependent on data scale."  Why is that?
- line 310 "are driven by": Really? Or should this read as "may be driven by"?
- line 292 "sortnregress": Why not emphasize more that this is a novel contribution?
- Why Figure 1 only for ER-2 graphs and Table 2 only for ER-4 and SF-4 graphs?

limitations_and_societal_impact: One of the limitations of this work was already mentioned in the main review section. Moreover, the authors discuss the further limitations, such as examining the role of varsortability in non-linear and empirical settings in detail in Section 6. The authors aim to raise awareness of the severity by which scaling properties in data may distort structure learning algorithm performance makes this paper a valuable contribution to NeurIPS.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper considers common DAG simulation schemes and and defines and studies a property that results, which does occur in real data, but which may be specifically exploited by continuous causal structure learning algorithms like NOTEARS and GOLEM which case causal structure learning as a differentiable optimization problem, essentially allowing them to game common benchmarks.

The key property that results from common DAG simulation schemes is downstream nodes in the causal graph having higher variance. The paper defines "varsortability" as the fraction of directed paths of any length which start from a node with strictly lower variance than the node they end in. The paper finds that forward sampling in the linear and additive noise model settings result in varsortability of ~ .7 and ~.9, respectively. 

The paper then goes on to show how varsortability guides the gradient-based steps when continuous causal structure learning algorithms like NOTEARS and GOLEM are used. 

Next, the paper caries out simulations which compare combinatorial vs. continuous causal structure learning algorithms (and a baseline that relies only on varsortability to obtain the causal order) in settings where the simulated data is standardized and not standardized. The results show that continuous causal structure learning algorithms show significantly higher performance due to varsortability whereas combinatorial algorithms are roughly unaffected.

The paper further elaborates that even standardized data where prior to standardization there was high varsortability may still allow allow gaming due to distinct covariance patterns.
main_review: The paper presents a very clear and thoughtful analysis of current methods for benchmarking causal discovery algorithms and identifies an important problem, which I don't think has previously been addressed rigorously. 

Since the paper also addresses the core question of how we should be benchmarking causal discovery algorithms I think the significance and novelty are both high. 

Finally the paper is well written and organized, reviews existing literature thoroughly and the main observations regarding varsortability and continuous causal discovery algorithms are well supported with experiments. 

I don't have any significant concerns. My main criticism would be the claims in section 5 remain unclear - this may be more clear in the appendix (which I am unable to review given the overload of papers this NeurIPS cycle), but I still support accepting this paper based on the content in the other sections. 
limitations_and_societal_impact: Not really discussed - but I don't foresee negative impacts
review_title: null

Meta-review: A summary from one of the reviews:

"This paper shows that one needs to be cautious when rescaling your data prior to using a causal structure learning algorithms. The authors introduce the concept of varsortability that measures the agreement in how much the marginal variance tends to increase along the causal order. They show that standardization of your data can hurt performance in identifying the DAG (or its equivalence class) which can be explained by varsortability. The authors claim that this concept of varsortability also explains why even after standardization certain continuous structure learning algorithm perform well. The authors focuses on additive noise models and perform an extensive benchmark."

While the initial reviewer opinions were split, the eventual consensus on this paper is that it brings a valuable message of caution regarding developing and benchmarking causal inference methods with simulated data.
Decision: Accept (Poster)