Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The present submission discusses a "causal regularizer", which promotes the use of causal dependencies (X -> Y, where X is a feature of the learning problem, and Y is the target variable) in predictive models. Similarly, such causal regularizer penalizes the use of non-causal dependencies, which can arise due to reverse causation (Y -> X) or confounding (X <- Z -> Y, where Z is a hidden confounder).

+ Overall, this submission tackles one of the most important problems in machine learning, which is to build causal models. The paper discusses and addresses this issue effectively when applied to a dataset in heart disease. In their experiments, the authors correctly identify some of the common causes of heart disease by virtue of their causal regularizer.

- The authors do not discuss the robustness of their approach with respect to choice of hyper-parameters (both describing the neural network architecture and the generative model that synthesizes artificial causal data). This seems like a crucial issue, in particular when dealing with medical data.

- The conclusions of the experimental evaluation should be discussed in greater length. On the one hand, Figure 4.a shows that there are no differences between L1 and causal regularization in terms of predictive performance, but it is difficult to conclude if this result is statistically significant without access to error-bars. On the other hand, Table 3 describes the qualitative differences between L1 and causal regularization. However, this table is hard to read: How were the 30 rows selected? What does the red highlighting mean? Are these red rows some true causal features that were missed? If so, this is related to precision. What about recall? Did the causal regularization pick up many non-causal features as causal?

- Regarding causal classifiers, this paper should do a much better job at reviewing previous work. For instance, the paper "Towards a Learning Theory of Cause-Effect Inference" from Lopez-Paz et al. is missing from the references. However, this prior work studies many of the aspects that are hinted as novel in this submission. In particular, the prior work of Lopez-Paz 1) introduces the concept of Mother distribution (referred as Nature hyper-prior in this submission) which explicitly factorizes the distribution over causes and mechanisms, 2) circumvented intractable likelihoods by synthesizing and training on causal data, 3) tackled the confounding case (compare Figure 1 of this submission and Appendix C of Lopez-Paz), and 4) dealt with discrete data seamlessly (such as the ChaLearn data from Section 5.3 in Lopez-Paz).

On a positive note, this is a well-written paper that addresses the important, under-appreciated problem of incorporating causal reasoning into machine learning. On a negative note, the novelty of the technical contributions is modest and the qualitative evaluation of the results could be greatly extended. In short, I am leaning slightly towards acceptance.
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to use a causality score to weight a sparsity regularizer.  In that way, selected variables trade off between being causal and discriminative.  The framework is primarily evaluated on a proprietary health dataset.  While the dataset does give a good motivation to the problem setting, the paper falls a bit short for ICLR due to the lack of additional controlled experiments, relatively straightforward methodology (given the approach of Chalupka et al., arXiv Preprint, 2016, which is a more interesting paper from a technical perspective), and paucity of theoretical motivation.

At the core of this paper, the approach is effectively to weight a sparsity regularizer so that "causal" variables (as determined by a separate objective) are more likely to be selected.  This is generally a good idea, but we do not get a proper validation of this from the experiments as ground truth is absent.  A theorem on identifiability of causal+discriminative variables from a data sample combined with adequate synthetic experiments would have probably been sufficient, for example, to push the paper towards accept from a technical perspective, but as it is, it is lacking in insight and reproducibility.
review_title: Possibly practical method, but methodology and analysis fall short

final_score: rating: null
review_content: Algorithm 2 has notation that I find somewhat confusing.  Step 1 generates data from $p_{X,Y}$, but in my reading of the paper there is no place that actually indicates what those distributions are: I understand that Fig 1 has the graphical models, but what is the actual form of the distribution?  Does this choice influence the results?  Next, Step 2 says that the classifier is a map from measures $\mathbbm{P}_{X,Y}$ to $[0,1]$.  But how is that the input?  What form is given for the measure?  Sufficient statistics or some other form?  Alternatively, perhaps you mean that the input are samples.  If so, how many samples, and how does this affect the analysis?  This is one of the two key methodological points in the paper, and I think it warrants more specificity and mathematical detail than is currently present.
review_title: samples from an observational distribution

final_score: rating: null
review_content: The method is to apply a weighted L1 regularizer where the weights are proportional to a causality score determined by some method (in this case the authors' own).  The experiments then show that selected variables on a real world data set (with no ground truth as far as I saw) have a higher causality score as regularization is increased.  This isn't so surprising.  Perhaps a more appropriate evaluation would be for some dataset where causality is known, e.g. from controlled trials.
review_title: ground truth causality?

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors extend their method of causal discovery (Chalupka et al 2016) to include assumptions about sparsity via regularization.  They apply this extension to an interesting private dataset from Sutter Health.  While an interesting direction, I found the presentation somewhat confused, the methodological novelty smaller than the bulk of ICLR works, and the central results (or perhaps data; see below) inadequate to address questions of causality.

First, I found the presentation somewhat unclear.  The paper at some points seems to be entirely focused on healthcare data, at other points it uses it as a motivating example, and at other points it is neglected.  Also, algorithm 1 seems unreferenced, and I'm not entirely sure why it is needed.  Figure 2 is not needed for this community.  The key methodological advance in this work appears in section 2.1 (Causal regularizer), but it is introduced amidst toy examples and without clear terminology or standard methodological assumptions/build-up.  In Section 3.1 (bottom of first paragraph), key data and results seem to be relegated to the appendices.  Thus overall the paper read rather haphazardly.  Finally, there seems to be an assumption throughout of fairly intimate familiarity with the Cholupka preprint, which i think should be avoided.  This paper should stand alone.

Second, while the technical contributions/novelty are not a focus of the paper's presentation, I am concerned by the lack of methodological advance.  Essentially a regularization objective is added to the previous method, which of itself is not a bad idea, but I can't point to a technical novelty in the paper that the community can not do without.

Third, fundamentally i don't see how the experiments address the central question of causality; they show regularization behaving as expected (or rather, influencing weights as expected), but I don't think we really have any meaningful quantitative evidence that causality has been learned.  This was briefly discussed (see "ground truth causality?" and the response below).  I appreciate the technical challenges/impossibility of having such a dataset, but if that's the case, then I think this work is premature, since there is no way to really validate.

Overall it's clearly a sincere effort, but I found it wanting in terms of a few critical areas.
review_title: a good paper but a bit short of the mark in terms of presentation, methodology, and results

final_score: rating: null
review_content: While the paper provides enough detail to re-implement the proposed methods, there is no discussion as to whether the heart failure dataset will be released. This would be key for reproducibility.
review_title: Reproducibility

Meta-review: The reviewers pointed out several issues with the paper, and all recommended rejection. The revision seems to not have been enough to change their minds.
Decision: Reject