Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors propose to use the so-called implicit model to tackle Genome-Wide Association problem. The model can be viewed as a variant of Structural Equation Model. Overal the paper is interesting and relatively well-written but some important details are missing and way more experiments need to be done to show the effectiveness of the approach.

*  How do the authors call a variant to be associated with the phenotype (y)? More specifically, what is the distribution of the null hypothesis? Section D.3 in the appendix does not explain the hypothesis testing part well. This method models $x$ (genetic), $y$ (phenotype), and $z$ (confounder) but does not have a latent variable for the association. For example, there is no latent indicator variable (e.g., Spike-Slab models [1]) for each variant.  Did they do hypothesis testing separately after they fit the model? If so, this has double dipping problem because the data is used once to fit the model and again to perform statistical inference. 

* In GWAS, a method resulting in high power with control of FP is favored. In traditional univariate GWAS, the false positive rate is controlled by genome-wide significant level (7e-8), Bonferroni correction or other FP control approaches.  Why Table 1 does not report FP? I need Table 1 to report the following: What is the power of this method if FPR is controlled(False Positive Rate < 0.05)? Also, the ROC curve for FPR<0.05 should be reported for all methods. 

* I believe that authors did a good job in term of a survey of the available models for GWA from marginal regression to mixed effect model, etc. The authors account for typical confounders such as cryptic relatedness which I liked. However, I recommend the authors to be cautious calling the association detected by their method " a Causal Association." There are tons of research done to understand the causal effect of the genetic variants and this paper (and this venue) is not addressing those.  There are several ways for an associated variant to be non-causal and this paper does not even scratch the surface of that. For example, in many studies, discovering the causal SNPs means finding a genetic variant among the SNPs in LD of each other (so-called fine mapping). The LD-pruning procedure proposed in this paper does not help for that purpose. 

* This approach jointly models the genetic variants and the phenotype (y). Let us assume that one can directly maximize the ML (ELBO maximizes a lower bound of ML). The objective function is disproportionally influenced by the genetic variants (x) than y because M is very large (  $\prod_{m=1}^M p(w) p(x|z,w,\phi)   >>   p(z) p(y|x,z,\theta) $  ). Effectively, the model focuses on the genetic variants, not by the disease. This is why multi-variate GWAS focuses on the conditional p(y|x,z) and not p(y,x,z). Nothing was shown in the paper that this focusing on p(y,x,z) is advantageous to p(y|x,z). 

* In this paper, the authors use deep neural networks to model the general functional causal models. Since estimation of the causal effects is generally unidentifiable (Sprites 1993), I think using a general functional causal model with confounder modeling would have a larger chance to weaken the causal effects because the confounder part can also explain part of the causal influences. Is there a theoretical guarantee for the proposed method? Practically, how did the authors control the model complexity to avoid trivial solutions?

Minor
-------
* The idea of representing (conditional) densities by neural networks was proposed in the generative adversarial networks (GAN). In this paper, the authors represent the functional causal models by neural networks, which is very related to the representation used in GANs. The only difference is that GAN does not specify a causal interpretation. I suggest the authors add a short discussion of the relations to GAN.

* Previous methods on causal discovery rely on restricted functional causal models for identifiability results. They also use Gaussian process or multi-layer perceptron to model the functions implicitly, which can be consider as neural networks with one hidden layer. The sentence “These models typically focus on the task of causal discovery, and they assume fixed nonlinearities or smoothness which we relax using neural networks.” in the related work section is not appropriate. 

[1] Scalable Variational Inference for Bayesian Variable Selection in Regression, and Its Accuracy in Genetic Association Studies
review_title: Interesting idea but needs more experiments and justification since it is a vast field and not all aspects of the problem is accounted for.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a non-linear generative model for GWAS that models population structure.
Non-linearities are modeled using neural networks as non-linear function approximators and inference is performed using likelihood-free variational inference.
The paper is overall well-written and makes new and non-trivial contributions to model inference and the application.
Stated contributions are that the model captures causal relationships, models highly non-linear interactions between causes and accounts for confounders. However, not all claims are well-supported by the data provided in the paper. 
Especially, the aspect of causality does not seem to be considered in the application beyond a simple dependence test between SNPs and phenotypes.

The paper also suffers from unconvincing experimental validation:
- The evaluation metric for simulations based on precision is not meaningful without reporting the recall at the same time.

- The details on how significance in each experiment has been determined are not sufficient.
From the description in D.3 the p-value a p-value threshold of 0.0025 has been applied. Has this threshold been used for all methods?
The description in D.3 seems to describe a posterior probability of the weight being zero, instead of a Frequentist p-value, which would be the probability of estimating a parameter at least as large on a data set that had been generated with a 0-weight.

- Genomic control is applied in the real world experiment but not on the simulations. Genomic control changes the acceptance threshold of each method in a different way. Both precision and recall depend on this acceptance threshold. Genomic control is a heuristic that adjusts for being too anti-conservative, but also for being too conservative, making it hard to judge the performance of each method on its own. Consequently, the paper should provide additional detail on the results and should contrast the performance of the method without the use of genomic control.

minor:

The authors claim to model nonlinear, learnable gene-gene and gene-population interactions.
While neural networks may approximate highly non-linear functions, it still  seems as if the confounders are modeled largely as linear. This is indicated by the fact that the authors report performance gains from adding the confounders as input to the final layer.

The two step approach to confounder correction is compared to PCA and LMMs, which are stated to first estimate confounders and then use them for testing.
For LMMs this is not really true, as LMMs treat the confounder as a latent variable throughout and only estimate the induced covariance.

review_title: The paper is overall well-written and makes new and non-trivial contributions to model inference and the application. However, not all claims are well-supported by the data provided in the paper. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tackles two problems common in genome-wide association studies: confounding (i.e. structured noise) due to population structure and the potential presence of non-linear interactions between different parts of the genome. To solve the first problem this paper effectively suggests learning the latent confounders jointly with the rest of the model. For the second problem, this paper proposes “implicit causal models’, that is, models that  leverage neural architectures with an implicit density. 

The main contribution of this paper is to create a bridge between the statistical genetics community and the ML community. The method is technically sound and does indeed generalize techniques currently used in statistical genetics. The main concerns with this paper is that 1) the claim that it can detect epistatic interactions is not really supported. Yes, in principle the neural model used to model y could detect them, but no experiments are shown to really tease this case apart 2) validating GWAS results is really hard, because no causal information is usually available. The authors did a great job on the simulation framework, but table 1 falls short in terms of evaluation metric: to properly assess the performance of the method on simulated data, it would be good to have evidence that the type 1 error is calibrated (e.g. by means of qq plots vs null distribution) for all methods. At the very least, a ROC curve could be used to show the quality of the ranking of the causal SNPs for each method, irrespective of p-value cutoff.

Quality: see above. The technical parts of this paper are definitely high-quality, the experimental side could be improved.
Clarity: if the target audience of this paper is the probabilistic ML community, it’s very clear. If the statistical genetics community is expected to read this, section 3.1 could result too difficult to parse. As an aside: ICLR might be the right venue for this paper given the high ML content, but perhaps a bioinformatics journal would be a better fit, depending on intended audience.

review_title: good paper, unclear audience

Meta-review: The reviewers agree that the work is high quality, clear, original, and could be significant.

Despite this, the scores are borderline. The reason is due to rough agreement that the empirical evaluations are not quite there yet. In particular, two reviewers agree that, in the synthetic experiments, the method is evaluated on data that is an order of magnitude too easy and quite far from the nature of real data, which has a much lower signal to noise ratio.

However, the authors have addressed the majority of the concerns and there is little doubt that the authors are capable of carrying out this new experiment and reporting its results. Even if the results are surprising, they should shed light on what seems to be an interesting new approach.
Decision: Accept (Poster)