Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper studies the identifiability of the causal relationships between continuous and discrete variables.
Main review: # Quality and clarity
The paper is not clear, not well-organized, and very hard to read, and it seems to be missing some important points.
- The aim of this study is not clearly stated in Abstract or Introduction. The aim of this study seems extending the studies by Li and Shimizu (2018) and  Wei et al. (2018). However, it is not clear in Abstract or Introduction. In addition, there is no explanation about what is the gap in Li and Shimizu (2018) and Wei et al. (2018).
- The details about the models of Li and Shimizu (2018) and  Wei et al. (2018) can be found in Section 2 (Model definition), but should be moved to Section 1 (Introduction) or an additional section about related studies.
- There is no definition about the probabilistic distribution of $e$ in Definition 1.
- The subscript of function $f$ is abused in Definition 1. $f_c$ and $f_d$ refer to the functions of continuous variables and discrete variables respectively. However, the subscript of $f_i$ corresponds to the index of $x_i$.
- $c$ is also abused: the subscript of $f_c$ means "continuous", and variable $c$ in "$bx+e+c$" means the intercept.
- The last sentence in page 3 says "Now we characterize the condition about the non-Gaussian distributions.", but page 4 says "it’s noteworthy that basic non-Gaussian distributions satisfy the Condition 1". Therefore, it is hard to see if the authors are saying that all the non-Gaussian variables satisfy Condition 1 or not. In addition, it is not clear what "basic non-Gaussian distributions" means.
- The authors "assume that the two models give the same joint distribution of observed variables $x_i$ and $x_j$." in page 5, but there is no explanation about why it does not lose generality.
- The proof (ii) of Theorem 2 aims to prove the identifiability between discrete variables. However, it is not clear how the causal direction is determined by the proof.
- In Equation (11), $e_j$ is limited to follow Logistic(0,1). However, such definition cannot be found in Definition 1.
- There is no explanation about "Notears" in page 10.
- It is hard to evaluate the result shown in Figure 3 because there is no information about the ground truth. In addition, the experiment on the real-world data does not include comparative evaluation with previous methods.

# Originality and significance
As I wrote above, the originality and the significance of this study is not clearly stated in this paper. The contribution of this study seems that the study extended the work of Li and Shimizu (2018) and Wei et al. (2018). However, the significance of it is still unclear.
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The authors propose a method for causal discovery from mixed data under the linearity and causal sufficiency assumptions. The proposed method relies on assuming non-Gaussian error terms for continuous variables and logistic error terms for discrete variables. The authors show that, under certain conditions, the generating causal DAG in their Linear Mixed causal model (LiM) is fully identifiable. Finally, they showcase the performance of their LiM model on a number of simulations and in a real-world experiment on the Boston housing dataset.
Main review: **Update**: After reading the other reviews and discussing with the authors, I have increased my score from 5 to 6.

The authors build upon the work of Shimizu et al. (2006) for handling continuous variables, and on the work of Wei et al. (2018) for handling discrete variables. The extension over Wei's approach is that the LiM model can incorporate more types of non-Gaussian distributions for the continuous variables. However, the method's applicability is severely limited by the causal sufficiency assumption, which will not hold for most realistic settings. I would have liked to have seen this limitiation discussed more explicitly in the paper. 

I think the paper could be significantly improved in terms of clarity. The paper is unfortunately pockmarked with writing errors, which hinder readability. Moreover, there are parts such as Corollary 1 whose purpose in the text is unclear to me. The authors require Condition 1 on the tails of the distribution to prove Theorem 2, but I am not sure why the condition of having a finite non-zero limit needs to be separated in the two cases S1 and S2. Conversely, some parts are not explained in enough detail, as far as I am concerned. For instance, the authors claim, if I understand correctly, that by setting $x_i = x_j = 0$ in equations (9) and (10), which are actually supposed to be a single equation, we conclude that $k_i = k_j$. However, what we get is that $\frac{1-k_i}{1 - e^{-c_i}} = \frac{1-k_j}{1 - e^{-c_j}}$. It seems to me that some non-obvious steps are missing there.

On page 2, what do you mean by "The output causal structure here may fall into the solutions up to the ground truth's skeleton"? Similarly, on page 9, what do you mean by "an optimization method may easily fall into its skeleton solutions" and what is the "sub-optimality" you are referring to?

On page 7, the authors relate their reasoning to the proof of Theorem 28 in Peters et al. (2014). I do not think it is necessary to copy the text of Remark 30 from Peters et al. (2014), especially since it is hard to understand without context. 

On page 10, in the Experiments section, the authors mention that they take the instance of a PDAG returned by PC for evaluation. Which instance do they use exactly? Depending on how the evaluation is performed, one instance might perform better than another. It is also not exactly clear to me how the performance is being evaluated. Are the precision, recall, and F1 score computed in terms of the skeleton, that is, having the corrected edges, or do you also consider the edgemarks? When is a discovery correct and when is it wrong? 
 
Overall, I think that while the LiM model may improve over the state-of-the-art, albeit incrementally, the paper lacks clarity and rigor, and is need of some serious rewriting before it is ready for submission.

Typos, writing errors, and other comments:
- In the introduction, the categorization of causal discovery algorithms into constraint-based and score-based is introduced twice (on page 1, and then again at the beginning of page 2).
- Page 1: "may output Markov equivalence class" -> "output Markov equivalence classes"; "same conditional independence" -> "same conditional independences"; "These methods including" -> "These methods, including"; "on cases when" -> "on cases where"
- Page 2: "which are naive and are easy to lose data information" -> "which is naive and can make it easy to lose data information"; "can not" -> "cannot"; "may return Markov equivalence class" -> "may return Markov equivalence classes"; "accompanied with" -> "accompanied by"; "ratio by" -> "ratio of"; "T class" -> "T categories"; "And further" -> "Further"
- Page 3: "equals to 1" -> "equals 1"; "equals to 0" -> "equals 0"; "the assumptions A1-A3" -> "assumptions A1-A3"
- Page 4: "condition probability" -> "conditional probability"
- Page 5, Equation (17): $b_{ij}$ should be $b_{ji}$ for both branches
- Page 6: "Due to the Condition 1" -> "Due to Condition 1"
- Page 6, Definition 3: The authors use the notation $\perp$ presumably to denote (conditional) independence, but do not explain what it means. 
- Page 7, Definition 4: It would be nice to introduce the sets **pa** and especially **nd** for those who are not so familiar with the notation. What do you mean by "$x_j$'s component with $f_i$" when explaining the underbrace?
- Page 7, Theorem 5: "faithful" -> "faithfulness"
- Page 9: "that the estimated $b_{ij}$ is hard to receive absolute zeros" -> "that it is hard for the estimated $b_{ij}$ to receive absolute zeros";
- Page 10: "Logistic (logistic)" -> "logistic"; "Laplace (laplace)" -> "Laplace"; "Re." -> "recall" & "Pre." -> "precision" (The abbreviations seem completely unnecessary.); "to the sample sizes" -> "to sample sizes"
- Page 12: "lower status of the population" -> "percentage lower status of the population"; "may not an effect variable" -> "may not be an effect variable"; The variable INDUS is later referred to as IND, which is inconsistent.
- Page 14: "form" should be "from"
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary: The main contribution of the paper is an introduction into a novel causal discovery algorithm for mixed (i.e., continuous and categorical) data. In the present approach, continuous variables are assumed to deviate from the Gaussian distribution and categorical variables are assumed to be the result of discretization of a latent variable that follows a logistic distribution. The authors show that the presented approach enables one to uncover causal structures beyond Markov equivalent classes in both, bivariate and multivariate data settings. A two-step hybrid approach is presented that, in the first step, estimates the log-likelihood score function for the joint distribution of mixed data, and, in the second step, identifies the causal graph with the best score value among structures of the skeleton space. The performance of the algorithm is evaluated using artificial data experiments. Here, simulation results suggest that the presented approach outperforms five competing algorithms (2 variants of the NOTEARS algorithm considering logistics and Laplace distributions, the PC algorithm, LiNGAM originally proposed for continuous variables only, and a second variant of the proposed algorithm without the second search phase). The Boston housing dataset is used to illustrate the presented approach in practice. Here, the results are mixed. Some of the identified paths come with high face validity, other paths are more likely to be spurious due to logical inconsistencies. 
Main review: The paper introduces a novel approach to discover causal structures in mixed (continuous/categorical) data. Such data occur across all empirical sciences. Thus, the presented approach can be considered of high relevance for the audience of causal machine learning and beyond. However, there is a small number of aspects (summarized below) that requires additional attention prior to recommending acceptance of the manuscript in the Proceeding of Machine Learning Research. 

First, the paper is well organized. However, in general, the paper might benefit from careful proofreading of a professional editor. Some parts of the paper are a bit hard to follow which may mitigate clarity.

p. 2, 2nd paragraph:  For sake of completeness, I suggest including another mixed data causal discovery algorithm proposed by Yamayoshi et al. (2020). These authors proposed a Latent-LiNGAM (L-LiNGAM) algorithm. Here, it is assumed that observed categorical variables are the result of discretizing latent continuous variables via link functions (in case of the identity link L-LiNGAM reduces to the classic LiNGAM). The paper would benefit from incorporating this previous work. Also, I recommend including this algorithm in the artificial data experiment as a benchmark. 

p. 3, last paragraph: Similar to Wei et al. (2018), the presented algorithm makes use of the fact that causal models are identifiable due to systematic differences in the marginal distributions of binary variables. It may be worth noting that a similar result has already been discussed by Wiedermann and von Eye (2017) in the domain of skewed latent variables. Following the idea of Dodge and Rousson (2000) – who showed that, when the error is Gaussian, the skewness of the true outcome variable will always be smaller than the skewness of the true predictor –these authors showed that, for the bivariate case, asymmetry exists between causally competing non-hierarchical log-linear models (i.e., models without one of the two main effects). Because the marginal distribution of the true outcome will be closer to symmetry than the marginal distribution of the predictor, causal model selection can be performed.  

p. 10, Section 5.1 (Synthetic data): The selection of competing algorithms seems somewhat suboptimal. For example, why was LiNGAM included in the simulation? Since this algorithm has been designed for continuous variables only, it does not serve as a reliable benchmark. Instead, I suggest including previously suggested algorithms for mixed data (e.g., the L-LiNGAM approach mentioned above). 

p. 10: Please provide a citation for the NOTEARS algorithm. 

p. 10 & 11 (Results section): While the accuracy results for the proposed approach are impressive and convincing, I do not fully agree with some of the write-up in the corresponding results section. For example, the authors state that accuracies of the proposed approach and PC “…increase remarkably along with the sample sizes”, this statement only holds for sample sizes up to 1000. Beyond n = 1000, it seems that the accuracy of the proposed approach slightly decreases again. Here, the authors should clarify whether this is an ignorable instability due to the selected number of iterations, or whether this is a phenomenon inherent to the approach. Also, please note that the performance of the PC algorithm seems to be stable for the bivariate case which contradicts the authors’ statement cited above. Finally, when discussing results of varying the number of discrete variables, it may be worth noting that the accuracy of the proposed method tends to decrease, suggesting that the accuracy may be affected by the number of categorical variables. 

p. 12 (Conclusion section): I fully agree that future work needs to be done with respect to nonlinear causal relations. However, given that the authors state that identification results also hold for the ANM, it seems that this work has already been done. Thus, I suggest being a little more specific. Which aspects of causally modeling nonlinear relations requires future attention? Finally, another important future direction may be the inclusion of categorical variables with more than two categories. Currently, the authors make use of the fact that multi-categorical variables can be represented by k – 1 binary variables. However, such an approach might be suboptimal, in particular, in the presence of ordinal variables. It is well-known that ignoring ordinal information can have adverse effects on model estimation (see von Eye and Wiedermann, 2018, for a recent illustration). The paper might benefit from incorporating these perspectives. 

References 

Dodge, Y., & Rousson, V. (2000). Direction dependence in a regression line. Communications in Statistics-Theory and Methods, 29(9–10), 1957–1972. https://doi.org/10.1080/03610920008832589

von Eye, A., & Wiedermann, W. (2018). Strengthening arguments based on scale levels? Journal for Person-Oriented Research, 4(1), 45-47.

Wiedermann, W., & von Eye, A. (2017). Log-linear models to evaluate direction of effect in binary variables. Statistical Papers. (in press).
review_title: null

Meta-review: The paper tackles the problem of causal discovery from mixed data, i.e. when continuous and discrete variables may be present, for both bivariate as well as multivariate settings.
It introduces the so-called Linear Mixed (LiM) model and proves full identifiability under certain assumptions, including causal sufficiency, linear continuous functions with additive non-Gaussian noise, and a linear logistic model for the discrete variables. It also describes the associated LiM algorithm that can find this model, consisting of a global search optimisation phase (based on the so-called quadratic penalty method), followed by a local likelihood optimisation phase. The experimental evaluation shows that, when the assumptions hold, the method compares favourably to other approaches that are not designed to handle this setting.

Reviewers were initially fairly critical of the paper, in particular on clarity of the presentation and several errors / omissions in the text. However the authors made an excellent attempt at answering the points raised, clarifying some and promising to address / resolve others in the final version, so that ultimately all reviewers agreed on recommending acceptance, although insisting presentation could/should still be imporved.

I have to admit on reading the paper I am actually slightly more critical than the reviewers. I fully agree presentation should be improved, but I also notice a tendency to emphasise weaknesses of other methods while focussing on strong points of the proposed method, rather than trying to give a fair and balanced assessment of pros and cons. It is technically ok, but rather niche, relying on strong assumptions where the only ‘real-world’ application (Boston) leads to a questionable output model. Essentially the algorithm is likely to obtain the touted ‘full identifiability’ by overfitting on the data, even though for larger sample sizes the performance actually seems to decrease in some cases. That said, it does at least try to expand existing causal discovery methods to the important mixed data realm, and the proposed approach is new and sound in principle, so despite the caveat above I will follow the reviewers and recommend accept.

Decision: Accept (Poster)