Question: Did the authors not compare against any competing method?
Evidence: We compare the scalability and accuracy of PreDITEr to those of two competing methods under various settings: the ψ-FCI algorithm of and the FCI-JCI123 algorithm of. We note that both of these algorithms solve a more general problem than the linear SEMs we are considering. To the best of our knowledge, there is no algorithm specifically designed for linear SEMs, and these are the only two methods that can be applied to our setting. Therefore, we compare our results to those of these two methods. do not provide simulations for graphs that have more than a few nodes since ψ-FCI requires an exponentially growing number of conditional independence and invariance tests. report experiments with larger graphs, and we compare our algorithm to their FCI-JCI123 algorithm. We focus on scalability and provide additional experiments on small graphs and MEC refinement results in supplementary material section D.2.
To enable comparisons under soft interventions, we adopt mechanism changes of Mooij et al., in which a constant offset is added to the intervention targets (see page 53, Section 5.2 for details). We note that this is different from our model of soft interventions and results in slight degrading of the performance of our algorithm, but since we are using FCI-JCI123 as our benchmark, we adopt its setting.
We consider two environments and one intervention target for simplicity of the comparisons. We generate 30 Erdős-Rényi random DAGs. The probability of an edge being present in the random graphs is set to 2/p where p is the number of observed and latent variables. We report the precision and recall rates of both algorithms along with their runtimes in Table. While both methods have similar performance, there is a significant discrepancy in their runtime. More importantly, the runtime of FCI-JCI123 becomes prohibitive very quickly, even with graphs with as few as 40 nodes. In contrast, PreDITEr has a significantly lower runtime even though the considered setting (i.e., mechanism changes) is not the setting for which it is designed. All simulations are run on a computer with i7-4960HQ, 16GB 1600MHz RAM. We apply the PreDITEr algorithm to a real dataset with data from observational and multiple interventional settings. Since PreDITEr estimates the intervention targets and their corresponding parent-spouse sets for each pair of available settings, we combine the findings from each pair and yield a mixed graph estimate of the associated causal structure. Protein signaling data. We consider the dataset of, which is a standard benchmark in causal inference literature. The data is obtained from measurements of the proteins involved in T-4 cell signaling. The protein signaling network consists of 11 nodes. In each interventional setting, various drugs are injected into the cells to inhibit or activate different signaling proteins. The target proteins are considered sites of intervention. Data from observational and five interventional settings are provided. The true ground truth network is not exactly known, and the accepted ground truth has been updated over the years. Notably, it is represented by a DAG without latent confounders. We use the recent version of Ness et al., which consists of 16 edges, and use the preprocessed real data provided by.
Figure shows the output of our algorithm. For a pair of nodes, if they are found to be in the parent-spouse sets of one another, they must be spouses, and we assign a bidirected edge. If only one of them lies in the parent-spouse set of the other node, it must be the parent, and we assign a directed edge to them. If we do not have either of the above results, the relationship can be either parent or spouse, and we denote it by → • on the graph. The recovered edges that are also present in the skeleton of the ground truth DAG are marked in blue. This result illustrates that even though our algorithm is designed for linear models, it has the potential to be applied to real datasets with non-linear models.
We run PreDITEr with a varying number of samples on graphs with varying sizes p. Figure 2 illustrates the target recovery performance. Specifically, it shows that our method recovers the intervention target with high F1 scores. We emphasize that PreDITEr can easily process large graphs (e.g., p = 100 nodes), and have less than 1 second average runtime for the simulations shown in Figure 2.****We compare the scalability and accuracy of PreDITEr to those of two competing methods under various settings: the ψ-FCI algorithm of Jaber et al. [2020] and the FCI-JCI123 algorithm of Mooij et al. [2020]. We note that both of these algorithms solve a more general problem than the linear SEMs we are considering. To the best of our knowledge, there is no algorithm specifically designed for linear SEMs, and these are the only two methods that can be applied to our setting. Therefore, we compare our results to those of these two methods. Jaber et al. [2020] do not provide simulations for graphs that have more than a few nodes since ψ-FCI requires an exponentially growing number of conditional independence and invariance tests. Mooij et al. [2020] report experiments with larger graphs, and we compare our algorithm to their FCI-JCI123 algorithm. We focus on scalability and provide additional experiments on small graphs and MEC refinement results in supplementary material section D.2. To enable comparisons under soft interventions, we adopt mechanism changes of Mooij et al. [2020], in which a constant offset is added to the intervention targets (see page 53, Section 5.2 for details). We note that this is different from our model of soft interventions and results in slight degrading of the performance of our algorithm, but since we are using FCI-JCI123 as our benchmark, we adopt its setting.
Answer: Before reporting the results, we note that none of the algorithms are thoroughly optimized for the best results. For instance, changing the parameters of the CI tests for Jaber et al. (2020) and Mooij et al. (2020), or changing the regularization of PDE’s for our method directly affect the final results. The reported results on all 3 papers are to provide conceptual understanding. Furthermore, due to the true network being unknown, it is difficult to report the edge orientation results. We refer to Figure 39 of Mooij et al. (2020) and Figure 8 of Jaber et al. (2020) to report the skeleton-wise results with respect to the consensus network of Ness et al. (2017).In summary, we have demonstrated that our method works at least as effectively as the competing algorithms in the smaller graphs. Our method’s strong performance in the larger graphs where the other algorithms cannot accommodate was already displayed in Section 6.1. We believe the above discussions should clarify any experimental concerns.