Question: Could  the issue of stabilization be elaborated in the Discussion?
Evidence: In this section, we discuss the properties of the proposed method, specifically the computational complexity and the stabilization.
In the proposed method, we use neural networks to represent models and estimate them by optimizing their parameters using stochastic gradient descent methods. Since each neural network represents a simple scalar function, it is unnecessary to be a complex design. Therefore, the train-ing time is shorter than the general deep models such as image recognition and language processing.
On the other hand, the number of estimations grows exponentially as the number of variables increases. Given n variables, the identification of causal order requires n − 1 iterations. In each iteration, we compare k models for the remaining k variables. Therefore, a total of n + (n − 1) + • • • + 2 = n(n − 1)/2 − 1 model estimations are required, which is the same as RESIT and DirectLiNGAM. Additionally, the proposed method trains one model several times to stabilize results. Consequently, the total number of training becomes t(n(n−1)/2−1), where t is the number of training trials per one model. Similar to the identification of causal order, the pruning procedure also requires an exponential number of estimations. For the worst case, where all the edges will be pruned, a total of t(k(k − 1)/2 − 1) estimations are required for each multi-cause substructure with k cause variables 4 . The number of estimations increases as the ground truth DAG becomes sparse.
One effective approach to alleviate the computational time is parallelization. Since all the estimations in one iteration of the identification of causal order can be performed separately, we can reduce the time by parallelizing them. If enough computational resources are available, we can identify the causal order with n − 1 serial iterations. Similarly, the time of the pruning can be reduced by the same manner. In the experiment in Section 4, we used this parallelization. We additionally applied the proposed method to synthetic problems with n = 15 and confirmed that it requires roughly one day for one problem on a server capable of (maximum) 40-threading 5 .
Estimating one model multiple times in the proposed method has a high impact on the total computational time. Therefore, we think improving the stability is an important issue that should be addressed not only for improving the performance of causal discovery but for reducing the computational time. One of the typical approaches is to change the optimizing algorithm, while there is a trade-off between stability and speed. Another possible option is to add a regularization term to the loss. To use this idea, we need to consider carefully the impact of changing the loss on using its value when comparing models in the sink identification.
In this section, we discuss the properties of the proposed method, specifically the computational complexity and the stabilization.
In the proposed method, we use neural networks to represent models and estimate them by optimizing their parameters using stochastic gradient descent methods. Since each neural network
represents a simple scalar function, it is unnecessary to be a complex design. Therefore, the training time is much shorter than the general deep models such as image recognition and language
processing.
On the other hand, the number of estimations grows exponentially as the number of variables increases. Given n variables, the identification of causal order requires n − 1 iterations. In each
iteration, we compare k models for the remaining k variables. Therefore, a total of n + (n − 1) + · · · + 2 = n(n − 1)/2 − 1 model estimations are required, which is the same as RESIT and DirectLiNGAM. Additionally, the proposed method trains one model several times to stabilize results. Consequently, the total number of training becomes t(n(n−1)/2−1), where t is the number of training trials per one model. Similar to the identification of causal order, the pruning procedure also requires an exponential number of estimations. For the worst case, where all the edges will be pruned, a total of t(k(k − 1)/2 − 1) estimations are required for each multi-cause substructure with k cause variables4 . The number of estimations increases as the ground truth DAG becomes sparse.
One effective approach to alleviate the computational time is parallelization. Since all the estimations in one iteration of the identification of causal order can be performed separately, we can reduce the time by parallelizing them. If enough computational resources are available, we can identify the causal order with n − 1 serial iterations. Similarly, the time of the pruning can be reduced by the same manner. In the experiment in Section 4, we used this parallelization. We additionally applied the proposed method to synthetic problems with n = 15 and confirmed that it requires roughly one day for one problem on a server capable of (maximum) 40-threading5 .
Estimating one model multiple times in the proposed method has a high impact on the total computational time. Therefore, we think improving the stability is an important issue that should be addressed not only for improving the performance of causal discovery but also for reducing the computational time. One of the typical approaches is to change the optimizing algorithm, while there is a trade-off between stability and speed. Another possible option is to add a regularization term to the loss. To use this idea, we need to consider carefully the impact of changing the loss on using its value when comparing models in the sink identification
Answer: We will set up the Discussion section and add more detailed discussions.
As stated in the paper, we estimate one PNL model by training the corresponding neural networks several times to stabilize results. Therefore, the stabilization of the training is important for not only improving the performance but also reducing the computational cost. One of the typical approaches is to change the optimizer, while there is a trade-off between stability and speed. Another possible option is to add a regularization term to the loss. To use this idea, we need to consider carefully the impact of changing the loss on using its value when comparing models in the sink identification.
My main concern is about the numerical comparison. In fact only one single DAG with a few SCMs is considered and the proposed approach compared to the RESIT estimator of ANM and DirectLinGAM. As mentioned above, there seems to be another relevant neural network estimator that could be considered here.
In the experiment in section 4.1, we randomly generated 100 linear and nonlinear problems, respectively, as stated in the caption of Table 1. In this procedure, not only causal equations but also DAG structures are generated randomly. Two examples in Fig. 2 are (coincidentally) have the same DAG structure.
We apologize for insufficient and misleading expressions. To make it clear, we will add the descriptions in the main text and change the examples in Fig. 2.