Title: Analysis of Variance of Multiple Causal Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: causal inference

large graphs

multi-task learning

structural model

directed cyclic graph
Abstract: Constructing a directed cyclic graph (DCG) is challenged by both algorithmic difficulty and computational burden. Comparing multiple DCGs is even more difficult, compounded by the need to identify dynamic causalities across graphs. We propose to unify multiple DCGs with a single structural model and develop a limited-information-based method to simultaneously construct multiple networks and infer their disparities, which can be visualized by appropriate correspondence analysis. The algorithm provides DCGs with robust non-asymptotic theoretical properties. It is designed with two sequential stages, each of which involves parallel computation tasks that are scalable to the network complexity. Taking advantage of high-performance clusters, our method makes it possible to evaluate the statistical significance of DCGs using the bootstrap method. We demonstrated the effectiveness of our method by applying it to synthetic and real datasets.

Analysis of Variance of Multiple Causal Networks

Abstract

Constructing a directed cyclic graph (DCG) is challenged by both algorithmic difficulty and computational burden. Comparing multiple DCGs is even more difficult, compounded by the need to identify dynamic causalities across graphs. We propose to unify multiple DCGs with a single structural model and develop a limited-information-based method to simultaneously construct multiple networks and infer their disparities, which can be visualized by appropriate correspondence analysis. The algorithm provides DCGs with robust non-asymptotic theoretical properties. It is designed with two sequential stages, each of which involves parallel computation tasks that are scalable to the network complexity. Taking advantage of high-performance clusters, our method makes it possible to evaluate the statistical significance of DCGs using the bootstrap method. We demonstrated the effectiveness of our method by applying it to synthetic and real datasets.

1 Introduction

Study of causal networks described by directed cyclic graphs (DCGs) becomes increasingly popular in a variety of fields, such as biomedical and social sciences [10, 3]. Detecting dynamic structures across networks sheds light on mechanistic dynamics, with promising applications. For example, gene regulatory networks related to various types of cancer differ from that of healthy cells in a complicated way and discovery of deviated regulations helps identify cancer-related biological pathways and hence design target-specific drugs [23, 18]. Interpersonal networks in social media have become a favored way for people to get informed of breaking news and natural disasters nowadays. Understanding their evolution over time may provide promising information to businesses, educators, and governments [7]. However, the complex structure and large number of nodes impose challenges on constructing even a single network, not to mention modelling and comparing multiple networks.

Structural models are widely used to describe causal networks and conduct causal inference [8, 15]. With available instrumental variables (IVs) from genomic variations, many methods have been developed to construct large causal networks of gene regulation by integrating transcriptomic and genotypic data. For instance, Liu et al. [14] proposed to assemble driver-responder relationships identified from local structural models, a formidable task due to enormous numbers of possible driver-responder pairs. Cai et al. [4] developed a sparsity-aware maximum likelihood (SML) method by putting glyph[lscript] 1 penalty on causal effects. It may reach local maximum, leading to the identifiability issue. Chen et al. [5] proposed a two-stage penalized least square (2SPLS) approach which allows parallel computing and shows superior performance.

To infer sparse differences between two large networks, Ren and Zhang [16] recently developed a parallel algorithm based on 2SPLS and achieved better performance than separately constructing the

two networks using 2SPLS. Zhou and Cai [25] proposed a fused sparse SEM (FSSEM) method to learn the differences through maximum likelihood inference of joint networks which is computationally infeasible with large networks. Li et al. [13] employed a Bayesian scheme in their method BFDSEM, unrealistically assuming equal sample sizes from both cohorts.

To the best of our knowledge, there is no algorithm available for investigating causal effects varying across multiple cohorts while identifying stable ones. In this paper, we develop a novel algorithm, designed for parallel computation, to construct a unified structural model for multiple causal networks. While each causal network may be depicted by a DCG, we deliver an analysis of variance (ANOVA) of these networks to identify causalities that are different across networks as well as important drivers and responders. Our algorithm is scalable in several aspects. Firstly, it is scalable to data size including sample size and number of variables. It would still be efficient and powerful under high dimension scheme. Secondly, it is scalable to the computational environment with its parallel computation. The fast computation allows to attack a daunting task in studying multiple networks, i.e., calculating p -values via the bootstrap method and thus controlling the false discovery rate. Thirdly, the algorithm is scalable to model complexity as it is able to infer causal networks beyond directed acyclic graphs (DAGs), such as DCGs which are demanded to depict gene regulatory networks [14].

The rest of the paper is organized as follows. We first state the model and discuss its identifiability guaranteed by available instrumental variables in Section 2. In Section 3, we present our proposed algorithm AN alysis O f VA riance of directed Net works, termed as NetANOVA , and introduce measures to quantify an endogenous variable's contribution as either drivers or responders. The theoretical justification is shown in Section 4 with detailed proofs in Supplemental Material. We demonstrate the feasibility and promise of our algorithm with a large-scale simulation study shown in Section 5 and a real data analysis to compare gene regulatory networks in healthy lung tissues and lung tissues with two types of cancer in Section 6. We conclude the paper with a discussion in Section 7.

2 A Unified Structural Model of Multiple Causal Networks

2.1 Model specification

We consider K pertinent causal networks, each for a cohort. For each, say k -th, cohort we have n ( k ) observations in ( Y ( k ) , X ( k ) ) , where Y ( k ) is an n ( k ) × p matrix including values of p endogenous variables, and X ( k ) is an n ( k ) × q matrix including values of q exogenous variables. Each variable is assumed to have mean zero. For the k -th causal network, we consider each, say i -th, endogenous variable is causally affected by other variables as follows,

where Y ( k ) i is the i -th column of Y ( k ) , and Y ( k ) -i is the submatrix of Y ( k ) excluding the i -th column. The ( p -1) -dimensional column vector γ ( k ) i includes the direct causal effects of other endogenous variables (drivers) on the i -th one (responder). The set I i includes the indices of exogenous variables that serve as IVs of the i -th endogenous variable so X ( k ) I i is a n ( k ) ×|I i | matrix including values of corresponding IVs. Each IV has a direct causal effect on its corresponding endogenous variable and can be identified through domain knowledge. φ ( k ) I i includes causal effects from corresponding IVs. glyph[epsilon1] ( k ) i includes disturbance errors which are independently distributed with mean zero and standard deviation σ ( k ) i .

Pooling together equation (1) for each endogenous variable, we model the k -th causal network as,

which describes a DCG as shown in Figure 1.

The p × p matrix Γ ( k ) describes the causal relationships between each pair of endogenous variables and has zero diagonal elements to prohibit self-regulation. The q × p matrix Φ ( k ) encodes the causal effects of exogenous variables on endogenous variables. The n ( k ) × p matrix glyph[epsilon1] ( k ) includes all disturbance errors. We assume that both Y ( k ) and X ( k ) have been appropriately centralized within the cohort, so no intercepts are needed in the above model.

(a) DCG I

Γ¡ Γ¸

Γ¡ Γˇ

Γ¡ Γ˚

Γ¡ Γ˘

Γ¡ Γ¯

Γ; Γˇ

Γ; Γ˘

Γ; Γ˚

Γ8

Γ2

Γ; Γ¯

(b) DCG II

Γ¡ Γ¸

Γ¡ Γˇ

Γ¡ Γ˚

Γ¡ Γ˘

Γ¡ Γ¯

Γ; Γˇ

Γ; Γ˘

Γ; Γ˚

Γ8

Γ2

Γ; Γ¯

(c) DCG III

Γ¡ Γ¸

Γ¡ Γˇ

Γ¡ Γ˚

Γ¡ Γ˘

Γ¡ Γ¯

Γ; Γˇ

Γ; Γ˘

Γ; Γ˚

Γ8

Γ2

Γ; Γ¯

2.2 A unified structural model

Many algorithms have been developed to construct a structural model for single causal network [14, 4, 5]. We will show that, in the interest of comparing multiple causal networks, we can also unify them into a single structural model and hence develop an appropriate algorithm to construct all networks at the same time to deliver an ANOVA of multiple networks.

Suppose we have a baseline, say K -th, network and are interested in others' deviation from the baseline. We first reparameterize the causal effects of all endogenous variables in (1) with

glyph[negationslash]

Therefore β ( K ) i includes baseline effects and, for k = K , β ( k ) i includes deviated effects of k -th network from the baseline. When it is of interest to compare the networks with each other, we can similarly reparameterize to consider the deviated effects of each network from the average effects.

For any K sets of l × m matrices, say A i with i = 1 , 2 , · · · , K , we define a matrix-valued function,

where A 1: K is constructed by stocking A 1 , · · · , A K row-wisely, and 0 m ( K -1) is a m ( K -1) -dimensional column vector with all elements zero. Further denote

Notice that γ ( k ) i = β ( K ) i + β ( k ) i for k = 1 , 2 , . . . , K -1 from (3), we have a unified structural model for K causal networks described by (2),

2.3 Instrumental Variables, model identifiability, and confounding variables

As shown in Figure 1, the ability to reveal causal effects relies on the available IVs. An IV is an exogenous variable that only affects directly the driver but not the responder, except indirectly through its effect on the driver. This means that IVs can be used to address the model identifiability issue, which is complicated by the model's inherent endogeneity, and allow us to isolate the causal effects of endogenous variables. We specify the following assumption on available IVs following the rank condition [20], a necessary and sufficient condition for model identification.

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Assumption 1. a . X ( k ) ⊥ ⊥ glyph[epsilon1] ( k ) ; b . ∃C ⊂ { 1 , 2 , · · · , p } but C glyph[negationslash] = ∅ ; c . ∀ i ∈ C , ∃I i ⊂ { 1 , 2 , · · · , q } with I i = ∅ and φ ( k ) I i = 0 ; d . ∀ i, j ∈ C with i = j , I i ∩ I j = ∅ ; e . ∀ j / ∈ C , γ ( k ) ij = 0 .

Assumption 1.a specifies IVs observed in X ( k ) that are uncorrelated to the disturbance errors. Assumption 1.b specifies the endogenous variables in C that have available IVs and Assumptions 1.c and 1.d state the restrictions on the available IVs, i.e., their availability and uniqueness, respectively. While γ ( k ) ij is the j -th component of γ ( k ) i , Assumption 1.e states that we are not able to identify any causal relation with j -th endogenous variable as a driver if it has no IV available, e.g., Y 5 in Figure 1.

A striking advantage of the IV method is its robustness to confounding but exogenous variables, which may be observable or unobservable, as shown in Figure 1. As will be demonstrated by our later algorithm, available IVs help disassociate an endogenous variable from disturbance errors of other endogenous variables in model fitting. Therefore, we may explicitly incorporate observable confounding variables to improve causal inference. However, when there are unobservable confounding variables, we can still effectively construct multiple networks with available IVs.

3 Model Building and Interpretation

The single structural model in (4) makes it possible to employ a limited-information approach to construct multiple causal networks even in the case of a large number of endogenous variables. Therefore we develop a two-stage algorithm NetANOVA which allows parallel computing for fast computation and hence bootstrapping the data for significance assessment. Another daunting but necessary task is to understand the multiple networks and catch the important variables in the networks and important dynamic causalities across them. We therefore propose coefficents of determination and cause as well as correspondence analysis.

3.1 The algorithm NetANOVA

The disassociation stage: We first predict each endogenous variable in C , solely based on all available exogenous variables, to disassociate it from the disturbance errors of other endogenous variables, following aforementioned assumption 1. For this purpose, we rearrange the terms in (2) to obtain the reduced model,

As shown in the real data analysis in constructing gene regulatory networks, the number of exogenous variables, i.e., the dimension q for X ( k ) , can be much larger than the sample size n ( k ) , rising issues of prediction consistency and computational time. To address them, we first apply the iterative sure independence screening (ISIS; [9]) to screen for exogenous variables and then perform regression with glyph[lscript] 2 penalty. As later theoretical analysis shows, it allows for q glyph[lessorsimilar] exp ( n ( k ) θ ) for some θ > 0 .

Specifically for each endogenous variable i ∈ C and k ∈ { 1 , . . . , K } , we apply ISIS with computational complexity O ( n ( k ) p ) to screen for a set of exogenous variables, indexed by set M ( k ) i . Using these d = |M ( k ) i | exogenous variables, we apply ridge regression to obtain the ridge estimator,

where λ ( k ) i is a tuning parameter that can be selected via generalized cross validation [11].

The inference stage: We use the predicted endogenous variables in the previous stage to identify and estimate the causal effects, hence constructing the causal networks. We first calculate the projection matrix,

Multiplying the projection matrix P i = diag { P (1) i , P (2) i , . . . , P ( K ) i } to both sides of model (4), we can eliminate the exogenous variables from the model and get,

and predict Y ( k ) i with

where Υ -i = T ( Y (1) -i , Y (2) -i , · · · , Y ( K ) -i ) . Since P i Υ -i and glyph[epsilon1] i are correlated, we instead regress P i Y i against P i ˆ Υ -i with ˆ Υ -i = T ( ˆ Y (1) -i , ˆ Y (2) -i , · · · , ˆ Y ( K ) -i ) which is disassociated from glyph[epsilon1] i . With possibly high-dimensional β i , we apply adaptive lasso [26] to obtain its estimator,

where ν i is a tuning parameter, and ˆ ω i = | ˆ β 0 i | -γ for some γ > 0 where | ˆ β 0 i | 1 contains the absolute values of ˆ β 0 i , which is a preliminary estimator. The original networks can be recovered by calculating ˆ γ ( k ) i = ˆ β ( K ) i + ˆ β ( k ) i for k = 1 , 2 , . . . , K -1 .

In summary of these two stages, we summarize the algorithm in Algorithm 1.

Algorithm 1 AN alysis O f VA riance of directed Net works (NetANOVA)

Input: ( Y ( k ) , X ( k ) ) , k ∈ { 1 , 2 , ..., K } , with each variable centralized within cohort but scaled according to the baseline cohort; Predefined index set I i for each i ∈ C ; d ← O ( n 1 -θ min ) .

STAGE 1

for i ∈ C do

Reduce the dimension of X ( k ) by ISIS to get X ( k ) M ( k ) i ; Set X ( k ) M ( k ) i = X ( k ) if q ≤ n ( k ) .

Estimate ˆ Y ( k ) i by regressing Y ( k ) i against X ( k ) M ( k ) i with ridge regression.

end for

STAGE 2

for i = 1 , 2 , . . . , p do

Calculate projection matrices P i .

Predict ˆ Υ -i from ˆ Y ( k ) i , i ∈ C .

Estimate ˆ β i by regressing P i Y i against P i ˆ Υ -i with adaptive lasso.

end for

Output: ˆ β 1 , . . . , ˆ β p which contains the baseline and differential causal effects.

3.2 Coefficient of determination and coefficient of cause

For each endogenous variable i , we can calculate its coefficient determination to measure the proportion of its variation due to the effects of its drivers, i.e., Y ( k ) -i γ ( k ) i in (1), for cohort k ,

On the other hand, we can also calculate the coefficient of cause for each endogenous variable i which summarizes proportions that it contributes to the variation of its responders, for cohort k ,

While we name the above coefficient of determination as causal R 2 with its value between zero and one, we will simply call the coefficient of cause C 2 which is positive but may be greater than one.

3.3 Correspondence analysis of causal effects

We can conduct a correspondence analysis of causal effects in Γ ( k ) to reveal clusters of drivers, responders, or driver-responder pairs which outstand from the rest in each cohort k . We can also conduct a correspondence analysis of deviated causal effects to reveal these clusters which deviate the most from a baseline or the rest. However, each causal effect or its deviation may vary differently and we need to standardize them, based on the bootstrap results, before correspondence analysis. For example, to compare cohorts k and l for their causal effects, we should instead obtain the following

summary statistic,

where ˆ γ ( k ) ij and ˆ γ ( k,b ) ij are estimates of corresponding effects from the observed data and b -th set of bootstrap data, respectively; we also let z ( k ) ij = 0 when both of its numerator and denominator are sufficiently small. We can define similar statistics to investigate effects' variation from their means.

A singular value decomposition (SVD) of Z ( k,l ) = ( z ( k,l ) ij ) p × p can obtain its left and right singular vectors, say { U ( k,l ) i } and { V ( k,l ) i } , respectively. While the left singular vectors help identify responders that differ between the two cohorts, the right singular vectors help identify drivers that differ between the two cohorts. We can overlay the plot of U ( k ) 2 vs. U ( k ) 1 with the plot of V ( k ) 2 vs. V ( k ) 1 to identify responder-driver pairs for their causalities varying the most across networks.

4 Theoretical Analysis

In this section, we will establish non-asymptotic guarantees and show that our constructed DCGs have good theoretical properties. We characterize the properties with a prespecified sequence δ ( k ) = exp {-o ( n ( k ) ) } with δ ( k ) → 0 as n ( k ) → ∞ , i.e., each δ ( k ) approaches zero slower than exp {-n ( k ) } . We denote δ min = min 1 ≤ k ≤ K δ ( k ) and only present main results here, with details including general notations, assumptions, and proofs in Supplementary Material.

Theorem 4.1. Let n = ∑ K k =1 n ( k ) , S i = supp ( β i ) , and g n is a function of n and δ min specified in (9) in Supplemental Material. Then under Assumption 1 and Assumptions B.1-B.4 in Supplementary Material, we have that, with probability at least 1 -δ min -δ with δ = p ∑ K k =1 δ ( k ) ,

(Causality Consistency) sign ( ˆ β i ) = sign ( β i ) .

We will next show that, with proper choice of { δ ( k ) } , we can bound the estimation errors system-wise with ultra high probability. We can pick { δ ( k ) } such that δ min glyph[equivasymptotic] e -n t where t ∈ (0 , min ( θ, 1 -θ )) . Then we have f n glyph[lessorsimilar] n (2 -θ ) / 2 , and the restriction on the number of true signals can be reduced to |S i | glyph[lessorsimilar] n θ/ 2 , which is a requirement that can be fulfilled especially in a sparse model. We have d = n 1 -θ , so g n glyph[lessorsimilar] n (1 -2 θ ) / 2 . As a result, the bound can be dominated by the order of | S i | n (1 -4 θ ) / 2 .

Note that the error for the whole system of p nodes can be controlled by a similar bound, replacing each occurrence by | S i | with max 1 ≤ i ≤ p | S i | , with probability at least 1 -p ( δ min + δ ) . Following the

(4 θ -1) / 2

former calculation, we learn that, when max 1 ≤ i ≤ p | S i | grow slower than n , the error will approach zero when n →∞ . To achieve the bound with high probability, we only need pδ to diminish. That is, we could have the error well controlled even if the dimension grows up to e n s where s ∈ (0 , t 2 ) .

Given the auspicious estimation performance, we will further discuss the benefit of pooling cohorts compared to that of learning in a single-cohort analysis, where the estimators β ( k ) i for each cohort are estimated separately using the algorithm without re-parameterization. For each k , we can separately conduct the analysis on each cohort and obtain the following property for single task estimation.

Corollary 4.1.1. Under the same conditions as Theorem 4.1, we have that, with probability at least 1 -δ min -δ/K ,

The proof directly follows by treating each cohort as a baseline cohort in Theorem 4.1. Note that the denominator is making a crucial difference in inferring the bound. The algorithm indicates that when we aggregate the samples that does not differ too much, the estimator tends to converge faster,

especially when n glyph[greatermuch] n ( k ) , where we have a plethora of samples in addition to the samples for the base cohort.

With the same choice of δ min and δ as the above theorem, the probability for consistent variable selection can approach one, with a less stringent requirement in the size of S i . The above theorem implies that our proposed method can identify both baseline and differential regulatory effects, among all of the K networks with a sufficiently large probability, not only in terms of the set of true signals but also the sign of signals. In the case of gene regulatory networks, for instance, our method could correctly distinguish the up and down regulations between genes.

Next, we will present a theorem for the coefficient of determination and coefficient of cause defined in Section 3.2. Denote the statistics R 2 and C 2 calculated with real parameters as, respectively,

Then we can derive the following properties.

Theorem 4.2. (Coefficients of Determination/Cause Consistency) Under the same conditions as in Theorem 4.1, with h n = √ n ( k ) + √ -log( δ min ) + 1 , we have that,

With probability at least 1 -p (2 δ min + δ/K ) ,

2. With probability at least 1 -p ( δ min + δ/K ) ,

As considered in Theorem 4.1, we set δ min glyph[equivasymptotic] δ/ ( pK ) glyph[equivasymptotic] e -n t . Then h n glyph[lessorsimilar] √ n ( k ) , and ∑ p i =1 | R 2( k ) i -R 2( k ) 0 i | glyph[lessorsimilar] max 1 ≤ i ≤ p | S i | n (1 -4 θ ) / 2 + ( max 1 ≤ i ≤ p | S i | ) 1 / 2 n -2 θ -1 2 which would vanish if

to 0 when max 1 ≤ i ≤ p | S i | glyph[lessorsimilar] n (4 θ -1) / 4 . From the above discussion, both R 2 , C 2 for all the nodes in the whole system is glyph[lscript] 1 consistent. It is worth noting that, although our algorithm fits a model for each responder, estimate of each node's explanatory power as a driver is also well controlled over the system, but with a slightly larger bound as shown in the above theorem.

5 Simulation Study

We examine the performance of NetANOVA by simulating data of sample size 200, 500 and 1000 from each of three pertinent DCGs, consisting of 1000 endogenous variables. However, only 50 endogenous variables are involved with causality, with each regulated by 3 others on average. Each of the three networks has 5 unique causal effects. First two networks share five causal effects but with opposite signs to that of the third one, resulting in the first two networks having 15 different effects from the third one and the first network have 20 different effects from the second one. The size for causal effects is taken from a uniform distribution over [ -0 . 8 , -0 . 3] ⋃ [0 . 3 , 0 . 8] . The IVs are generated from a multinomial distribution with 3 outcomes 0, 1, 2, with probabilities 0.25, 0.5, 0.25 respectively. The disturbance errors are sampled independently from N (0 , 0 . 1 2 ) .

For each sample size, we simulated 100 data sets and applied NetANOVA to each by bootstrapping 100 times to calculate p values and construct Receiver Operating Characteristic (ROC) curves as shown in Figure 2. For causal effects of all three DCGs, Figure 2.a shows an almost perfect area under curve (AUC) for each sample size. When comparing causal effects of the first two DCGs vs. the baseline one, i.e., the third DCG, the AUC bottoms at 0.867 with sample size at 200, but reaches one with sample size at 1000, showing excellent performance of NetANOVA.

(a) Three DCGs

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

1-Specificity

Sensitivity

200

500

1000

(b) DCG 1 vs. DCG 3

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

1-Specificity

Sensitivity

200

500

1000

(c) DCG 2 vs. DCG 3

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

1-Specificity

Sensitivity

200

500

1000

(d) DCG 1 vs. DCG 2

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

1-Specificity

Sensitivity

200

500

1000

Taking one simulated dataset with sample size at 500, we evaluated our proposed statistics R 2 and C 2 by plotting estimated statistics against the statistics calculated with true causal effects in Figure 3. For each of the three networks, we observe linear trends with slope almost one for both R 2 and C 2 statistics, indicating their excellent performance.

(a)

Y 1

=

0

+

1.01

GLYPH<215>

X 1,

R

2

=

1

Y 2

=

0

+

1.01

GLYPH<215>

X 2,

R

2

=

1

Y 3

=

0

+

1.01

GLYPH<215>

X 3,

R

2

=

1

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

R 0

2

R

2

net1

net2

net3

(b)

Y 1

=

0

+

1

GLYPH<215>

X 1,

R

2

=

1

Y 2

=

0

+

1

GLYPH<215>

X 2,

R

2

=

1

Y 3

=

0

+

1

GLYPH<215>

X 3,

R

2

=

1

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

C 0

2

C

2

net1

net2

net3

We also took one simulated data with sample size 500 and conducted our proposed correspondence analysis to identify drivers and responders that show important dynamic causality across the three DCGs. When comparing DCG I vs. DCG III, Figure 4.b shows a driver-responder pair (14, 45) with different causal effects, which is verified by observing 14 significantly up regulates 45 in DCG I vs. DCG III in Figure 4.a (blue connections). However, we also see the driver 18 and responder 45 stay opposite on the y -axis, because 18 significantly down regulates 45 implying in DCG I vs. DCG III in Figure 4.a. On x -axis, we see pairs (12,13) and (12,7) which correspond to another cluster of causal networks as shown in the right of Figure 4.a.

6 Real Data Analysis

We applied NetANOVA to investigate the gene regulatory networks of lung tissues of healthy individuals and patients with lung adenocarcinoma (LUAD) or lung squamous cell carcinoma (LUSC). We obtained transcriptomic and genotypic data of healthy lungs ( n = 482 ) from the Genotype-Tissue Expression (GTEx) project [6] and of both LUAD ( n = 485 ) and LUSC ( n = 406 ) from The Cancer

(a) Part of the True Networks

0

\,

,

0.36

- 0 . 5 5

0.24

- 0 . 3 6

0.55

0.43

0.24

0.43

- 0 . 1 3

- 0 . 1 3

- 0 . 8 7

- 0 . 8 7

0.68

0.68

- 0 . 5 6

0.56

- 0 . 3 8

- 0 . 3 8

0.15

0.15

14

4

2

32

34

21

11

18

27

45

25

7

13

45

12

14

18

-4

0

4

-4

0

4

Dimension 1 (21.7%)

Dimension 2 (20.2%)

Type

Regulator

Regulatee

13

14

34

11

19

21

-6

-3

0

3

6

-6

-3

0

3

6

Dimension 1 (36%)

Dimension 2 (18.1%)

Type

Regulator

Regulatee

(d) DCG I vs. DCG II

7

13

14

25

34

4

11

12

19

21

-4

0

4

-4

0

4

Dimension 1 (20.7%)

Dimension 2 (16.2%)

Type

Regulator

Regulatee

Genome Atlas (TCGA) project [22]. After pre-processing, there are 15,135 genes and 427,820 single nucleotide polymorphisms (SNPs) being shared by three cohorts. Cis-eQTL mapping identified 7059 genes with at least one significant SNPs inside their genetic regions (p-value < 0 . 05 ), i.e., valid IVs. We bootstrapped the data 100 times to assess the significance of all effects with results shown in Table 1.

Controlling adjusted p -value at 0.1, we have the largest subnetwork bearing differential structures shown in Figure 5.a, which is verified via STRING [21] as in Figure 5.b. STRING reports a protein-protein interaction (PPI) enrichment with p -value < 10 -16 , implying significant causal effects between the genes shown in Figure 5.a. We identified many previous validated relationships, such as the connected pair RPS14 and RACK1 (GNB2L1) which were experimentally verified by [1]. Our results suggest new findings, particularly causal relationship rather than mere association, e.g., deviated regulations between ARL10 and CLTB from healthy lung tissues in both LUAD and LUSC. Our correspondence analysis of these deviated effects, shown in Figure 5.c-e, confirms these important driver-responder pairs, and the calculated statistics R 2 and C 2 in Table 2 also show such deviation between the three cohorts.

Regulator

Regulatee

(a) Constructed Largest Subnetwork

RPA2

GTF2F1

RRM1

CLPP

EIF3F

RACK1

RPS14

ALKBH7

DNAJC8

LETMD1

UBXN6

MINOS1

CHMP1A

CAPZB

SPATA33

EIF3E

APRT

EIF3L

RPL13

GRB14

LRRC36

KCTD19

PLEKHG4

ZCCHC17

KHDC1

ACTL6A

NDUFB5

HIGD2A

BLOC1S1

MFN1

CLTB

ARL10

ATP5G2

(b) Enrichment in STRING

RPS14

HIGD2A

ARL10

RPA2

ZCCHC17

CLPP

GRB14

EIF3F

PLEKHG4

EIF3E

MINOS1

ACTL6A

RRM1

KHDC1

ALKBH7

BLOC1S1

DNAJC8

CAPZB

APRT

GTF2F1

SPATA33

UBXN6

CLTB

LRRC36

EIF3L

CHMP1A

LETMD1

NDUFB5

KCTD19

ATP5G2

RPL13

GNB2L1

MFN1

HIGD2A

ARL10

CLTB

HIGD2A

ARL10

CLTB

-4

-2

0

2

4

-4

-2

0

2

4

Dimension 1 (29.4%)

Dimension 2 (29.2%)

Type

Regulator

Regulatee

HIGD2A

ARL10

CLTB

ARL10

CLTB

-2.5

0.0

2.5

-2.5

0.0

2.5

Dimension 1 (40.7%)

Dimension 2 (28.7%)

Type

Regulator

Regulatee

(e) LUAD vs. LUSC

EIF3E

KHDC1

GRB14

EIF3E

KHDC1

GRB14

-2

0

2

-2

0

2

Dimension 1 (51.1%)

Dimension 2 (19.4%)

Type

7 Discussion

Though having promising applications, construction and dynamic study of multiple causal networks are computationally challenging in practice due to the involved large systems. We develop the algorithm NetANOVA which avoids optimizing full-information objective functions of the whole system and instead takes limited-information objective functions, each for revealing all drivers of a single responder. Such a responder-focused method allows to deploy parallel computation in two sequential stages and makes it possible to be computationally scalable to the number of involved responders. With available clusters of computers, we can further take the bootstrap method for the usually infeasible task, i.e., evaluating the significance of the constructed causalities in each network and dynamic causalities across networks. Our theoretical analysis and simulation study demonstrate the utility and efficiency of NetANOVA.

With the model complexity of multiple causal networks, especially DCGs that NetANOVA supports, it is a daunting task to visualize them and comprehend each network and variation across multiple networks. We have proposed statistics R 2 and C 2 to quantify the contributions of responders and drivers within a causal network, respectively. While comparing these two statistics across networks may help identify responders and drivers involved in network dynamics, we develop a correspondence analysis to visualize the key players. While our simulation study and real data analysis show promising results, correspondence analysis could be further developed to realize its full potential.

Acknowledgments and Disclosure of Funding

This research is partially supported by the Office of Research of Purdue University and the National Cancer Institute under Grant R03 CA235363. The results shown here are in whole or part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga. The Genotype-Tissue Expression (GTEx) Project was supported by the Common Fund of the Office of the Director of the National Institutes of Health, and by NCI, NHGRI, NHLBI, NIDA, NIMH, and NINDS. The data used for the analyses described in this manuscript were obtained from the GTEx Portal on 10/03/2021 and dbGaP accession number phs000424.v8.p2 on 7/23/2019.

References

Francisco Acosta-Reyes, Ritam Neupane, Joachim Frank, and Israel S Fernández. The Israeli acute paralysis virus IRES captures host ribosomes by mimicking a ribosomal state with hybrid tRNAs. The EMBO Journal , 38(21):e102226, 2019.

Peter J Bickel, Yaacov Ritov, and Alexandre B Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. The Annals of Statistics , 37(4):1705-1732, 2009.

Stephen P Borgatti, Ajay Mehra, Daniel J Brass, and Giuseppe Labianca. Network Analysis in the Social Sciences. Science , 323(5916):892-895, 2009.

Xiaodong Cai, Juan Andrés Bazerque, and Georgios B Giannakis. Inference of Gene Regulatory Networks with Sparse Structural Equation Models Exploiting Genetic Perturbations. PLoS Computational Biology , 9(5):e1003068, 2013.

Chen Chen, Min Ren, Min Zhang, and Dabao Zhang. A Two-Stage Penalized Least Squares Method for Constructing Large Systems of Structural Equations. The Journal of Machine Learning Research , 19(1):40-73, 2018.

GTEx Consortium et al. The GTEx Consortium atlas of genetic regulatory effects across human tissues. Science , 369(6509):1318-1330, 2020.

Mário Cordeiro, Rui P Sarmento, Pavel Brazdil, and João Gama. Evolving Networks and Social Network Analysis Methods and Techniques. Social Media and Journalism: Trends, Connections, Implications , 101(2), 2018.

Henriette Engelhardt, Hans-Peter Kohler, and Alexia Prskawetz. Causal analysis in population studies . Springer, 2009.

Jianqing Fan and Jinchi Lv. Sure Independence Screening for Ultrahigh Dimensional Feature Space. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 70(5): 849-911, 2008.

Ilias Georgakopoulos-Soares, Chengyu Deng, Vikram Agarwal, Candace SY Chan, Jingjing Zhao, Fumitaka Inoue, and Nadav Ahituv. Transcription factor binding site orientation and order are major drivers of gene regulatory activity. Nature communications , 14(1):2333, 2023.

Gene H Golub, Michael Heath, and Grace Wahba. Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics , 21(2):215-223, 1979.

Roger A Horn and Charles R Johnson. Matrix Analysis . Cambridge University Press, 2012.

Yan Li, Dayou Liu, Tengfei Li, and Yungang Zhu. Bayesian differential analysis of gene regulatory networks exploiting genetic perturbations. BMC Bioinformatics , 21(1):1-13, 2020.

Bing Liu, Alberto de La Fuente, and Ina Hoeschele. Gene Network Inference via Structural Equation Modeling in Genetical Genomics Experiments. Genetics , 178(3):1763-1776, 2008.

Judea Pearl. Causality . Cambridge university press, 2009.

Min Ren and Dabao Zhang. Differential Analysis of Directed Networks. arXiv preprint arXiv:1807.10173 , 2018.

Phillippe Rigollet and Jan-Christian Hütter. High Dimensional Statistics. Lecture notes for course 18S997 , 813:814, 2015.

Da Ruan, Alastair Young, and Giovanni Montana. Differential analysis of biological networks. BMC Bioinformatics , 16(1):1-13, 2015.

Mark Rudelson, Roman Vershynin, et al. Hanson-Wright inequality and sub-gaussian concentration. Electronic Communications in Probability , 18, 2013.

Peter Schmidt. Econometrics . New York, Marcel Dekker, 1976.

Damian Szklarczyk, Annika L Gable, David Lyon, Alexander Junge, Stefan Wyder, Jaime Huerta-Cepas, Milan Simonovic, Nadezhda T Doncheva, John H Morris, Peer Bork, et al. String v11: protein-protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets. Nucleic Acids Research , 47(D1):D607-D613, 2019.

John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Mills Shaw, Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, Joshua M Stuart, Cancer Genome Atlas Research Network, et al. The Cancer Genome Atlas Pan-Cancer analysis project. Nature Genetics , 45 (10):1113, 2013.

James West, Ginestra Bianconi, Simone Severini, and Andrew E Teschendorff. Differential network entropy reveals cancer system hallmarks. Scientific Reports , 2, 2012.

Peng Zhao and Bin Yu. On Model Selection Consistency of Lasso. The Journal of Machine Learning Research , 7:2541-2563, 2006.

Xin Zhou and Xiaodong Cai. Inference of differential gene regulatory networks based on gene expression and genetic perturbation data. Bioinformatics , 36(1):197-204, 2020.

Hui Zou. The Adaptive Lasso and Its Oracle Properties. Journal of the American statistical association , 101(476):1418-1429, 2006.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a single structural model to simultaneously construct multiple networks so as to identify causalities varying across multiple cohorts while identifying stable ones.  Each causal network is represented via a directed cyclic graph, and the authors propose an analysis of variance (ANOVA) algorithm (NetANOVA) to identify causalities that are different across networks and an SVD based correspondence analysis to identify the key drivers and responders. Theoretical properties of the algorithm for DCG construction are established and simulations and a data example illustrate the working of their methodology.\n\nSTRENGTHS:\nThe paper deals with an important problem and appears to be a novel contribution as the problem of investigating causality with multiple networks as done here, has not been studied in the literature. It is presented in a very clear manner and is easy to read in most parts. Some technical aspects could be clarified further (as mentioned under Questions below). The simplicity of their framework; theoretical guarantees and computational scalability are the main strengths in my opinion.\n\nWEAKNESSES:\nThe svd based technique employed for correspondence analysis to detect key responder drivers pairs, is not clear. It would be helpful if the authors could elaborate further on this while also providing the intuition behind this. Bootstrapping is used to achieve this- what sort of a bootstrap method is used and why? \nIntuitively, I would expect the accuracy of inferred DCGs  to depend on the range of cohort sizes (min of n^k's to max of n^k's) with lower accuracy as the range increases. Is this the case? If yes, is it evident from the theoretical results?\n\nQUESTIONS:\nPlease answer questions mentioned under weaknesses above. \n\nSection 5: why is error variance of 0.1^2 a good choice? Can you provide the corresponding signal-to-noise ratio? \nSection 3.2: Would it help in interpretation to normalize the coefficient of cause as well so that it lies in (0,1] as R^2? \n\nPlease check the paper for typos (e.g. p8, line 257: n_2, n_3 instead?; p5, line 156: responders; p2, line 54: endogenous etc).\n\nLIMITATIONS:\nI did not find a discussion of limitations of the method. Please include a brief discussion of these in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper, the authors proposed NetANOVA, an algorithm that simultaneously constructs multiple causal networks and infer their disparities. Theoretical justification of the proposed method is also derived. The paper further proposes measures for variable’s contribution to receivers and responders. Overall, the problem seems solid and experiment results reveal its effectiveness.

STRENGTHS:
The task this paper is tackling is of interest and importance. The proposed method is unified on handling multiple causal networks simultaneously and is scalable in various aspects.

The derivation of the method is clear and reasonable. The authors discuss annotations and assumptions of their proposed method.  They show theoretical justification of the algorithm, which consolidates their work.

WEAKNESSES:
The paper is not really good to read. Section 2 starts deriving the method directly without further explanation and formulation of the problem. The authors should consider adding a subsection before the method to formally describe the problem.

The experiments seem good. However, no related methods are compared. The authors  mentioned several related works in the introduction and they highlighted advantages of the proposed method (scalability in various aspects). The experiments do not discuss how their methods compare with others.

QUESTIONS:
How is the proposed method compared with related works?

The model is derived on the basis of several assumptions. While in experiments, do those assumptions really hold on the datasets?

Do the authors conduct runtime analysis of their proposed method?

LIMITATIONS:
It will be good if the authors can provide a formal definition on the problem they are trying to solve.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The paper introduces NetANOVA, an algorithm designed for parallel computation to construct a unified structural model for multiple causal networks, or DCFs. NetANOVA utilizes analysis of variance (ANOVA) to identify causalities that differ across networks, as well as important drivers and responders. It is scalable to large data sizes, computational environments, and model complexities, allowing for efficient analysis of multiple networks.

STRENGTHS:
1. The paper introduces a two-stage parallelizable algorithm that scales when multiple cohorts exist.
2. The paper gives solid theoretical results on the consistency for coefficients of determination and cause.

WEAKNESSES:
1. Experimental results on real data is limited to three DCGs, and it does not contain comparisons with baseline approaches.
2. The theoretical result does not include the computation complexity of NetANOVA.

QUESTIONS:
1. In NetANOVA, most computations are matrix products and inversions. Can you briefly estimate the computation cost required when n and k scales?
2. In Figure 4, are DCG(I/II/III) curated by humans? If so, what is the cost of curating each DCG?

LIMITATIONS:
See "weaknesses."
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a unified structural model that describes multiple DCGs in one model and develops a limited-information-based method to simultaneously infer networks and their disparities. Furthermore, it provides robust non-asymptotic theoretical properties. And it is applied to synthetic and real datasets to show its performance.\n\nSTRENGTHS:\nThe paper proposes a new model that manages to describes multiple DCGs within a single model, and further performs a comprehensive study on it, including algorithmic development, theoretical analysis and synthetic and real-data experiments.\n\nWEAKNESSES:\nThe paper proposes such a model that describes multiple DCGs by stacking K-network-specific structural models in matrix forms with one network chosen as the baseline. However, we fail to see the benefits of proposing such a \"unified\" model, nor any mathematical novelty in model construction. In the problem setup, there is no relation between or information shared by K networks. And in algorithm development, the algorithm is almost like dealing with network-specific parameters independently. In experiments, we did not see any benefits this approach manifests.\n\nDue to this straightforward way of construction, it can be foreseen that the algorithmic development and theoretical analyses naturally follow without particular technical issues. For algorithm development, it seems more or less doing network-specific inference and then stacking results. The theoretical results are mainly these from high-dimensional statistics (specifically l1-related theory on variable screening and selection). Considering the almost independent treatment to each network in algorithm, the theoretical results naturally follow by repeating these theories from analysis of structural models; thus, the introduction of these high-dim stats tools may not be highly appreciated.\n\nThe simulation study and real-data applications only show the \"success\" application of the proposed model. Without the basic comparison to the state-of-the-art or the independent treatments to each network using classical structural model methods, we have no idea on how good the performance of the proposed method is and what additional benefits we gain by modelling multiple networks via a single model.\n\nThe literature review in Introduction is not good, which takes less than 1/3 space of Introduction. It is not a critical review, and it basically lists 6 papers (line 24-38) and tells what they did. These work are neither being explained on their relations to this paper nor commented on their strengths/weakness/debates. Just a list of work, which however is not sufficient to cover important work that closely relates to methods and theories in this paper. What's worse, the sentence at lien 25-26 reads like something irrelevant copied from other biological papers.\n\nQUESTIONS:\nSee the weakness section. We do not have any further technical questions.\n\nLIMITATIONS:\nWe do not see any potential negative societal impact of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm, called NetANOVA, for constructing a unified structural model for multiple causal networks with cycles. The algorithm is designed for parallel computation and is scalable to data size and network complexity. It is able to infer causal networks beyond directed acyclic graphs (DAGs), such as directed cyclic graphs (DCGs), which are demanded to depict gene regulatory networks. The paper provides theoretical justification for the algorithm and demonstrates its feasibility and promise with a large-scale simulation study and a real data analysis to compare gene regulatory networks in healthy lung tissues and lung tissues with two types of cancer.\n\nSTRENGTHS:\n* It proposes a method to unify multiple cyclic graphs with a single structural model and also accommodate their disparities, which is a promising method for analyzing gene regulatory networks where feedback loops are regularly encountered.\n* The algorithm is designed for parallel computation and is scalable to data size and network complexity.\n\nWEAKNESSES:\n* Instrumental variables-based identification of causal effect are proposed for DAGs. There is a lack of discussion on how this extends to causal structures that contains cycles. It is unclear to me if the identification results in 2.3 is correct when cycles exists.\n\nQUESTIONS:\n* The word \"variational\" is used frequently. However, this word usually refers to well-known methods/concepts in machine learning, such as variational inference. Is this word choice for \"variational causality\" validated?\n* Regarding the weakness, how would the existence of cycles affect identification of causal effects using IV? Since most results on IV are either discussed under the potential outcome framework, or discussed using do-calculus when the underlying causal structure can be described by DAGs.\n\nLIMITATIONS:\nThe authors do not explicitly discuss the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}