SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhongli Jiang
Department of Statistics, Purdue University, West Lafayette, IN
jiang548@purdue.edu
Dabao Zhang
Department of Epidemiology & Biostatistics, University of California, Irvine, CA
dabao.zhang@uci.edu
Constructing a directed cyclic graph (DCG) is challenged by both algorithmic difficulty and computational burden. Comparing multiple DCGs is even more difficult, compounded by the need to identify dynamic causalities across graphs. We propose to unify multiple DCGs with a single structural model and develop a limited-information-based method to simultaneously construct multiple networks and infer their disparities, which can be visualized by appropriate correspondence analysis. The algorithm provides DCGs with robust non-asymptotic theoretical properties. It is designed with two sequential stages, each of which involves parallel computation tasks that are scalable to the network complexity. Taking advantage of high-performance clusters, our method makes it possible to evaluate the statistical significance of DCGs using the bootstrap method. We demonstrated the effectiveness of our method by applying it to synthetic and real datasets.
Study of causal networks described by directed cyclic graphs (DCGs) becomes increasingly popular in a variety of fields, such as biomedical and social sciences [10, 3]. Detecting dynamic structures across networks sheds light on mechanistic dynamics, with promising applications. For example, gene regulatory networks related to various types of cancer differ from that of healthy cells in a complicated way and discovery of deviated regulations helps identify cancer-related biological pathways and hence design target-specific drugs [23, 18]. Interpersonal networks in social media have become a favored way for people to get informed of breaking news and natural disasters nowadays. Understanding their evolution over time may provide promising information to businesses, educators, and governments [7]. However, the complex structure and large number of nodes impose challenges on constructing even a single network, not to mention modelling and comparing multiple networks.
Structural models are widely used to describe causal networks and conduct causal inference [8, 15]. With available instrumental variables (IVs) from genomic variations, many methods have been developed to construct large causal networks of gene regulation by integrating transcriptomic and genotypic data. For instance, Liu et al. [14] proposed to assemble driver-responder relationships identified from local structural models, a formidable task due to enormous numbers of possible driver-responder pairs. Cai et al. [4] developed a sparsity-aware maximum likelihood (SML) method by putting  $\ell_1$  penalty on causal effects. It may reach local maximum, leading to the identifiability issue. Chen et al. [5] proposed a two-stage penalized least square (2SPLS) approach which allows parallel computing and shows superior performance.
To infer sparse differences between two large networks, Ren and Zhang [16] recently developed a parallel algorithm based on 2SPLS and achieved better performance than separately constructing the
two networks using 2SPLS. Zhou and Cai [25] proposed a fused sparse SEM (FSSEM) method to learn the differences through maximum likelihood inference of joint networks which is computationally infeasible with large networks. Li et al. [13] employed a Bayesian scheme in their method BFDSEM, unrealistically assuming equal sample sizes from both cohorts.
To the best of our knowledge, there is no algorithm available for investigating causal effects varying across multiple cohorts while identifying stable ones. In this paper, we develop a novel algorithm, designed for parallel computation, to construct a unified structural model for multiple causal networks. While each causal network may be depicted by a DCG, we deliver an analysis of variance (ANOVA) of these networks to identify causalities that are different across networks as well as important drivers and responders. Our algorithm is scalable in several aspects. Firstly, it is scalable to data size including sample size and number of variables. It would still be efficient and powerful under high dimension scheme. Secondly, it is scalable to the computational environment with its parallel computation. The fast computation allows to attack a daunting task in studying multiple networks, i.e., calculating  $p$ -values via the bootstrap method and thus controlling the false discovery rate. Thirdly, the algorithm is scalable to model complexity as it is able to infer causal networks beyond directed acyclic graphs (DAGs), such as DCGs which are demanded to depict gene regulatory networks [14].
The rest of the paper is organized as follows. We first state the model and discuss its identifiability guaranteed by available instrumental variables in Section 2. In Section 3, we present our proposed algorithm ANalysis Of VAriance of directed Networks, termed as NetANOVA, and introduce measures to quantify an endogenous variable's contribution as either drivers or responders. The theoretical justification is shown in Section 4 with detailed proofs in Supplemental Material. We demonstrate the feasibility and promise of our algorithm with a large-scale simulation study shown in Section 5 and a real data analysis to compare gene regulatory networks in healthy lung tissues and lung tissues with two types of cancer in Section 6. We conclude the paper with a discussion in Section 7.
2 A Unified Structural Model of Multiple Causal Networks
2.1 Model specification
We consider  $K$  pertinent causal networks, each for a cohort. For each, say  $k$ -th, cohort we have  $n^{(k)}$  observations in  $(\mathbf{Y}^{(k)},\mathbf{X}^{(k)})$ , where  $\mathbf{Y}^{(k)}$  is an  $n^{(k)} \times p$  matrix including values of  $p$  endogenous variables, and  $\mathbf{X}^{(k)}$  is an  $n^{(k)} \times q$  matrix including values of  $q$  exogenous variables. Each variable is assumed to have mean zero. For the  $k$ -th causal network, we consider each, say  $i$ -th, endogenous variable is causally affected by other variables as follows,
$$
\mathbf {Y} _ {i} ^ {(k)} = \mathbf {Y} _ {- i} ^ {(k)} \boldsymbol {\gamma} _ {i} ^ {(k)} + \mathbf {X} _ {\mathcal {I} _ {i}} ^ {(k)} \phi_ {\mathcal {I} _ {i}} ^ {(k)} + \boldsymbol {\epsilon} _ {i} ^ {(k)}, \tag {1}
$$
where  $\mathbf{Y}_i^{(k)}$  is the  $i$ -th column of  $\mathbf{Y}^{(k)}$ , and  $\mathbf{Y}_{-i}^{(k)}$  is the submatrix of  $\mathbf{Y}^{(k)}$  excluding the  $i$ -th column. The  $(p - 1)$ -dimensional column vector  $\gamma_i^{(k)}$  includes the direct causal effects of other endogenous variables (drivers) on the  $i$ -th one (responder). The set  $\mathcal{I}_i$  includes the indices of exogenous variables that serve as IVs of the  $i$ -th endogenous variable so  $\mathbf{X}_{\mathcal{I}_i}^{(k)}$  is a  $n^{(k)} \times |\mathcal{I}_i|$  matrix including values of corresponding IVs. Each IV has a direct causal effect on its corresponding endogenous variable and can be identified through domain knowledge.  $\phi_{\mathcal{I}_i}^{(k)}$  includes causal effects from corresponding IVs.  $\epsilon_i^{(k)}$  includes disturbance errors which are independently distributed with mean zero and standard deviation  $\sigma_i^{(k)}$ .
Pooling together equation (1) for each endogenous variable, we model the  $k$ -th causal network as,
$$
\mathbf {Y} ^ {(k)} = \mathbf {Y} ^ {(k)} \boldsymbol {\Gamma} ^ {(k)} + \mathbf {X} ^ {(k)} \boldsymbol {\Phi} ^ {(k)} + \boldsymbol {\epsilon} ^ {(k)}, \tag {2}
$$
which describes a DCG as shown in Figure 1.
The  $p \times p$  matrix  $\Gamma^{(k)}$  describes the causal relationships between each pair of endogenous variables and has zero diagonal elements to prohibit self-regulation. The  $q \times p$  matrix  $\Phi^{(k)}$  encodes the causal effects of exogenous variables on endogenous variables. The  $n^{(k)} \times p$  matrix  $\epsilon^{(k)}$  includes all disturbance errors. We assume that both  $\mathbf{Y}^{(k)}$  and  $\mathbf{X}^{(k)}$  have been appropriately centralized within the cohort, so no intercepts are needed in the above model.
(a) DCG I
(b) DCG II
Figure 1: DCGs of three causal networks. The nodes outside the shaded regions are exogenous variables with IVs in light blue and confounding variables (unobservable  $U$  and observed  $O$ ) in white but disturbance errors are omitted. Causal relations in red differ across networks, and dashed ones cannot be revealed by available data due to unobservable  $U$  or unavailable IV for  $Y_{5}$ .
(c) DCG III
2.2 A unified structural model
Many algorithms have been developed to construct a structural model for single causal network [14, 4, 5]. We will show that, in the interest of comparing multiple causal networks, we can also unify them into a single structural model and hence develop an appropriate algorithm to construct all networks at the same time to deliver an ANOVA of multiple networks.
Suppose we have a baseline, say  $K$ -th, network and are interested in others' deviation from the baseline. We first reparameterize the causal effects of all endogenous variables in (1) with
$$
\boldsymbol {\beta} _ {i} ^ {(k)} = \boldsymbol {\gamma} _ {i} ^ {(k)} - \boldsymbol {\gamma} _ {i} ^ {(K)}, \text {f o r} k = 1, 2, \dots , K - 1; \quad \boldsymbol {\beta} _ {i} ^ {(K)} = \boldsymbol {\gamma} _ {i} ^ {(K)}. \tag {3}
$$
Therefore  $\beta_{i}^{(K)}$  includes baseline effects and, for  $k \neq K$ ,  $\beta_{i}^{(k)}$  includes deviated effects of  $k$ -th network from the baseline. When it is of interest to compare the networks with each other, we can similarly reparameterize to consider the deviated effects of each network from the average effects.
For any  $K$  sets of  $l \times m$  matrices, say  $\mathbf{A}_i$  with  $i = 1,2,\dots,K$ , we define a matrix-valued function,
$$
\mathcal {T} (\mathbf {A} _ {1}, \dots , \mathbf {A} _ {K}) = (\mathbf {A} _ {1: K}, (\mathrm {d i a g} (\mathbf {A} _ {1} ^ {T}, \dots , \mathbf {A} _ {K - 1} ^ {T}), \mathbf {0} _ {m (K - 1)}) ^ {T}),
$$
where  $\mathbf{A}_{1:K}$  is constructed by stocking  $\mathbf{A}_1,\dots ,\mathbf{A}_K$  row-wisely, and  $\mathbf{0}_{m(K - 1)}$  is a  $m(K - 1)$ -dimensional column vector with all elements zero. Further denote
$$
\mathbf {Y} _ {i} = \left(\mathbf {Y} _ {i} ^ {(1) T}, \mathbf {Y} _ {i} ^ {(2) T}, \dots , \mathbf {Y} _ {i} ^ {(K) T}\right) ^ {T},
$$
$$
\boldsymbol {\beta} _ {i} = (\boldsymbol {\beta} _ {i} ^ {(K) T}, \boldsymbol {\beta} _ {i} ^ {(1) T}, \boldsymbol {\beta} _ {i} ^ {(2) T}, \dots , \boldsymbol {\beta} _ {i} ^ {(K - 1) T}) ^ {T},
$$
$$
\phi_ {\mathcal {I} _ {i}} = (\phi_ {\mathcal {I} _ {i}} ^ {(1) T}, \phi_ {\mathcal {I} _ {i}} ^ {(2) T}, \dots , \phi_ {\mathcal {I} _ {i}} ^ {(K) T}),
$$
$$
\boldsymbol {\epsilon} _ {i} = \left(\boldsymbol {\epsilon} _ {i} ^ {(1) T}, \boldsymbol {\epsilon} _ {i} ^ {(2) T}, \dots , \boldsymbol {\epsilon} _ {i} ^ {(K) T}\right) ^ {T},
$$
Notice that  $\gamma_{i}^{(k)} = \beta_{i}^{(K)} + \beta_{i}^{(k)}$  for  $k = 1,2,\ldots ,K - 1$  from (3), we have a unified structural model for  $K$  causal networks described by (2),
$$
\mathbf {Y} _ {i} = \mathcal {T} \left(\mathbf {Y} _ {- i} ^ {(1)}, \mathbf {Y} _ {- i} ^ {(2)}, \dots , \mathbf {Y} _ {- i} ^ {(K)}\right) \beta_ {i} + \operatorname {d i a g} \left(\mathbf {X} _ {\mathcal {I} _ {i} ^ {(1)}}, \mathbf {X} _ {\mathcal {I} _ {i} ^ {(2)}} \dots , \mathbf {X} _ {\mathcal {I} _ {i} ^ {(K)}}\right) \phi_ {\mathcal {I} _ {i}} + \boldsymbol {\epsilon} _ {i}. \tag {4}
$$
2.3 Instrumental Variables, model identifiability, and confounding variables
As shown in Figure 1, the ability to reveal causal effects relies on the available IVs. An IV is an exogenous variable that only affects directly the driver but not the responder, except indirectly through its effect on the driver. This means that IVs can be used to address the model identifiability issue, which is complicated by the model's inherent endogeneity, and allow us to isolate the causal effects of endogenous variables. We specify the following assumption on available IVs following the rank condition [20], a necessary and sufficient condition for model identification.
Assumption 1. a.  $\mathbf{X}^{(k)}\perp \epsilon^{(k)}$  ; b.  $\exists \mathcal{C}\subset \{1,2,\dots ,p\}$  but  $\mathcal{C}\neq \emptyset$  ; c.  $\forall i\in \mathcal{C},\exists \mathcal{I}_i\subset \{1,2,\dots ,q\}$  with  $\mathcal{I}_i\neq \emptyset$  and  $\phi_{\mathcal{I}_i}^{(k)}\neq 0$  ; d.  $\forall i,j\in \mathcal{C}$  with  $i\neq j,\mathcal{I}_i\cap \mathcal{I}_j = \emptyset$  ; e.  $\forall j\notin \mathcal{C},\gamma_{ij}^{(k)} = 0$
Assumption 1.a specifies IVs observed in  $\mathbf{X}^{(k)}$  that are uncorrelated to the disturbance errors. Assumption 1.b specifies the endogenous variables in  $\mathcal{C}$  that have available IVs and Assumptions 1.c and 1.d state the restrictions on the available IVs, i.e., their availability and uniqueness, respectively. While  $\gamma_{ij}^{(k)}$  is the  $j$ -th component of  $\gamma_{i}^{(k)}$ , Assumption 1.e states that we are not able to identify any causal relation with  $j$ -th endogenous variable as a driver if it has no IV available, e.g.,  $Y_{5}$  in Figure 1.
A striking advantage of the IV method is its robustness to confounding but exogenous variables, which may be observable or unobservable, as shown in Figure 1. As will be demonstrated by our later algorithm, available IVs help disassociate an endogenous variable from disturbance errors of other endogenous variables in model fitting. Therefore, we may explicitly incorporate observable confounding variables to improve causal inference. However, when there are unobservable confounding variables, we can still effectively construct multiple networks with available IVs.
3 Model Building and Interpretation
The single structural model in (4) makes it possible to employ a limited-information approach to construct multiple causal networks even in the case of a large number of endogenous variables. Therefore we develop a two-stage algorithm NetANOVA which allows parallel computing for fast computation and hence bootstrapping the data for significance assessment. Another daunting but necessary task is to understand the multiple networks and catch the important variables in the networks and important dynamic causalities across them. We therefore propose coefficients of determination and cause as well as correspondence analysis.
3.1 The algorithm NetANOVA
The disassociation stage: We first predict each endogenous variable in  $\mathcal{C}$ , solely based on all available exogenous variables, to disassociate it from the disturbance errors of other endogenous variables, following aforementioned assumption 1. For this purpose, we rearrange the terms in (2) to obtain the reduced model,
$$
\mathbf {Y} ^ {(k)} = \mathbf {X} ^ {(k)} \boldsymbol {\pi} ^ {(k)} + \boldsymbol {\xi} ^ {(k)}, \tag {5}
$$
where  $\pmb{\pi}^{(k)} = \pmb{\Phi}^{(k)}(\mathbf{I} - \pmb{\Gamma}^{(k)})^{-1}$  and  $\pmb{\xi}^{(k)} = \epsilon^{(k)}(\mathbf{I} - \pmb{\Gamma}^{(k)})^{-1}$ .
As shown in the real data analysis in constructing gene regulatory networks, the number of exogenous variables, i.e., the dimension  $q$  for  $\mathbf{X}^{(k)}$ , can be much larger than the sample size  $n^{(k)}$ , rising issues of prediction consistency and computational time. To address them, we first apply the iterative sure independence screening (ISIS; [9]) to screen for exogenous variables and then perform regression with  $\ell_2$  penalty. As later theoretical analysis shows, it allows for  $q \lesssim \exp(n^{(k)\theta})$  for some  $\theta > 0$ .
Specifically for each endogenous variable  $i \in \mathcal{C}$  and  $k \in \{1, \dots, K\}$ , we apply ISIS with computational complexity  $O(n^{(k)}p)$  to screen for a set of exogenous variables, indexed by set  $\mathcal{M}_i^{(k)}$ . Using these  $d = |\mathcal{M}_i^{(k)}|$  exogenous variables, we apply ridge regression to obtain the ridge estimator,
$$
\hat {\boldsymbol {\pi}} _ {\mathcal {M} _ {i} ^ {(k)}} ^ {(k)} = \left(\mathbf {X} _ {\mathcal {M} _ {i} ^ {(k)}} ^ {T} \mathbf {X} _ {\mathcal {M} _ {i} ^ {(k)}} + \lambda_ {i} ^ {(k)} I\right) ^ {- 1} \mathbf {X} _ {\mathcal {M} _ {i} ^ {(k)}} ^ {T} \mathbf {Y} _ {i} ^ {(k)} \tag {6}
$$
and predict  $\mathbf{Y}_i^{(k)}$  with
$$
\hat {\mathbf {Y}} _ {i} ^ {(k)} = \mathbf {X} _ {\mathcal {M} _ {i} ^ {(k)}} ^ {(k)} \hat {\boldsymbol {\pi}} _ {\mathcal {M} _ {i} ^ {(k)}} ^ {(k)}, \tag {7}
$$
where  $\lambda_i^{(k)}$  is a tuning parameter that can be selected via generalized cross validation [11].
The inference stage: We use the predicted endogenous variables in the previous stage to identify and estimate the causal effects, hence constructing the causal networks. We first calculate the projection matrix,
$$
\mathbf {P} _ {i} ^ {(k)} = \mathbf {I} _ {n ^ {(k)}} - \mathbf {X} _ {\mathcal {I} _ {i}} ^ {(k)} \left(\mathbf {X} _ {\mathcal {I} _ {i}} ^ {(k) T} \mathbf {X} _ {\mathcal {I} _ {i}} ^ {(k)}\right) ^ {- 1} \mathbf {X} _ {\mathcal {I} _ {i}} ^ {(k) T}.
$$
Multiplying the projection matrix  $\mathbf{P}_i = \mathrm{diag}\{\mathbf{P}_i^{(1)},\mathbf{P}_i^{(2)},\ldots ,\mathbf{P}_i^{(K)}\}$  to both sides of model (4), we can eliminate the exogenous variables from the model and get,
$$
\mathbf {P} _ {i} \mathbf {Y} _ {i} = \mathbf {P} _ {i} \boldsymbol {\Upsilon} _ {- i} \boldsymbol {\beta} _ {i} + \mathbf {P} _ {i} \boldsymbol {\epsilon} _ {i}, \tag {8}
$$
where  $\mathbf{Y}_{-i} = \mathcal{T}(\mathbf{Y}_{-i}^{(1)},\mathbf{Y}_{-i}^{(2)},\dots ,\mathbf{Y}_{-i}^{(K)})$  . Since  $\mathbf{P}_i\mathbf{\Psi}_{-i}$  and  $\epsilon_{i}$  are correlated, we instead regress  $\mathbf{P}_i\mathbf{Y}_i$  against  $\mathbf{P}_i\hat{\mathbf{Y}}_{-i}$  with  $\hat{\mathbf{Y}}_{-i} = \mathcal{T}(\hat{\mathbf{Y}}_{-i}^{(1)},\hat{\mathbf{Y}}_{-i}^{(2)},\dots ,\hat{\mathbf{Y}}_{-i}^{(K)})$  which is disassociated from  $\epsilon_{i}$  . With possibly high-dimensional  $\beta_{i}$  , we apply adaptive lasso [26] to obtain its estimator,
$$
\hat {\boldsymbol {\beta}} _ {i} = \underset {\boldsymbol {\beta} _ {i}} {\arg \min} \left\{\frac {1}{n} | | \mathbf {P} _ {i} \mathbf {Y} _ {i} - \mathbf {P} _ {i} \hat {\mathbf {Y}} _ {- i} \boldsymbol {\beta} _ {i} | | _ {2} ^ {2} + \nu_ {i} \hat {\boldsymbol {\omega}} _ {i} ^ {T} | \boldsymbol {\beta} _ {i} | _ {1} \right\},
$$
where  $\nu_{i}$  is a tuning parameter, and  $\hat{\omega}_{i} = |\hat{\beta}_{0i}|^{-\gamma}$  for some  $\gamma > 0$  where  $|\hat{\beta}_{0i}|_{1}$  contains the absolute values of  $\hat{\beta}_{0i}$ , which is a preliminary estimator. The original networks can be recovered by calculating  $\hat{\gamma}_{i}^{(k)} = \hat{\beta}_{i}^{(K)} + \hat{\beta}_{i}^{(k)}$  for  $k = 1,2,\ldots,K-1$ .
In summary of these two stages, we summarize the algorithm in Algorithm 1.
Algorithm 1 ANalysis Of VAriance of directed Networks (NetANOVA)
Input:  $(\mathbf{Y}^{(k)},\mathbf{X}^{(k)})$ ,  $k\in \{1,2,\dots,K\}$ , with each variable centralized within cohort but scaled according to the baseline cohort; Predefined index set  $\mathcal{I}_i$  for each  $i\in \mathcal{C}$ ;  $d\gets O(n_{\min}^{1 - \theta})$ .
for  $i\in \mathcal{C}$  do
1. Reduce the dimension of  $\mathbf{X}^{(k)}$  by ISIS to get  $\mathbf{X}_{\mathcal{M}_i^{(k)}}^{(k)}$ ; Set  $\mathbf{X}_{\mathcal{M}_i^{(k)}}^{(k)} = \mathbf{X}^{(k)}$  if  $q \leq n^{(k)}$ .
2. Estimate  $\hat{\mathbf{Y}}_i^{(k)}$  by regressing  $\mathbf{Y}_i^{(k)}$  against  $\mathbf{X}_{\mathcal{M}_i^{(k)}}^{(k)}$  with ridge regression.
for  $i = 1,2,\ldots ,p$  do
1. Calculate projection matrices  $\mathbf{P}_i$
2. Predict  $\hat{\mathbf{Y}}_{-i}$  from  $\hat{\mathbf{Y}}_i^{(k)}$ ,  $i\in \mathcal{C}$ .
3. Estimate  $\hat{\beta}_i$  by regressing  $\mathbf{P}_i\mathbf{Y}_i$  against  $\mathbf{P}_i\hat{\mathbf{Y}}_{-i}$  with adaptive lasso.
Output:  $\hat{\beta}_1, \dots, \hat{\beta}_p$  which contains the baseline and differential causal effects.
3.2 Coefficient of determination and coefficient of cause
For each endogenous variable  $i$ , we can calculate its coefficient determination to measure the proportion of its variation due to the effects of its drivers, i.e.,  $\mathbf{Y}_{-i}^{(k)}\pmb{\gamma}_i^{(k)}$  in (1), for cohort  $k$ ,
$$
R _ {i} ^ {2 (k)} = 1 - | | \mathbf {Y} _ {i} ^ {(k)} - \mathbf {Y} _ {- i} ^ {(k)} \hat {\gamma} _ {i} ^ {(k)} | | _ {2} ^ {2} / | | \mathbf {Y} _ {i} ^ {(k)} | | _ {2} ^ {2},
$$
On the other hand, we can also calculate the coefficient of cause for each endogenous variable  $i$  which summarizes proportions that it contributes to the variation of its responders, for cohort  $k$ ,
$$
C _ {i} ^ {2 (k)} = \sum_ {j = 1} ^ {p} \left(1 - \left| \left| \mathbf {Y} _ {j} ^ {(k)} - \mathbf {Y} _ {i} ^ {(k)} \hat {\gamma} _ {j i} ^ {(k)} \right| \right| _ {2} ^ {2} / \left| \left| \mathbf {Y} _ {j} ^ {(k)} \right| \right| _ {2} ^ {2}\right).
$$
While we name the above coefficient of determination as causal  $R^2$  with its value between zero and one, we will simply call the coefficient of cause  $C^2$  which is positive but may be greater than one.
3.3 Correspondence analysis of causal effects
We can conduct a correspondence analysis of causal effects in  $\Gamma^{(k)}$  to reveal clusters of drivers, responders, or driver-responder pairs which outstand from the rest in each cohort  $k$ . We can also conduct a correspondence analysis of deviated causal effects to reveal these clusters which deviate the most from a baseline or the rest. However, each causal effect or its deviation may vary differently and we need to standardize them, based on the bootstrap results, before correspondence analysis. For example, to compare cohorts  $k$  and  $l$  for their causal effects, we should instead obtain the following
summary statistic,
$$
z _ {i j} ^ {(k, l)} = \left(\hat {\gamma} _ {i j} ^ {(k)} - \hat {\gamma} _ {i j} ^ {(l)}\right) \Bigg / \left(\sum_ {b = 1} ^ {B} (\hat {\gamma} _ {i j} ^ {(k, b)} - \hat {\gamma} _ {i j} ^ {(l, b)}) ^ {2} / (B - 1)\right) ^ {1 / 2}.
$$
where  $\hat{\gamma}_{ij}^{(k)}$  and  $\hat{\gamma}_{ij}^{(k,b)}$  are estimates of corresponding effects from the observed data and  $b$ -th set of bootstrap data, respectively; we also let  $z_{ij}^{(k)} = 0$  when both of its numerator and denominator are sufficiently small. We can define similar statistics to investigate effects' variation from their means.
A singular value decomposition (SVD) of  $Z^{(k,l)} = (z_{ij}^{(k,l)})_{p\times p}$  can obtain its left and right singular vectors, say  $\{U_i^{(k,l)}\}$  and  $\{V_i^{(k,l)}\}$ , respectively. While the left singular vectors help identify responders that differ between the two cohorts, the right singular vectors help identify drivers that differ between the two cohorts. We can overlay the plot of  $U_2^{(k)}$  vs.  $U_1^{(k)}$  with the plot of  $V_2^{(k)}$  vs.  $V_1^{(k)}$  to identify responder-driver pairs for their causalities varying the most across networks.
In this section, we will establish non-asymptotic guarantees and show that our constructed DCGs have good theoretical properties. We characterize the properties with a prespecified sequence  $\delta^{(k)} = \exp\{-o(n^{(k)})\}$  with  $\delta^{(k)} \to 0$  as  $n^{(k)} \to \infty$ , i.e., each  $\delta^{(k)}$  approaches zero slower than  $\exp\{-n^{(k)}\}$ . We denote  $\delta_{min} = \min_{1 \leq k \leq K} \delta^{(k)}$  and only present main results here, with details including general notations, assumptions, and proofs in Supplementary Material.
Theorem 4.1. Let  $n = \sum_{k=1}^{K} n^{(k)}$ ,  $S_i = \text{supp}(\beta_i)$ , and  $g_n$  is a function of  $n$  and  $\delta_{\min}$  specified in (9) in Supplemental Material. Then under Assumption 1 and Assumptions B.1-B.4 in Supplementary Material, we have that, with probability at least  $1 - \delta_{\min} - \delta$  with  $\delta = p \sum_{k=1}^{K} \delta^{(k)}$ ,
1. (Bounded Errors)  $||\hat{\beta}_i - \beta_i||_2^2 \lesssim |S_i| g_n \{d \vee \log(pK / \delta) \vee ||\pmb{\pi}||_{2max}^2 \vee \log(d / \delta_{min})\} / n;$
2. (Causality Consistency)  $\text{sign}(\hat{\beta}_i) = \text{sign}(\beta_i)$ .