SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We will next show that, with proper choice of  $\{\delta^{(k)}\}$ , we can bound the estimation errors system-wise with ultra high probability. We can pick  $\{\delta^{(k)}\}$  such that  $\delta_{min} \asymp e^{-n^t}$  where  $t \in (0, \min(\theta, 1 - \theta))$ . Then we have  $f_n \lesssim n^{(2 - \theta)/2}$ , and the restriction on the number of true signals can be reduced to  $|S_i| \lesssim n^{\theta/2}$ , which is a requirement that can be fulfilled especially in a sparse model. We have  $d = n^{1 - \theta}$ , so  $g_n \lesssim n^{(1 - 2\theta)/2}$ . As a result, the bound can be dominated by the order of  $|S_i| n^{(1 - 4\theta)/2}$ .
Note that the error for the whole system of  $p$  nodes can be controlled by a similar bound, replacing each occurrence by  $|S_i|$  with  $\max_{1 \leq i \leq p} |S_i|$ , with probability at least  $1 - p(\delta_{\min} + \delta)$ . Following the former calculation, we learn that, when  $\max_{1 \leq i \leq p} |S_i|$  grow slower than  $n^{(4\theta - 1)/2}$ , the error will approach zero when  $n \to \infty$ . To achieve the bound with high probability, we only need  $p\delta$  to diminish. That is, we could have the error well controlled even if the dimension grows up to  $e^{n^s}$  where  $s \in (0, \frac{t}{2})$ .
Given the auspicious estimation performance, we will further discuss the benefit of pooling cohorts compared to that of learning in a single-cohort analysis, where the estimators  $\beta_{i}^{(k)}$  for each cohort are estimated separately using the algorithm without re-parameterization. For each  $k$ , we can separately conduct the analysis on each cohort and obtain the following property for single task estimation.
Corollary 4.1.1. Under the same conditions as Theorem 4.1, we have that, with probability at least  $1 - \delta_{\min} - \delta / K$ ,
$$
\left| \left| \hat {\gamma} _ {i} ^ {(k)} - \gamma_ {i} ^ {(k)} \right| \right| _ {2} ^ {2} \lesssim \left| S _ {i} \right| g _ {n} \left\{d \vee \log (p K / \delta) \vee \left\| \boldsymbol {\pi} \right\| _ {2 m a x} ^ {2} \vee \log (d / \delta_ {m i n}) \right\} / n ^ {(k)}.
$$
The proof directly follows by treating each cohort as a baseline cohort in Theorem 4.1. Note that the denominator is making a crucial difference in inferring the bound. The algorithm indicates that when we aggregate the samples that does not differ too much, the estimator tends to converge faster,
especially when  $n \gg n^{(k)}$ , where we have a plethora of samples in addition to the samples for the base cohort.
With the same choice of  $\delta_{min}$  and  $\delta$  as the above theorem, the probability for consistent variable selection can approach one, with a less stringent requirement in the size of  $S_{i}$ . The above theorem implies that our proposed method can identify both baseline and differential regulatory effects, among all of the  $K$  networks with a sufficiently large probability, not only in terms of the set of true signals but also the sign of signals. In the case of gene regulatory networks, for instance, our method could correctly distinguish the up and down regulations between genes.
Next, we will present a theorem for the coefficient of determination and coefficient of cause defined in Section 3.2. Denote the statistics  $R^2$  and  $C^2$  calculated with real parameters as, respectively,
$$
R _ {0 i} ^ {2 (k)} = 1 - | | \mathbf {Y} _ {i} ^ {(k)} - \mathbf {Y} _ {- i} ^ {(k)} \gamma_ {i} ^ {(k)} | | _ {2} ^ {2} / | | \mathbf {Y} _ {i} ^ {(k)} | | _ {2} ^ {2}, C _ {0 i} ^ {2 (k)} = \sum_ {j = 1} ^ {p} (1 - | | \mathbf {Y} _ {j} ^ {(k)} - \mathbf {Y} _ {i} ^ {(k)} \gamma_ {j i} ^ {(k)} | | _ {2} ^ {2} / | | \mathbf {Y} _ {j} ^ {(k)} | | _ {2} ^ {2}).
$$
Then we can derive the following properties.
Theorem 4.2. (Coefficients of Determination/Cause Consistency) Under the same conditions as in Theorem 4.1, with  $h_n = \sqrt{n^{(k)}} + \sqrt{-\log(\delta_{min})} + 1$ , we have that,
1. With probability at least  $1 - p(2\delta_{min} + \delta / K)$ ,
$$
\sum_ {i = 1} ^ {p} | R _ {i} ^ {2 ^ {(k)}} - R _ {0 i} ^ {2 ^ {(k)}} | \lesssim \frac {| \mathcal {S} _ {i} | g _ {n} \{d \vee \log (p / \delta) \vee \| \boldsymbol {\pi} \| _ {2 m a x} ^ {2} \vee \log (d / \delta_ {m i n}) \}}{n ^ {(k) 3 / 4}} (1 + | | \phi_ {\mathcal {I} _ {i}} ^ {(k)} | | _ {2} h _ {n});
$$
2. With probability at least  $1 - p(\delta_{\min} + \delta /K)$
$$
\sum_ {i = 1} ^ {p} \left| C _ {i} ^ {2 ^ {(k)}} - C _ {0 i} ^ {2 ^ {(k)}} \right| \lesssim \frac {\left| \mathcal {S} _ {i} \right| g _ {n} \left\{d \vee \log (p / \delta) \vee \| \boldsymbol {\pi} \| _ {2 m a x} ^ {2} \vee \log (d / \delta_ {m i n}) \right\}}{n ^ {(k)}} (1 + | | B | | _ {1} \vee 1).
$$
As considered in Theorem 4.1, we set  $\delta_{min} \asymp \delta / (pK) \asymp e^{-n^t}$ . Then  $h_n \lesssim \sqrt{n^{(k)}}$ , and  $\sum_{i=1}^{p} |R_i^{2(k)} - R_{0i}^{2(k)}| \lesssim \max_{1 \leq i \leq p} |S_i| n^{(1-4\theta)/2} + (\max_{1 \leq i \leq p} |S_i|)^{1/2} n^{\frac{-2\theta-1}{2}}$  which would vanish if  $\max_{1 \leq i \leq p} |S_i| \lesssim n^{(4\theta-1)/2}$ . Similarly,  $\sum_{i=1}^{p} |C_i^{2(k)} - C_{0i}^{2(k)}| \lesssim \max_{1 \leq i \leq p} |S_i| n^{(1-4\theta)/4}$  and will converge to 0 when  $\max_{1 \leq i \leq p} |S_i| \lesssim n^{(4\theta-1)/4}$ . From the above discussion, both  $R^2, C^2$  for all the nodes in the whole system is  $\ell_1$  consistent. It is worth noting that, although our algorithm fits a model for each responder, estimate of each node's explanatory power as a driver is also well controlled over the system, but with a slightly larger bound as shown in the above theorem.
5 Simulation Study
We examine the performance of NetANOVA by simulating data of sample size 200, 500 and 1000 from each of three pertinent DCGs, consisting of 1000 endogenous variables. However, only 50 endogenous variables are involved with causality, with each regulated by 3 others on average. Each of the three networks has 5 unique causal effects. First two networks share five causal effects but with opposite signs to that of the third one, resulting in the first two networks having 15 different effects from the third one and the first network have 20 different effects from the second one. The size for causal effects is taken from a uniform distribution over  $[-0.8, -0.3] \cup [0.3, 0.8]$ . The IVs are generated from a multinomial distribution with 3 outcomes 0, 1, 2, with probabilities 0.25, 0.5, 0.25 respectively. The disturbance errors are sampled independently from  $N(0, 0.1^2)$ .
For each sample size, we simulated 100 data sets and applied NetANOVA to each by bootstrapping 100 times to calculate  $p$  values and construct Receiver Operating Characteristic (ROC) curves as shown in Figure 2. For causal effects of all three DCGs, Figure 2.a shows an almost perfect area under curve (AUC) for each sample size. When comparing causal effects of the first two DCGs vs. the baseline one, i.e., the third DCG, the AUC bottoms at 0.867 with sample size at 200, but reaches one with sample size at 1000, showing excellent performance of NetANOVA.
(a) Three DCGs
(b) DCG 1 vs. DCG 3
(c) DCG 2 vs. DCG 3
Figure 2: ROC curves of NetANOVA on simulated data. (a) ROC curves of causal effects constructed in all three networks; (b) ROC curves of deviated effects of the first network from the third one; (c) ROC curves of deviated effects of the second network from the third one; (d) ROC curves of deviated effects of the first network from the second one.
(d) DCG 1 vs. DCG 2
Taking one simulated dataset with sample size at 500, we evaluated our proposed statistics  $R^2$  and  $C^2$  by plotting estimated statistics against the statistics calculated with true causal effects in Figure 3. For each of the three networks, we observe linear trends with slope almost one for both  $R^2$  and  $C^2$  statistics, indicating their excellent performance.
(a)
Figure 3: Statistics  $R^2$  and  $C^2$  on a simulated data with sample size 500. We plot each statistic ( $R^2$  or  $C^2$ ) vs. its value calculated with true causal effects ( $R_0^2$  or  $C_0^2$ ) for each network. We simply label the three networks as 1, 2, and 3. Shown in the top-left corners are the fitted linear regression models between the pairs as well as the corresponding coefficients of determination.
We also took one simulated data with sample size 500 and conducted our proposed correspondence analysis to identify drivers and responders that show important dynamic causality across the three DCGs. When comparing DCG I vs. DCG III, Figure 4.b shows a driver-responder pair (14, 45) with different causal effects, which is verified by observing 14 significantly up regulates 45 in DCG I vs. DCG III in Figure 4.a (blue connections). However, we also see the driver 18 and responder 45 stay opposite on the  $y$ -axis, because 18 significantly down regulates 45 implying in DCG I vs. DCG III in Figure 4.a. On  $x$ -axis, we see pairs (12,13) and (12,7) which correspond to another cluster of causal networks as shown in the right of Figure 4.a.
6 Real Data Analysis
We applied NetANOVA to investigate the gene regulatory networks of lung tissues of healthy individuals and patients with lung adenocarcinoma (LUAD) or lung squamous cell carcinoma (LUSC). We obtained transcriptomic and genotypic data of healthy lungs ( $n = 482$ ) from the Genotype-Tissue Expression (GTEX) project [6] and of both LUAD ( $n = 485$ ) and LUSC ( $n = 406$ ) from The Cancer
(a) Part of the True Networks
(b) DCG I vs. DCG III
(c) DCG II vs. DCG III
Figure 4: Correspondence analysis of NetANOVA on a simulated data with sample size 500. (a) Plot of some deviated causal effects among the three true networks (shown in Figure 6) with DCG I vs. DCG III, DCG II vs. DCG III, and DCG I vs. DCG II in blue, red, and green, respectively. There are a total of 1000 endogenous variables with the first 50 involved in causal relations. (b)-(d) Plots for correspondence analysis with drivers in red and responders in blue.
(d) DCG I vs. DCG II
Genome Atlas (TCGA) project [22]. After pre-processing, there are 15,135 genes and 427,820 single nucleotide polymorphisms (SNPs) being shared by three cohorts. Cis-eQTL mapping identified 7059 genes with at least one significant SNPs inside their genetic regions (p-value  $< 0.05$ ), i.e., valid IVs. We bootstrapped the data 100 times to assess the significance of all effects with results shown in Table 1.
Table 1: Summary of causal effects identified by NetANOVA. Shown in the columns are the results from the original data, different bootstrap cutoffs (80% - 100%), and adjusted by Benjamini-Hochberg adjustment (BH), respectively.
Type of Effects	Original	80%	90%	95%	100%	BH
Healthy Tissue	79833	16594	11481	8760	4602	3165
LUAD vs Healthy Tissue	13711	1185	848	670	458	296
LUSC vs Healthy Tissue	13104	1385	980	768	477	274
LUSC vs LUAD	18139	1615	976	665	289	38
Controlling adjusted  $p$ -value at 0.1, we have the largest subnetwork bearing differential structures shown in Figure 5.a, which is verified via STRING [21] as in Figure 5.b. STRING reports a protein-protein interaction (PPI) enrichment with  $p$ -value  $< 10^{-16}$ , implying significant causal effects between the genes shown in Figure 5.a. We identified many previous validated relationships, such as the connected pair RPS14 and RACK1 (GNB2L1) which were experimentally verified by [1]. Our results suggest new findings, particularly causal relationship rather than mere association, e.g., deviated regulations between ARL10 and CLTB from healthy lung tissues in both LUAD and LUSC. Our correspondence analysis of these deviated effects, shown in Figure 5.c-e, confirms these important driver-responder pairs, and the calculated statistics  $R^2$  and  $C^2$  in Table 2 also show such deviation between the three cohorts.
(a) Constructed Largest Subnetwork
(b) Enrichment in STRING
(c) LUAD vs. Healthy
(d) LUSC vs. Healthy
(e) LUAD vs. LUSC
Figure 5: Partial results of gene regulatory networks for human lung. (a) The largest subnetworks of gene regulation with deviated causal effects between healthy, LUAD, and LUSC lung tissues. Shown in black is the gene regulatory network for healthy lung tissue. Shown in blue, red, green are significantly deviated causal effects of LUAD vs. healthy, LUSC vs. healthy, and LUAD vs. LUSC, respectively. (b) Enrichment of the largest subnetwork in STRING. (c)-(e) Correspondence analysis of the deviated effects between the largest subnetworks with drivers in red and responders in blue.
Though having promising applications, construction and dynamic study of multiple causal networks are computationally challenging in practice due to the involved large systems. We develop the algorithm NetANOVA which avoids optimizing full-information objective functions of the whole system and instead takes limited-information objective functions, each for revealing all drivers of a single responder. Such a responder-focused method allows to deploy parallel computation in two sequential stages and makes it possible to be computationally scalable to the number of involved responders. With available clusters of computers, we can further take the bootstrap method for the usually infeasible task, i.e., evaluating the significance of the constructed causalities in each network and dynamic causalities across networks. Our theoretical analysis and simulation study demonstrate the utility and efficiency of NetANOVA.
Table 2: Statistics  $R^2$  and  $C^2$  for Important Genes
Cohort	ARL10	CLTB	EIF3E	GRB14	HIGD2A	KHDC1
R²	LUAD	0.90	0.74	0.73	0.58	0.71	0.60
LUSC	0.79	0.79	0.77	0.75	0.54	0.00
Healthy	0.00	0.00	0.56	0.00	0.30	0.00
C²	LUAD	1.41	0.74	1.18	0.00	0.86	0.60
LUSC	0.79	0.79	0.75	0.75	0.16	0.00
Healthy	0.00	0.00	0.00	0.00	0.34	0.00
With the model complexity of multiple causal networks, especially DCGs that NetANOVA supports, it is a daunting task to visualize them and comprehend each network and variation across multiple networks. We have proposed statistics  $R^2$  and  $C^2$  to quantify the contributions of responders and drivers within a causal network, respectively. While comparing these two statistics across networks may help identify responders and drivers involved in network dynamics, we develop a correspondence analysis to visualize the key players. While our simulation study and real data analysis show promising results, correspondence analysis could be further developed to realize its full potential.
This research is partially supported by the Office of Research of Purdue University and the National Cancer Institute under Grant R03 CA235363. The results shown here are in whole or part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga. The Genotype-Tissue Expression (GTEX) Project was supported by the Common Fund of the Office of the Director of the National Institutes of Health, and by NCI, NHGRI, NHLBI, NIDA, NIMH, and NINDS. The data used for the analyses described in this manuscript were obtained from the GTEX Portal on 10/03/2021 and dbGaP accession number phs000424.v8.p2 on 7/23/2019.
[1] Francisco Acosta-Reyes, Ritam Neupane, Joachim Frank, and Israel S Fernandez. The Israeli acute paralysis virus IRES captures host ribosomes by mimicking a ribosomal state with hybrid tRNAs. The EMBO Journal, 38(21):e102226, 2019.
[2] Peter J Bickel, Yaacov Ritov, and Alexandre B Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. The Annals of Statistics, 37(4):1705-1732, 2009.
[3] Stephen P Borgatti, Ajay Mehra, Daniel J Brass, and Giuseppe Labianca. Network Analysis in the Social Sciences. Science, 323(5916):892-895, 2009.
[4] Xiaodong Cai, Juan Andres Bazerque, and Georgios B Giannakis. Inference of Gene Regulatory Networks with Sparse Structural Equation Models Exploiting Genetic Perturbations. PLoS Computational Biology, 9(5):e1003068, 2013.
[5] Chen Chen, Min Ren, Min Zhang, and Dabao Zhang. A Two-Stage Penalized Least Squares Method for Constructing Large Systems of Structural Equations. The Journal of Machine Learning Research, 19(1):40–73, 2018.
[6] GTEx Consortium et al. The GTEx Consortium atlas of genetic regulatory effects across human tissues. Science, 369(6509):1318-1330, 2020.
[7] Mário Cordeiro, Rui P Sarmento, Pavel Brazdil, and João Gama. Evolving Networks and Social Network Analysis Methods and Techniques. Social Media and Journalism: Trends, Connections, Implications, 101(2), 2018.
[8] Henriette Engelhardt, Hans-Peter Kohler, and Alexia Prskawetz. Causal analysis in population studies. Springer, 2009.
[9] Jianqing Fan and Jinchi Lv. Sure Independence Screening for Ultrahigh Dimensional Feature Space. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(5): 849-911, 2008.
[10] Ilias Georgakopoulos-Soares, Chengyu Deng, Vikram Agarwal, Candace SY Chan, Jingjing Zhao, Fumitaka Inoue, and Nadav Ahituv. Transcription factor binding site orientation and order are major drivers of gene regulatory activity. Nature communications, 14(1):2333, 2023.
[11] Gene H Golub, Michael Heath, and Grace Wahba. Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics, 21(2):215–223, 1979.
[12] Roger A Horn and Charles R Johnson. Matrix Analysis. Cambridge University Press, 2012.
[13] Yan Li, Dayou Liu, Tengfei Li, and Yungang Zhu. Bayesian differential analysis of gene regulatory networks exploiting genetic perturbations. BMC Bioinformatics, 21(1):1-13, 2020.
[14] Bing Liu, Alberto de La Fuente, and Ina Hoeschele. Gene Network Inference via Structural Equation Modeling in Genetical Genomics Experiments. Genetics, 178(3):1763-1776, 2008.
[15] Judea Pearl. Causality. Cambridge university press, 2009.
[16] Min Ren and Dabao Zhang. Differential Analysis of Directed Networks. arXiv preprint arXiv:1807.10173, 2018.
[17] Phillippe Rigollet and Jan-Christian Hütter. High Dimensional Statistics. Lecture notes for course 18S997, 813:814, 2015.
[18] Da Ruan, Alastair Young, and Giovanni Montana. Differential analysis of biological networks. BMC Bioinformatics, 16(1):1-13, 2015.
[19] Mark Rudelson, Roman Vershynin, et al. Hanson-Wright inequality and sub-gaussian concentration. Electronic Communications in Probability, 18, 2013.
[20] Peter Schmidt. Econometrics. New York, Marcel Dekker, 1976.
[21] Damian Szklarczyk, Annika L Gable, David Lyon, Alexander Junge, Stefan Wyder, Jaime Huerta-Cepas, Milan Simonovic, Nadezhda T Doncheva, John H Morris, Peer Bork, et al. String v11: protein-protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets. *Nucleic Acids Research*, 47(D1):D607–D613, 2019.
[22] John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Mills Shaw, Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, Joshua M Stuart, Cancer Genome Atlas Research Network, et al. The Cancer Genome Atlas Pan-Cancer analysis project. Nature Genetics, 45 (10):1113, 2013.
[23] James West, Ginestra Bianconi, Simone Severini, and Andrew E Teschendorff. Differential network entropy reveals cancer system hallmarks. Scientific Reports, 2, 2012.
[24] Peng Zhao and Bin Yu. On Model Selection Consistency of Lasso. The Journal of Machine Learning Research, 7:2541-2563, 2006.
[25] Xin Zhou and Xiaodong Cai. Inference of differential gene regulatory networks based on gene expression and genetic perturbation data. Bioinformatics, 36(1):197-204, 2020.
[26] Hui Zou. The Adaptive Lasso and Its Oracle Properties. Journal of the American statistical association, 101(476):1418-1429, 2006.