Pruning neural network models for gene regulatory dynamics using data and domain knowledge
Authors: Intekhab Hossain
Jonas Fischer
Rebekka Burkholz
John Quackenbush
Venue: NeurIPS 2024 poster (2024)
Keywords: neural network pruning
sparsification
domain knowledge
gene regulation
Primary area: machine_learning_for_healthcare
TL;DR: We leverage domain knowledge to inform neural network pruning, thereby obtaining interpretable models that align with known biology

Abstract: The practical utility of machine learning models in the sciences often hinges on their interpretability. It is common to assess a model's merit for scientific discovery, and thus novel insights, by how well it aligns with already available domain knowledge - a dimension that is currently largely disregarded in the comparison of neural network models. While pruning can simplify deep neural network architectures and excels in identifying sparse models, as we show in the context of gene regulatory network inference, state-of-the-art techniques struggle with biologically meaningful structure learning. To address this issue, we propose DASH, a generalizable framework that guides network pruning by using domain-specific structural information in model fitting and leads to sparser, better interpretable models that are more robust to noise. Using both synthetic data with ground truth information, as well as real-world gene expression data, we show that DASH, using knowledge about gene interaction partners within the putative regulatory network, outperforms general pruning methods by a large margin and yields deeper insights into the biological systems being studied.

Pruning neural network models for gene regulatory dynamics using data and domain knowledge
Intekhab Hossain" 8
Jonas Fischer"
Department of Biostatistics
Dep. for Computer Vision and Machine Learning
Harvard T.H. Chan School of Public Health
Max Planck Institute for Informatics
Boston, MA 02115
Saarbrücken, Germany
shossaingg. www.ara..adee... edu
j jonas lonas.ffscheee. www.safrreeeermpipipie. mpg.
Rebekka Burkholz'
John Quackenbush
Helmholtz Center CISPA for Information Security Saarbrüccen, Germany burkholz@cispa.dee de
Department of Biostatistics
Harvard T.H. Chan School of Public Health
Boston, MA 02115
j hwwphapp....aaaae.e.....
Abstract
The practical utility of machine learning models in the sciences often hinges on their interpretability. common to assess model's merit for scientific discov- ery, and thus novel insights, by how well it aligns with already available domain knowledge-a dimension that is currently largely disregarded in the comparison of neural network models. While pruning can simplify deep neural network ar- chitectures and excels identifying sparse models, as we show in the context of gene regulatory network inference, state-of-ther oott--f-t techniques struggle with bio- logically meaningful structure learning. To address this issue, we propose DASH generalizable framework that guides network pruning by using domain-specinf structural information in model fitting and leads to sparser, better interpretable models that are more robust to noise. Using both synthetic data with ground truth information, as well as real-world gene expression data, we show that DASH, using knowledge about gene interaction partners within the putative regulatory network, outperforms general pruning methods by large margin and yields deeper insights into the biological systems being studied.
Introduction
With ever-growe wer-growing neural network architectures encouraged by the success of overparametrization, with over trillion parameters in single model such as GPT4 there is a similarly growing demand for sparser, more parameter-efficient neural networks that are more frieuncceerrillyyy and interpretable. The Lottery Ticket Hypothesis (LTH) provides an empirical existence proof sparse, trainable network architectures [18), that eventually achieve similar performance their dense counterpart. Subsequent work introduced structured pruning approaches, facilitating group-wise neuron- B32 62
channel-sparsity [34] 25 (26), which are, however, focused on the structure of the architecture design, aiming for better alignment with hardware implementations to eliminate operations, rather than structure that reflects relevant domain information. Especially for scientific discovery,
&oode is available at htthttttt///giii //t::bb.b/tubbb..// com homm//uacceenbusllaabDAA.5 These authors contributed equally to this work. IH is currently employed by Analysis Group. Inc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In the aligmment of learned structure with such domain knowledge is, however, essential for interpretability. as only then the model represents meaningful domain- relevant relations. Such problem settings often occur for example in physies or biology where a learned model should give an explanation to be able to form hypothesis. This poses the question: How can we select among multiple predictive models and promote the search for meaningful neural network structures?
To guide the learning process, we argue that we need additional problem-specific structural informa- tion, and should leverage any available and reliable- domain knowledge. One of the fundamental tasks of molecular biology is to understand the gene regulatory dynamics in health and disease. Gene regulatory dynamics describe the changes of the expression of gene- generation of small copies of DNA segment that can serve among others as blueprint for proteins- dependent on other regulatory factors such as transcription factors, which are proteins that bind next to the gene (DNA segment) to modulate its expression. Yet, the exact dynamies are far from understood and changes in these dynamics can be drivers for diseases such as cancer. As such, improving an understanding of the mechanics behind these dynamies increases the understanding of the disease and can ultimately inform therapy design. This problem setting of estimating gene regulatory dynamics requires high interpretability, as an understanding of the true biological mechanics- relationship between generating
regulatory factors and gene's s expression-is needed, as well as sample-efficiency, time-course data even for a few patients is extremely expensive. While models to estimate gene regulatory dynamies have been suggested [14 I 60 227,, none of these is particularly sparse or interpretable. We propose new approach of network sparsification that guides pruning by domain knowledge implemented for neural model for estimating gene regulatory dynamics, which yields networks that are very sparse, align with underlying biology, while accurately predicting dynamics.
This idea of prior-informed or domain aware pruning is at the heart of this paper. In particular, we propose DASH (Domain-A A ware Sparsity Heuristic, Fig. [T. an iterative pruning approach that scores parameters taking into account structural Noomain-knowlede
With DASH, it is possible to control the level of prior information taken into account for pruning and it automatically finds an optimal sparsity level aligned with both the prior and the data. Considering the task of estimating gene regulatory dynamics, we first show in synthetic experiments that DASH generally outperforms standard (task- and architecture agnostic) pruning as well as task-specific pruning approaches. On real data with a reference gene regulatory network (GRN) derived from gold standard biological experiments, show that DASH better recovers the reference GRN and reflects more biologically plausible information. On recent single cell data on blood differentiation, we show that DASH, in contrast to existing work, identifies biologically relevant pathways that can be used to generate new insights and inform domain experts. We anticipate that our work serves both for future benchmarking on how well pruning approaches are in structure learning, as well blueprint for guided network pruning in fields where domain knowledge is readily available, such as in other hard sciences including physics or material science, where knowledge about variables, e.g., associations between atoms or molecules or equations relating quantities in a system, is available.
2 Related work
The Lottery Ticket Hypothesis (LTH) 18 provides an empirical existence proof of sparse, trainable neural network architectures. It conjectures that dense, randomly initialized neural networks contain subnetworks that can be trained in isolation with the same training algorithm that is successful for the dense networks. However, a strong version of this hypothesis (5]] which has also been proven theoretically [42, 48 46
I7 5 12, [5). suggests that the identified initial parameters are not only specific to the sparse structure but also the learning task and benefit from information about the larger dense network that has been pruned 47 Acknowledging this strong relation, other works have proposed to combine mask and parameter learning directly in continuous sparsification approaches [53] that employ regularization strategies that approximate LO penalties [53] 40
31 55].
following, we recap the key ideas behind the methods most relevant to ours.
Explicit pruning-based approaches
Magnitude pruning (MP) In magnitude pruning (MP) a neural network is trained and then (post-hoc) pruned to desired sparsity level by masking the corresponding proportion of smallest magnitude weights. This smaller, masked network is then further trained, reminiscent of fine-tuning [22].
2

---

91
dg1/ dt
91
1d11/d de
92
d dt
f= f(W)
92
d de
Data-centtic sparsification
d dt
d9n de
prior domain
f(W)
knowledge P
91
/ de
92
g(P)
( 1 f(W) 2 (PP)
dg2/ de
Domain-åware Sparsity Heurisicc (DASH)
dgu/
Figure I: DASH. A NN, here a neural ODE for gene regulatory dynamics, traditionally sparsified in data-centric way (top). Pruning is done based on data alone, the pruning score f is function of the learned weights W. Such sparsified models often do not learn plausible relationships in the data domain. We propose DASH (bottom), which additionally incorporates domain knowledge P into the pruning score Ω2 yielding sparse networks giving meaningful and useful insights into the domain.
More formally, start with fully connected neural network NNe with L layers parametrized by θ = {(W b) MP is performed after training is complete (i.e. post hoc). For a suitable threshold of y%, MP "prunes" the trained θ by setting the smallest (absolute value) p% of weights in @ to O. The choice of parameters to prune can either be made in an unstructured way, by choosing the the lowest p% across all (W") semi-structuredd, by pruning the lowest weights per layer, in a structured way such that, e.g., neurons with lowest p% average outgoing weight are pruned. Once pruning is complete the p%--sarse NN fine-tuned Iterative magnitude pruning (IMP) [18 suggest to alternate between training and magnitude pruning, iteratively sparsifying the network, date still of the most successful pruning strategies. In practice, a pruning schedule is used to iteratively sparsify O, until O' with desired target sparsity or predictive performance plateau is reached. Importantly, weights are reset to initial pre- pre-training values. either after each round of pruning or once after the target sparsity has been reached.
the data so that O' is learned appropriately.
Pruning with rewinding 52 have demonstrated that rewinding weights to an earlier training point-a compromise between fine-tuning of MP and reset to initialization of IMP- provides good performance, which also has been suggested in the context of Neural ODEs as SparseFlow [37].
Implicit penalty-based approaches
C-NODE (2
seek to reduce the overall number of input-output dependencies (i.c. paths of contri- bution from input neuron i to output neuron j) through (W') The approach can result both feature and weight sparsity in NeuralODEs.
Lo [40] incorporate a differentiable Ly norm regularizer term in the objective. implicitly prunes the network by encouraging weights to get exactly zero. The L, regularizer is operationalized using nog-negative stochastic gates which act as masks on the weights.
RathReg [I] innovatively combines the strengths of both C-NODE and the Ln approach promote both weight and feature sparsity in NeuralODEs. It uses stochastic gates similar to [Q] and add a C- NODE-inspired penalty terms that constrain the overall number of input-output paths by regularizing the probability of any path from input to output j being non-zero.
Modeling gene regulatory dynamics As application, we consider estimation of gene regulatory dynamics. Early work, such as COPASI 44 use fully parametric modeling approach that are limited in their prediction capabilities. With recent advances machine learning, tools such as Dynamo [50) PROB (57), and RNA-ODE 39 aim to learn regulatory ODEs using sparse kernel regression, Bayesian Lasso, and random forests, respectively. Leveraging high flexibility and performance of neural models, PRESCIENT (44 uses simple NN to learn regulatory ODEs, whereas tools such
3

---

Base Model
C-Nore
Pethiiigg DASH
ME (Duns)
ScorzoFlow
BisP (Durs)
100%
0.000
Baamnngu Delog/
Prodect eon error on heid-oo date (M6R8)
50%
0.002
ground
20%
0.004
20%
Sput bology
6
0.000
predictive
2.00
Denser
0,000
Spanser
20
233
214
25
231
20
2'
Number of remaining parometers after validation performance pistenurs
Number remaining parameters.
after validation performance plateaus
Figure 2: Results on simulated data. We visualize performance of pruning strategies in comparison original PHOENIX (baseline) in terms of achieved sparsity (x-axis) and balanced accuracy (y-axis) of the recovered gene regulatory network against the ground truth on the SIM350 data with 5% noise. Error bars are omitted when error is smaller than depicted symbol. indicate methods that leverage prior information. Top left is best: recovering true, inherently sparse biological relationships.
as Velo 9 and sctour 335 have variational autoencoder as backbone. The latest line of research 14 60 27 uses neural ordinary differential equations [8] However, key limitation of these methods the lack of interpretability arising from non-sparse dynamics that do not align with ground truth biology. Consequently, the induction of sparsity in gene regulatory ODEs has been an active area of research with C-NODE and PathReg as most recent advances [2, [,, the achieved sparsity levels are, however, not yet sufficient to capture the relevant biology.
3 Domain-aware pruning with DASH
While the above sparsification strategies have shown to perform well in various settings, the resulting models are often either not particularly sparse or do not reflect meaningful domain knowledge. We hypothesize that this is due to two reasons: (1) the difficulty of identifying a good sparse network and (2) the current focus on hardware-centee pruming of a model in terms of groups of neurons (layers, channels) over structural pruning reflecting task-specific knowledge. To overcome these problems, we suggest to ground the model search (here, the network training) with existing domain-speciff knowledge, which cases identifiability due to the introduced constraints and enables task-aware pruning to identify meaningful domain knowledge.
rather than task-centric pruning, valuing structural
In the following, we propose DASH (Domain-Award Sparsity Heuristic), an iterative pruning-bast based approach that accounts for prior knowledge by scoring parameters in terms of their alignment with this prior, and show its usefulness for a neural model for the inference of gene regulatory dynamics. We assume the domain knowledge to be given as input-output relations (e.g., known protein gene interactions in molecular biology), for which we want the network flow between any input and output to align with. Suppose our domain knowledge for task from this domain is given as real-valued relationship graph G = (V, E)), where nodes are relevant entities from the domain, e.g. genes or proteins, and edges are strength of association between these entities, c.g. evidence of association derived from literature or experiments. Examples for such relational information in case of protein gene associations can be derived from protein binding profiles [56]. For the rest of the paper, we will assume that this domain knowledge is given as matrix P E R Rixr, which encodes the strength of association between the inputs and r outputs for our task of interest, such as known proxies of protein gene interactions.. Intuitively, for one-layer neural network, we encourage pruning scores for (neural) network edge to be proportional to the corresponding edge in the prior knowledge graph G while still taking into account the data-specific knowledge, thus enabling learning

---

of new knowledge and robustness to wrong or missing information in the prior. We begin with this simple base case of task-aware pruning of a fully connected neural network with layer and extend to more layers below.
DASH for L. = 1. For a single layer NN, with & input and output neurons and corresponding weight matrix W E Ri*k, we compute non-negative pruning scores Q E R by leveraging the domain knowledge P E B In practice, we allow balancing between data-driven and prior-kooledde-driieen pruning, implemented through a convex combination of the learned weights W and prior domain knowledge P controlled by the parameter À [0, 1]. Alternating between training and pruning akin to Iterative Magnitude Pruning [18), we set the following pruning score during pruning phase:
(+) = (1 ( + A|PI,
where (()) represents the appropriately normalized matrix (details in Appendix B.2) of absolute weights as learned up to epoch t. We then prune the parameters in ( corresponding to the lowest absolute pt % of entries in (2(1), where Pt is the desired sparsity level at time given by a schedule.
DASH for L = 2. For two-layered NNs with weights W, E mx W2 E IR"x" i.e l inputs, r outputs, and hidden neurons, we consider knowledge about input-output relationships P E Brxk as before. We can additionally use further knowledge about input-input relationships C E TR In molecular biology this could be information about binding or interaction partners available in databases such as STRINGDB (58), which has also been employed to guide static gene regulatory network inference [6]. or co-regulators, derived from co-occurrence of proteins. Intuitively, we now project pruning scores for the first layer to the prior knowledge about input-input relations, encouraging closeness to this prior, while projecting the product of pruning scores of first and second layer to known input-output relations, reflecting the flow of information from input to output through these two layers. Given that W (1) represents how the k inputs are encoded by m neurons, and 52
the corresponding pruning scores, surmise that the matrix product 52 (()) 22 E k should approximately align with the prior knowledge C. Since solving (2( = C is not directly feasible we initialize _2 randomly, and resort to solving a recurrence relation version of problem, that is 52f /2 (t- (2 = C. Using the left and right pscudo-inverse to obtain solution to the above, defined as (Xnv = (XTX) xxxx)-1x0 -XX and PInv = XT(XXT) -1 respectively:
1 (1) == (1 - X1) ( + X PINVZ (1 cl.
PInvL (2s) (1) C encourages 22 1 (2 to iteratively align with C as increases (i.e. as training
progresses). With s2(4) fixed, can update scores 52.4² of the second layer parameters W(()) Since the product wr(+), (((+) E x represents the overall flow of information from inputs to outputs epoch i, we surmise that (22 (2()) E R should reflect P. We thus get
2 (t) = (1 X2)
+ A2 PInvg (2 (1)
Similar to the case for DASH for L = layer, we can now prune the parameters of w(()) and w based on the magnitude of pruning scores (2(4) and (tt)) respectively.
DASH for L > 2 For many interpretability-ccnnti tasks, including our application to gene regulatory networks, small architectures of L = 2 are common, as domain experts are interested in understanding the exact flow of information through the network. Furthermore, we know that two-layer neural networks exhibit universal approximation (II). We however hypothesize that the technique of computing pruning scores by fixing those of preceding layers can be extended to a larger number L of fully connected layers and elaborate in App. B.6
Flexibility While the \ can be tuned using cross-validation (see App. B.4 we note that allows for flexibly encoding different pruning philosophies. Specifically, when X = 0 VL. DASH corresponds to SparseFlow, and when X = VI, DASH represents fully prior-based sparsification (which we term "BioPrunc" and consider as experimental bascline).
5

---

Table 1: Synthetic data results. We give model sparsity, balanced accuracy with respect edges in the ground truth gene regulatory network, mean squared error of predicted gene regulatory dynamics on the test set, and number of epochs (till validation performance plateaus) proxy of runtime. v is used to indicate methods that leverage prior information. Results are on SIM350 data with 5% noise.
Strategy (V = prio--infrrme))
Sparsity(%)
Bal. Acc.(%)
MSE (10 (10-3)
None/Baseline [27
7.3 0.1
51.8 0.03
3.0 = 0.4
Lo 40
33.8 + 4.7
55.0 + 0.5
8.5 + 1.0
Penalty- based
C-NODE 2
6.2 0.5
55.9 + 0.1
2.8
PathReg [1]
56.5 1.5
11.910
8.0
(implicit)
PINN [27
9.9
5.7
2.5
DS
92.8 0.3
0.5
0.5
IMP 18
81.9 6.6
6.77
4.7
Pruning-
Iter. SynFlow
79.3 1.2
58.4
7.0 2.1
based
SparseFlow 37
96.0 0.01
1.5
3.6 0.6
(cxplicit)
BioPrune (Ours, see
83.5 1.9
8.8
2.6
DASH (Ours)
V
94.6 1.2
90.7
2.4
Hybrid
PINN MP (Ours)
87.0 + 0.01
82.4
2.3 + 0.3
4 Task-aware pruning for sparse gene regulatory dynamics
Perhaps one of the most interesting applications of Machine Learning is in the field of Molecular Biology with the goal understanding human health and disease. A central mechanisms humans is the process of gene expression in each cell. There, copies of short segments of our genome are produced. These copies are among other things the blueprint for the production of different proteins, which are needed virtually everywhere in our bodies. If this tightly regulated process of gene expression goes wrong, for example because of a mutation in our genome, this can have profoundly bad effects, such as in the case of cancer. As such, studying this process is of great interest to understand and improve human health and discover new therapeutic targets.
Here, we consider the task of predicting the regulatory dynamics of gene expression. To be able understand the model and transfer it to clinical practice, interpretability is key. The most recent developments in modeling gene regulatory systems allow to model actual (temporal) regulatory dynamics, but require complex models, such as NeuralODEs, that hinder interpretability. While state-of-the-ae results are now achieved with shallow architectures [27] that are more tractable than deep, heavily perepaarmeteri networks, these models still encode information across many thousands of weights and we show experimentally that such information does not reflect true biology well. In fact, true gene regulatory networks and hence their underlying dynamics are inherently sparse [6]. This sparsity should be properly reflected by neural dynamics models. The PHOENIX NeuralODE model will serve as our base model for applying sparsification strategies and we show that pruning aligned with prior domain knowledge improves interpretability as well as quality of inferred (new) knowledge.
In nutshell, given a time series gene expression sample for k genes, PHOENIX uses NeuralODEs to construct the predicted trajectory between gene expression (t) 12 (inputs) time to any future expression (f+++1) (outputs), by implicitly modeling the RNA velocity (dg/dt). PHOENIX uses biokinetics-inspired activation functions separately model additive and multiplicative co- regulatory effects. The trained model encodes the ODEs governing the dynamics of gene expression, which can directly extracted for biological insights. We apply DASH PHOENIX and give a brief review of the PHOENIX architecture in App. 3..10land a detailed account on how to apply DASH to this architecture in App- B.IT Next, we provide experiments on synthetic and real data showing the ad advantages of informedf pruning the task of predicting gene-regulatory dynamics.

---

X0
730
0.21
0:11
0.36
0.16
203
305
26 200 202
150
3/6
26 200
189 88
286
120
163
20
206
200
303
24
230
10%
15
30
300
0.54
0.39
0..0
0.51
0.24
0.23
302
200
340
More A21.
Figure 3: Reconstruction of ground truth relationships. Estimated effect of gene 9j (X-axis) on the dynamics of gene g, (y-axis) in SIM350 for different levels of noise (rows). Ground truth is given on the left, our suggested approach and baselines (DASH, BioPrune, and PINN+MP) the right with mean squared error between inferred regulatory relationships and ground truth in purple.
5 Experiments
For evaluation we consider synthetic data from an established simulator tool [4], as well as real world data of gene expression from breast cancer tissue [13, from yeast with synchronized cell cycle [49] and from human bone marrow [.. case of synthetic data, we use the ground truth regulatory system from the generating model for validation. For breast cancer and yest cell cycle data we use additional experimental data (ChIP-seq) from the corresponding studies, which are independent gold- standard biological experiment measuring sample-specific TF-gene interactions, to evaluate inferred regulatory relationships. We measure the correctness a GRN learned by a model (see App. 10.4) in terms of balanced accuracy, which measures whether an edge is correctly reconstructed weighted by the sparsity of the aforementioned ground truth graph. To evaluate predictive performance for 6% hold-out test set for breast cancer and one of the biological replicates hold out from training for the yeast data. As prior knowledge we leverage general information of transcription factor binding to gene promoter regions as prior information, which can be computed from binding motif matches with the corresponding genome (human respectively yeast). The result is a matching score that can be thresholded get a 0, 1-based matrix encoding which (TF-encoding) gene has relationship with which other gene. We follow the approach of Guebila et al. [3] to obtain matrix P. As prior C. we use the STRING database (58), which gives a general (i.c., not tissue- specifi) graph of pootein-protcin interaction. Here, we use the interactions based on experimental evidence only and employ cutoff of get a binary adjacency matrix. (for more details, see App. BB3)..
real data, we
To compare pruning strategies, we consider the PHOENIX model as a basis, which is the state-of-the- art NeuralODE for estimating gene regulatory dynamics (27] and provide an ablation on a standard MLP architecture (see App. Tab. 7 App. Tab. [0] and App 12). We compare DASH against the PHOENIX model without additional pruning performance reference, and suggest two simple yet powerful baselines, which is post-hoc magnitude pruning of weights followed by finetuning (PINN+MP), and BioPrune, fully brier-based pruning (cf. Sec. 33. From the literature, we consider Lo-regularized pruning [40], C-NODE (2), and PathReg M. which have been recently proposed for the inference of sparse gene-regulatory relationships, PHOENIX with biological regularization 27). and dynamic sparse training (DST) (38. all of which are implicit pruning approaches. We further consider explicit, iterative score-based pruning approaches including Iterative Magnitude Pruning (IMP) [8], the flw-based model-agnostic pruning method SynFlow [59], and the flow-based Neural ODE pruning SparseFlow 377. We tune hyperparameters, including À for DASH, on a validation set. Unlike other methods
(e.g., Lo) DASH does not prolong the runtime of training much. A common pruning schedule where pruning scores are computed once every 10 epochs only increases runtime by <2% for the full model fitting process.
7

---

Simulated gene regulatory systems We simulate gene expression time-series data from fixed dynamical system. the ground truth was thus known (see App. B.I. In short, we adapt SimulatorGRN 4 to generate noisy time-series expression data from two synthetic gene regulatory systems (SIM350 and SIM690, consisting of 350 and 690 genes, respectively). We split trajectories into training (88%), validation (6% for tuning X), and testing (6%).Wo evaluate all methods in terms of achieved sparsity and MSE of predicted gene expression values on the test set (App. B.2 B.4) and investigate biological plausibility by calculating balanced accuracy of regulatory relationships extracted from the PHOENIX model (for details, see App. [3.5 (.5]B.10.41 [.104)), We here report the results for the data of 350 genes and 5% noise, noting that results are consistent across different noise levels and with more number of genes (see App. Sec. A.I).
A general trend across all experiments that aligns with our initial motivation is that dense models (sparsity < 50%) have a significantly worse reconstruction of the underlying biology - the ground truth GRN than sparse models (sparsity > 80%) (see Fig. 2 Furthermore, we see that DASH retrieves not only among the sparsest networks, but also reflects the underlying GRN best across all methods, outperforming comparably sparse IMP by about 20 percentage points accuracy in different settings, even with decrease in quality of the prior (see sensitivity analysis in App. Tab. 6). Due to the prior-informed structured pruning, it is able to occupy the sweet spot of highly sparse at the same time biologically meaningful models.
Consistent with the literature, PathReg outperforms Ly as well as C-NODE in terms of sparsity [. we additionally find evidence that also delivers more biologically meaningful results. Yet. IMP as well as prior-informed pruning approaches outperform PathReg by large margin. The MSE of predicted gene expression of DASH among the best, within one standard error of the best overall method. The only better approach is our suggested baseline, a combination of posthoe magnitude pruning of PHOENIX combined with additional finetuning MMIN which is, however, impractical as requires to train and prune many PHOENIX models along grid of sparsity levels (see App. B.9.3).
Visualizing the estimated against ground truth regulatory effects (i.e., functional relationships between variables), we observe that DASH captures the effects much better than competitors (see Fig. (3) Virtually all existing approaches discover spurious regulatory effects, whereas prior-informed pruning identify the main regulatory effects correctly. Moreover, with increasing levels of noise in the simulation, we observe that both BioPrune as well as PINN+MP start finding spurious dependencies, while DASH still recovers the overall structure well. While not perfect, as seemingly there are more dependencies than in the sparse ground truth, potentially introduced by correlations between features, DASH provides a sparse estimation of regulatory effects that most closely resembles the ground truth relationships among existing work.
Pseudotimee-rddeeed breast cancer samples To investigate the performance of DASH on real data, we consider gene expression measurements from a cross-sectional breast cancer study [3] This data of 198 breast cancer patients with measurements for 22000 genes has been preprocessed and ordered in pseudotime [57], which we use as basis for our experiments (cf App. Across methods, we observe that implicit sparsification methods generally perform poorly terms of sparsity and accuracy of recovered relationships (see Tab. 2 While pruning-based sparsification approaches achieve greater sparsity and performance in predicted gene expression, with SparseFlow reaching the highest sparsity (95.7%) among all methods, the recovered biological relations are not better than random chance, which renders the underlying models useless for scientific discovery. DASH in contrast finds a comparably sparse network (92.7% sparsity) while having top of the line performance in terms of test MSE and high alignment with true biology (95.7% balanced accuracy). For this particular dataset, we observe that DASH primarily builds on the prior knowledge, not surprisingly performing similarly as BioPrune, which is our suggested bascline pruning approach taking only the prior into account. We will see for other real-world data that this weight of domain knowledge is highly task-specific and BioPrune yields sub-optimal results on different data.
To better understand whether the inferred gene regulatory dynamics align with meaningful biology, we additionally perform pathway analysis (see A... Such pathway analysis are a standard approach for domain experts to distill information for example for therapeutic design. The genes that show the highest impact on the dynamics within the derived model are tested whether they enrich in specific higher level biological pathways. For the top-20 most significantly enriched pathway per model (App. Fig. 5), we observe that contrast to prior-informed methods, the existing pruning approaches show only very few significant pathways, consistent with our quantitative results on
8

---

Table 2: Results on breast cancer and yeast data. Balanced accuracy is based on reference gold standard experiments (transcription factor binding ChIP-seq) available for this data. DASH found optimal X-values of (0.995, 0.95) respectively (0.75. 0.75) for breast cancer and yeast. marks our suggested baselines and method, marks methods that use prior information for sparsification.
Data
Breast cancer in pseudotime
Yeast cell cycle
Strategy
Sparsity
Bal. Acc.
MSE (10-5)
Sparsity
Bal. Acc.
MSE (10-2)
None/Baseline
0.03%
49.99%
7.78
0.10%
49.87%
4.84
Lo
10.77%
50.15%
7.90
34.43%
48.43%
5,33
C-NODE
11.20%
50.01%
8.06
10.89%
50.04%
4.87
PathReg
14.09%
50.24%
7.92
12.09%
50.11%
5,35
PINN V
0.11%
49.99%
7.82
0.17%
49.93%
5.77
DST
67.02%
50.42%
7.78
77.80%
49.92%
5.18
IMP
36.02%
50.34%
7.77
83.22%
49.99%
5,46
Iter. SynFlow
91.93%
49.37%
7.78
85.65%
49.57%
5,41
SparseFlow
95.70%
49.70%
7.76
95.22%
49.89%
5.38
BioPrune
93.44%
95.67%
7.80
94.69%
79.23%
5.94
DASH
92.71%
95.69%
7.76
97.18%
88.43%
5.27
PINN MP *,
92.00%
54.02%
7.79
95.01%
55.39%
6.09
inferred regulatory relations. Moreover, disease-relevant pathways such as TP53 activity or FOXO mediated cell death, both of which are highly relevant in cancer [8]] are only visible in models pruned with prior information. This provides evidence that pruning informed by biological prior recovers biological signals that are relevant the disease and which can not be picked up otherwise. Furthermore, we find Heme-signaling as pathway uniquely identified as relevant in our approaches (cf. App. Fig. 5). Heme as signaling molecule has key roles in the gene regulatory system [45), and turns out to have anti-tumor role in breast cancer specifically [19]. Subsequent approaches pharmaceutically targeting Heme signaling showed success B0), with one of the key regulators affected being Bachl. To suggest further targets for e.g. combination treatment, we hence examined the top-5 regulatory factors in terms of weights in our estimated gene regulatory dynamics. These factors include PBXI and FOXMI, for which a drug repurposing of existing compounds, such (54). could lead to potential new treatment for this specific cancer.
Yest cell-cycle data We next consider real data of synchronized yeast cell 49 (see App. B. for
training setup). We observe an overall trend similar to the breast cancer study in terms of achieved sparsity and balanced accuracy (cf. Tab. 2 with implicit sparsification methods generally finding significantly less sparse models and all methods that do not incorporate prior knowledge having inferred relationships that not better than random chance. For this data, however, DASH finds an optimal lambda value that incorporates more data-specific knowledge (A = 0.75) compared to the breast cancer study above. This shows the advantage of DASH over our BioPrune baseline model (prior-only pruning), as here we gain about 10% points in balanced accuracy over BioPrune for ChIP-seq validation data (23) and 2% points over BioPrune for an independent TF perturbation network curated to derive a "true" causal GRN [2D (see App. Tab, (8. We also retrieve a 3% points sparser model. Comparing inferred biological knowledge between BioPrune and DASH through pathway analysis, we see that DASH recovers more significantly enriched pathways related to cell cycle processes (cf. App. Fig. 6).
Cell differentiation in human bone marrow Lastly, we investigate the performance of DASH in an exploratory setting with single cell data of human bone marrow ordered pseudotime [ Here, we are interested in better understanding the gene regulatory dynamics of blood cell differentiation, process of hematopoietic stem cells specializing into cells taking over roles such as immune response (e.g., B- and T-cells). This process is called hematopoiesis. We follow the steps of (1 to first split samples (i.e., cells) into the three different lineages (paths of differentiation), and train separate models for each (see App. B.7 We will here focus the analysis of the Erythroid lineage.

---

As before. DASH yields highly sparse (95%) networks, the most sparse among all competitors (App. Tab. D. IMP shows similarly strong sparsification as DASH while PathReg achieves much less sparsity (1 (14%). In terms of performance, all methods achieve similar MSE of predicted gene expression dynamics on the test set, meaning that even though much sparser, both DASH and IMP predict equally well as an order of magnitude more dense network. For this data, there are no gold-standard experiments for regulatory relations available, we hence focus on analysing the network topology. From the literature, we would expect sparser networks to be better align with biology [6]. DASH indeed shows the lowest out-degree in the inferred regulatory network, less than half of what IMP recovers. PathReg shows order of magnitude larger average out out-degree. To confirm the biological plausability, we again do a pathway analysis. DASH seems to find significant enrichment in biologically relevant pathways (App. Fig. 7 that can directly be linked to hematopoiesis, such as heme signaling or RUNXI regulates differentiation of hematopoietic stem cells, which neither BioPrune nor SparseFlow- the only other method yielding a proper sparse model- recover.
6 Discussion & Conclusion
We considered the problem of identifying sparse neural networks in the context of interpretability with a focus on the application to gene regulatory systems modeling. In domains such as biology and contexts when the true underlying systems are sparse, interpretabillity is key for experts, rendering the use of the common prerpaaraetrrieed and complex neural network architectures difficult. Although NNs do not directly encode e.g. the regulatory relationship between genes its deep architecture is necessary to model complex functional relationships while ensureing stable learning. Yet, we can ensure that Recent advances in neural network pruning, such as those around the Lottery Ticket Hypothesis [88, promise sparse and well-performing models, yet, hardness results prove finding optimally sparse models to be challenging 42), which also reflected by recent benchmarking results 16). Our experiments confirmed that general pruning strategies provide sub-optimal sparsity, moreover, the underlying biological relationships are not properly reflected in the model. We proposed to guide pruning by domain knowledge, leveraging existing prior information to improve the interpretability and meaningfulness of pruned models.
In case studies on gene regulatory dynamic inference, key task in molecular biology with high relevance in cancer research, we showed based on simulated as well as real world data that our method, DASH, in contrast to a wide range of totte-of-the-aee art methods, is able to recover neural networks that are both very sparse and at the same time biologically meaningful, allowing for direct extraction of a sparse gene regulatory network. On real data, DASH not only better aligns with gold-standard experimental evidence of regulatory interactions, but also uniquely reflects the data- specific biological pathways, which can be used by domain experts to generate new insights.. It thus serves as proof of concept that in critical domains, where interpretability is essential and domain knowledge exists, pruning can be heavily improved by alignment with prior knowledge. While our guided pruning approach is in principle agnostic to the type of neural network and task, we here focused on a specific case study that we deemed important. In the future, it would be interesting to apply DASH to different cases and domains, including other biomedical tasks, but also to physics or material sciences, where interpretability is also key and domain knowledge exists in the form of physical constraints and models. For any application, an important consideration to apply DASH is on the one hand the availability of prior knowledge, but on the other hand its quality; while show here that even with incomplete and noisy prior knowledge we receive good results, factually wrong priors could steer the solution towards wrong model. We hence assume that DASH will be of primary use in classical hard sciences mentioned above, with priors that stood the test of time over several decades. Another line of future work includes different architectural designs, such as convolution or attention mechanisms, where input-output relationships are less straightforward to project across several layers.
In summary, we make a case for pruning informed by domain knowledge and provide evidence that such approaches can massively improve sparsity along with domain specific interpretability.
RB received funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101 1 16395 SPARSE-MIL. JQ was supported by a grant from the US National Cancer Institute (R35CA220523) and additional funding from the National Human Genome Research Institute (301H.
Acknowledgements
10

---

References
[1] Hananch Alice, Till Richter, Mikhail Solonin, Ignacio Ibarra, Fabian Theis, and Niki Kilbertus. Sparsity in continuous-depth neural networks. Advances in Neural Information Processing Systems, 2022.
[2] Hananch Alice, Fabian J Theis, and Niki Kilbertus. Beyond predictions in neural odes: cation and interventions. arXiv preprint arXiv:2106. 12430, 2021.
Identifi-
[3] Marouen Ben Guebila, Camila Lopes-Ramos, Deborah Weighill, Abhijeet Rajendra Son- awane, Rebekka Burkholz, Behrouz Shamsaei, John Platig, Kimberly Glass, Marieke I. Kuijjer, and John Quackenbush. GRAND: database of gene regulatory network models across human conditions. Nucleic Acids Research,
09 2021.
[4] Dharmesh D Bhuva, Joseph Cursons, Gordon K Smyth, and Melissa J Davis. Differential
co-expression-basee detection of conditional relationships in transcriptional data: comparative analysis and application breast cancer. Genome biology, 20(1):1-21, 2019.
[5] Rebekka Burkholz. Most activation functions can win the lottery without excessive depth. In Advances in Neural Information Processing Systems, 2022.
[6] Daniel M Busiello, Samir Suweis, Jorge Hidalgo, and Amos Maritan. Explorability and the origin of network sparsity in living systems. Scientific reports, 7(1): 2017.
[7] Yang Cao, Shengtai Li, Linda Petzold, and Radu Serban. Adjoint sensitivity analysis for differential-algeeeiie equations: The adjoint dae system and its numerical solution. SIAM journal on scientific computing, 24(3): 1076))0776-189, 2003.
[8] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[9] Zhanlin Chen, William C King, Aheyon Hwang, Mark Gerstein, and Jing Zhang. Deepvelo: Single-cell transcriptomic deep velocity field learning with neural ordinary differential equations. Science Advances, 8(48):cabg3745, 2022.
[10] Jeanne Chèneby, Marius Gheorghe, Marie Artufel, Anthony Mathelier, and Benoit Ballester. Remap 2018: an updated atlas of regulatory regions from integrative analysis of dna-binding chip-seq experiments. Nucleic acids research, )6266)222675 2018.
[11] George Cybenko. Approximation by superpositions of a sigmoidal function. Math. Control. Signals Syst., 2(4):303-314, 1989.
[12] Arthur da Cunha, Emanuele Natale, and Laurent Viennot. Proving the lottery ticket hypothesis for convolutional neural networks. In International Conference on Learning Representations, 2022.
[13] Christine Desmedt, Fanny Piette, Sherene Loi, Yixin Wang, Françoise Lallemand, Benjamin Haibe-Kains, Giuseppe Viale, Mauro Delorenzi, Yi Zhang, Mahasti Saghatchian d' Assignies et al. Strong time dependence of the 76-gene prognostic signature for node-negative breast cancer patients in the transbig multicenter independent validation series. Clinical cancer research, 13 (3113:3207-332 2007.
[14] Rossin Erbe, Genevieve Stein-O' Brien, and Elana J Fertig. Transcriptomic forecasting with neural ordinary differential equations. Putterns, 4(8), 2023.
[15] Damien Ferbach, Christos Tsirigotis, Gauthier Gidel, and Joey Bose. A general framework for proving the equivariant strong lottery ticket hypothesis. In International Conference Learning Representations, 2023.
[16] Jonas Fischer and Rebekka Burkholz. Plant 'n' seek: Can you find the winning ticket? In International Conference on Learning Representations, 2022.
[17] Jonas Fischer, Advait Harshal Gadhikar, and Rebekka Burkholz. Towards strong pruning for lottery tickets with non-zero biases. arXiv preprint arXiv:2110.11150, 2021.
11

---

[18] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse. trainable neural networks. In International Conference on Learning Representations, 2018.
[19] Norberto Ariel Gandini, Eliana Noelia Alonso, María Eugenia Fermento, Marilina Mascaró, Martín Carlos Abba, Georgina Pamela Coló, Julián Arévalo, María Julia Ferronato, Josefina Ale- jandra Guevara, Myriam Núñez, Alejandro Carlos Pichel, Pamela nd Curino, and María Marta Facchinetti. Heme oxygenase- has an antitumor role in breast cancer. Antioxidants & Redox Signaling, 30(18):2030-204 2019.
[20] Kimberly Glass, Curtis Huttenhower, John Quackenbush, and Guo-Cheng Yuan. Passing messages between biological networks to refine predicted interactions. PLOS ONE, 8(5)e64432, 2013.
[21] S. R. Hackett, E. A. Baltz, M. Coram, J. Wranik, G. Kim, A. Baker, M. Fan, D. G. Hendrick- son, M. Berndl, and R. S. Mclsaac. Learning causal networks using inducible transcription factors and transcriptome-wide time series. Mol Syst Biol, 16(3):99174 Mar 2020.
[22] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 2015.
[23] Christopher T Harbison, D Benjamin Gordon, Tong Ihn Lee, Nicola J Rinaldi, Kenzie D Macisaac, Timothy W Danford, Nancy M Hannett, Jean-Bosco Tagne, David B Reynolds, Jane Yoo, et al. Transcriptional regulatory code of a eukaryotic genome. Nature, 9770494))99-104 2004.
[24] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[25] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In International Joint Conference Artificial Intelligence, 2018.
[26] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In IEEE International Conference on Computer Vision (ICCV), 2017.
[27] Intekhab Hossain, Viola Fanfani, Jonas Fischer, John Quackenbush, and Rebekka Burkholz. Biologically informed neuralodes for genome-wide wide regulatory dynamics. Genome Biology, 25(127), 2024.
[28] Y. Jiramongkol and E. W. Lam. FOXO transcription factor family in cancer and metastasis. Cancer Metastasis Rev, 39(3):681-700 Sep 2020.
[29] W Evan Johnson, Cheng Li, and Ariel Rabinovic. Adjusting batch effects in microarray expression data using empirical bayes methods. Biostatistics, 8(1):118-177, 2007.
[30] Pritpal Kaur, Shreya Nagar, Madhura Bhagwat, Mohammad Uddin, Yan Zhu, Ivana Vancurova, and Ales Vancura. Activated heme synthesis regulates glycolysis and oxidative metabolism in breast and ovarian cancer cells. PLOS ONE, 16(11):1-19, 11 2021.
[31] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, 2020.
[32] Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In IEEE Conference Computer Vision and Pattern Recognition, 2016.
[33] Jeffrey T Leek, W Evan Johnson, Hilary Parker, Andrew E Jaffe, and John D Storey. The package for removing batch effects and other unwanted variation in high-throughpe experiments. Bioinformatics, 28(6):882-88) -0338888288888 2012.
[34] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
[35] Qian Li. sctour: deep learning architecture for robust inference and accurate prediction of cellular dynamics. Genome Biology, 24(1):1-33, 2023.
12

---

[36] Arthur Liberzon, Chet Birger, Helga Thorvaldsdótti, Mahmoud Ghandi, Jill P Mesirov, and Pablo Tamayo. The molecular signatures database hallmark gene set collection. Cell systems, 166):417-425, 2015.
[37] Lucas Liebenwein, Ramin Hasani, Alexander Amini, and Daniela Rus. Sparse flows: Pruning continouu--eptth models. Advances in Neural Information Processing Systems, 2021.
[38] Junjie Liu, Zhe Xu, Runbin Shi, Ray C. C. Cheung, and Hayden K.H. So. Dynamic sparse train- ing: Find efficient sparse network from scratch with trainable masked layers. In International Conference Learning Representations, 2020.
[39] Ruishan Liu, Angela Oliveira Pisco, Emelie Braun, Sten Linnarsson, and James Zou. Dynamical systems model of rma velocity improves inference of single-cell trajectory, pseudo-time and gene regulation. Journal of Molecular Biology, 434(15): 2022.
[40] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through ln regularization. In International Conference Learning Representations, 2018.
[41] Malte D Luecken, Daniel Bernard Burkhardt, Robrecht Cannoodt, Christopher Lance, Aditi Agrawal, Hananeh Aliee, Ann T Chen, Louise Deconinck, Angela M Detweiler, Alejandro A Granados, et al. A sandbox for prediction and integration of dna, rna, and proteins in single cells. In Advances in Neural Information Processing Systems, 2021.
[42] Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning. 2020.
[43] C. Marvalim, A. Datta, and S. C. Lee. Role of p53 in breast cancer progression: An insight into p53 targeted therapy. Theranostics, (0(4):1421-1) 2023.
[44] Pedro Mendes, Stefan Hoops, Sven Sahle, Ralph Gauges, Joseph Dada, and Ursula Kummer. Computational modeling of biochemical networks using copasi. Systems Biology, pages 17-59, 2009.
[45] S. M. Mense and L. Zhang. Heme: a versatile signaling molecule controlling the activities of diverse regulators ranging from transcription factors MAP kinases. Cell Res, 1668))681-69. Aug 2006.
[46] Laurent Orseau, Marcus Hutter, and Omar Rivasplata. Logarithmic pruning is all you need. In Advances in Neural Information Processing Systems, 2020.
[47] Mansheej Paul, Feng Chen, Brett W. Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dziugaite. Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask? In International Conference on Learning Representations, 2023.
[48] Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient. In Advances in Neural Information Processing Systems, 2020.
[49] T. Pramila, W. Wu, S. Miles, W. S. Noble, and L. L. Breeden. The Forkhead transcription factor Heml regulates chromosome segregation genes and fills the S-phase gap in the transcriptional circuitry the cell cycle. Genes Dev, 20(16):2266-22 Aug 2006.
[50] Xiaojic Qiu, Yan Zhang, Jorge D Martin-Rufing Chen Weng, Shayan Hosseinzadeh, Dian Yang, Angela N Pogson, Marco Y Hein, Kyung Hoi Joseph Min, Li Wang, al. Mapping transcriptomic vector fields of single cells. Cell, (35(4):690-11) 2022.
[51] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in randomly weighted neural network? In Conference on Computer Vision and Pattern Recognition, 2020.
[52] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
13

---

[53] Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsifi- cation. Advances in Neural Information Processing Systems, 2020.
[54] Yao-An Shen, Jin Jung, Geoffrey D. Shimberg, Fang-Chi Hsu, Yohan Suryo Rahmanto, Stephanie L. Gaillard, Jiaxin Hong, Jürgen Bosch, le-Ming Shih, Chi-Mu Chuang, and Tian-Li Wang. Development of small molecule inhibitors targeting pbx1 transcription signaling as a novel cancer therapeutic strategy. iScience, 24(11):103297 2021.
[55] Kartik Sreenivasan, Jy yong Sohn, Liu Yang, Matthew Grinde, Alliot Nagle, Hongyi Wang, Eric Xing. Kangwook Lee, and Dimitris Papailiopoulos. Rare gems: Finding lottery tickets at initialization. In Advances Neural Information Processing Systems, 2022.
[56] G. D. Stormo. Modeling the specificity of protein-DNA interactions. Quant Biol, 1(2):115-330 Jun 2013.
[57] X. Sun, J. Zhang, and Q. Nie. Inferring latent temporal progression and regulatory networks from cross-sectional transcriptomic data of cancer samples. PLOS Comput Biol, 1703):e1008339 Mar 2021.
[58] D. Szklarezyk, R. Kirsch, M. Koutrouli, K. Nastou, F. Mchryary, R. Hachilif, A. L. Gable, Fang. N. T. Doncheva, S. Pyysalo, P. Bork, L. J. Jensen, and C. von Mering. The STRING database in 2023: protein-protein association networks and functional enrichment analyses for any sequenced genome of interest. Nucleic Acids Res, (DD1):D33-DD444 Jan 2023.
(59] Hidenori Tanaka, Daniel Kunin, Daniel L.. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, 2020.
[60] Alexander Tong, Manik Kuchroo, Shabarni Gupta, Aarthi Venkat, Beatriz Perez San Juan, Laura Rangel, Brandon Zhu, John G Lock, Christine Chaffer, and Smita Krishnaswamy. Learning transcriptional and regulatory dynamics driving cancer cell plasticity using neural ode-based optimal transport. bioRxiv, pages 2023-03, 2023.
[61] Deborah Weighill, Marouen Ben Guebila, Camila Lopes- Sopes- Kimberly Glass, John Quack- enbush, John Platig, and Rebekka Burkholz. Gene regulatory network inference relaxed graph matching. AAAI Conference on Artificial Intelligence, 2021.
[62] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, 2016.
[63] YH Yang and AC Paquet. Preprocessing two-color spotted arrays. In Bioinformaties and Computational Biology Solutions Using R and Bioconductor, pages 49-69. Springer, 2005.
[64] Grace Hui Ting Yeo, Sachit D Saksena, and David K Gifford. Generative modeling of single-cell time series with prescient enables prediction of cell trajectories with interventions. Nature Communications, 12(1):3222, 2021.
[65] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, 2019.
14

---

A Supplementary results
A.1 Synthetic data
Base Modal
PIVN
DST
NODE
Rothiegg
MP (Durs)
IPP
Iter. SynFlow
Sparsellow
BloPrune (Ours)
DASH (Ours)
100%
0.000
(non-zero feattires vs ground truth biology)
Pradiction emror on held-out data (MSE)
90%
Balanced accuracy
80%
0.015
0.020
70%
0.025
60%
0.030
0.035
21
255
2/r
Number remaining pare meremeters after validation performance platnaus
Number remaining parameters after validation pertomance platnaus
Figure 4: SIM690 data with 5% noise. We visualize performance of pruning strategies in comparison to original PHOENIX (baseline) in terms achieved sparsity (x-axis) and balanced accuracy (y-axis) of the recovered gene regulatory network against the ground truth. Error bars omitted when error is smaller than depicted symbol. Checkmarks (V) are used to indicate methods that leverage prior information. Ideal models are in the top left quadrant; they recover the true, inherently sparse biological relationships.
Table 3: Simulation study results noise. We provide achieved model sparsity, balanced accuracy of inferred gene regulatory network, and MSE of predicted gene expression dynamics on test data at 0% noise level for SIM350 (the synthetic system 350 genes). * marks our suggested baselines and method, v marks methods that use prior information for sparsification.
Strategy
Sparsity(S))
Bal. Acc.(%)
MSE (10 3)
Epochs
None/Baseline 27
11.5 + 0.3
54.8 + 0.6
3.6 + 1.7
69 + 2
Lo 40
34.7 + 2.4
61.3 + 0.1
6.1 1.8
119 + 43
Penalty- based
C-NODE
10.7 0.2
6.1
1.9
213 5
PathReg [
59.7 + 1.5
6.
6.1
213 + 7
(implicit)
PINN 22
11.3 0.3
6.3
2.3 0.4
206 +
DST
94.3+0.. 0.5
7.2
1.4
216 86
IMP [18
8.11
1.7
4.1
25177
Pruning-
Iter. SynFlow 59
79.1
6..0
1.6
323 37
based
SparseFlow 37
9.3
7.78
2.5
195 10
(explicit)
BioPrune *,
8.5
8.5
3.8
94
DASH
92.6 + 1.2
91.1
1.9
164
Hybrid
PINN MP x,
90.0 0.01
89.8 0.3
2.5
1788 77
15

---

Table 4: Simulation study resulss 10% noise. We provide achieved model sparsity, balanced accuracy of inferred gene regulatory network, and MSE of predicted gene expression dynamics on test data at 10% noise level for SIM350 (the synthetic system of 350 genes). * marks our suggested baselines and method, marks methods that use prior information for sparsification.
Strategy
Sparsity(%)
Bal. Acc.(%)
MSE (10 3)
Epochs
None/Baseline
1.3 0.3
50.1 = 0.3
3.5 - 0.03
55 5
Lo 140
31.9 + 1.6
50.2 = 0.3
13.3
156 + 11
Penalty- based
C-NODE
7.2
50.5 0.3
36.6 12.5
189 11
PathReg 1
47.8 1.8
5.2
1.
2
(implicit)
PINN 22
2.9
5.3
5.9
211 1
DST 38
93.3 2.0
67.2 2.6
4.1
286 + 33
IMP 18
0.1
2.1
6.7
240
Pruning-
Iter. SynFlow (59
1.3
1.2
7.0
165
based
SparseFlow 37
8.0
6.0
5.1
142 13
(explicit)
BioPrune
1.3
8.3
0.3
67 18
DASH
90.8 4.7
4.2
5.3
154 + 4
Hybrid
PINN MP s, 92.0 0.01
83.8 = 0.7
4.1 1.8
1813 + 94
Table 5: Comparison of systems with different number of genes Ng. We provide achieved model spar- sity, balanced accuracy of inferred gene regulatory network, and MSE of predicted gene expression dynamics on test data at 5% noise level for SIM350 and SIM690. marks our suggested baselines and method, marks methods that use prior information for sparsification.
N
Strategy
Sparsity(%) (%)
Bal. Acc.(%)
MSE (10 3)
Epochs
None/Baseline
27
7.5 + 0.1
51.8 + 0.03
3.0 + 0.4
67 +
Lo 40
33.8 + 4.7
55.0 0.5
8.5
134 = 37
Penalty- based
C-NODE 2
6.2 0.5
55.9 + 0.1
2.8 0.6
214 + 4
PathReg
56.5 1.5
61.9
8.0
200 29
(implicit)
PINN 27
3.9+0..
5.7
2.2
160 +
DST
92.8
0.5
4.0
244
350
IMP 18
81.9 6.6
6..7
4.7
308 + 13
Pruning-
Iter. SynFlow [59]
1.2
0.6
7.0
271 + 38
based (explicit)
SparseFlow B37
96.0
7.5
0.6
220 =
BioPrune *,
83.5
0.8
2.6
79 +
DASH
94.6 + 1.2
90.7 0.4
2.4
192 + 24
Hybrid
PINN + MP
87.0 + 0.01
82.4 0.2
2.3
1721 + 50
None/Baseline 22
1.1
50.6 0.1
0.4
188 + 17
La
40
3.19
53.0 0.3
12.8 3.1
100 + 26
Penalty- based
C-NODE 2
4.3
5.2
24.4
210 +
PathReg
5.5
5.2
3.2
244 17
(implicit)
PINN 27
7.7
5.07
0.3
166 + 29
38
94.1
6.9
4.7
188 + 28
690
IMP 18
81.1
2.1
6.7
319 + 24
Pruning-
Iter. SynFlow 559
77.6
55.9
6.7 6.7 0.05
215 + 18
based
SparseFlow 37
95.8 + 0.3
1.5
6.1
205 + 10
(explicit)
BioPrune *,
80.8 + 4.3
1.7
5.9 + 2.4
71 23
DASH #,
93.9 0.01
0.2
4.3 0.5
178 =
Hybrid
PINN + MP *, V
87.0 + 0.01
75.9 + 0.2
3.7 + 0.2
1657 + 106
16

---

Table 6: Sensitivity of DASH to noise in prior. To understand the impact of the quality of the prior knowledge on the performance of DASH, we show results for different levels of prior corruption in the synthetic data (SIM 350). We keep expression noise constant at 0% understand the impact of prior corruption alone.
Strategy
Prior corruption
Sparsity(%)
Bal. Acc.(%)
MSE (10-3)
None/Baseline
Does not use prior
11.5
54.8
3.6
Lo
Does not use prior
34.7
61.3
6.1
C-NODE
Does not use prior
10.7
60.5
1.9
PathReg
Does not use prior
59.7
64.2
6.1
DST
Does not use prior
94.3
72.3
4.2
IMP
Does not use prior
86.1
63.2
4.1
Iter. SynFlow
Does not use prior
79.1
60.0
2.3
SparseFlow
Does not use prior
95.8
72.8
2.9
0%
11.3
60.3
2.3
PINN
20%
12.4
60.8
3.1
40%
11.2
60.6
2.7
0%
83.5
88.0
3.6
BioPrune
20%
80.9
81.5
7.6
40%
86.8
80.1
11.1
0%
92.6
91.1
1.9
DASH
20%
92,4
86.2
6.7
40%
85.9
79.5
Table 7: Prior-informed pruming on an MLP for simulated data, We compare sparsification strategies on PHOENIX base model and simple 2-layer MLP base model with ELU activations. We tested on SIM350 with 5% noise. Balanced Accuracy is included since ground truth regulatory structure is known.
PHOENIX base model
MLP base model (with ELU)
Strategy
Sparsity
Bal Acc
Test MSE (10-3)
Sparsity
Bal Ace
Test MSE (10 (10-3)
None
7.5%
51.8%
3,0
0%
50.0%
5.2
Lo
33.8%
55.0%
8.5
29.6%
51.2%
8.1
C-NODE
6.2%
55.9%
2.8
48.8%
50.3%
5.3
PathReg
56.5%
61.9%
8.0
47.1%
55.3%
8.2
DASH
94.6%
90.7%
2.4
89.4%
88.4%
3.8
17

---

A.2 Breast cancer data
regulation of tp53 activity through association With co factors- tyrosine catabalism interleukin processing transcriptional requla granuulppoles sumoylatidnn transchiption factors
stimulated transcription
foxo mediated transchitioo of cell death depes runx3 requlates wnt signa ing interleukin 4 and Interleuk
siona ino
heme sicna
regulates yapl mediated ated transcripton Clock
circad trapscriptional regulation oo oy runx2-
transcriptional regulation of wh te adipocyte differantia regulation of gene expression in beta cells mediated transcriptinn
cell cycle genes
myogenesis phase
runx2 regulates bone deve oomment regulation of gene expression in early pancreatic precursor cells regulation of lipid metabolism by poaralphe sumoyla lation
intrace receptors
insertion tail anchored proteins into the endopla endoplasmic reticul runx1 regu attes transcription estrogen dependent genes involved ocr regulation of gene expression by hypoxia inducible ractol
e
expression
inacivatioon of cdc42 and racl transcriptional regulation
testis differentia funx? regil ates osfegblat pluripotent
iifferentiation cells
transcriptional regulation gli proteins bind promoters hn responsivé genes to brompte transcriptin dermatan sulfate biasynthes sema4d mediated inhibition itioiioon of cell attachmen meneetane and migration nuclear recepto transcnition patnway
meco2 signaling transcription polipotioo factors
notch3. intrace (ulr domain regulates transcription activation of anterior hox genes hinthrrin development during farly Smmrrageness ptko expression eph ephrin mediated repulsion of cells clearance
mittrrranmitter ease
dopamine neurotransmitte ease cycle
SparsoFlow GNNDE Lo
PathReg
PINN
DST
BloPrune (Ours)
DASH (Ours)
PINN+MPP (Ours)
2-score =0
10
20
Figure 5: BRCA pathway analysis. We visualize the top-20 significant pathways for each method, showing the pathway z-score (x-axis) and indicate significant results after FWER correction (Bonfer- roni, p-value cutoff at .05) with
18

---

A.3
Yeast data
Traversing start control point of milolic cell (gu:0007089)
posilive regulation transcription involved in g3/s transition of
cycle (go:
g1/s transition of mitotic cell cycle (go:0000032)
regative regulation of mitotic cell cycle phase transition (go:1901991)
regulation of transcription involved in g2/m transition of mitotic cell cycle (go:0000117)
positive regulation of metaphase/onaphase positive reguletion of cell cycle (go:0045787)
tion of cycle (go:19021011)
regulation of transcription from ra polymerase promoter (g00000330000000033337
negative regulation of transcription from polymerase ii promoter (go:0000122)
positive regulation of transcription dna-templated (go:0443330
negative regulation transcription dna-templated (go:00458892)
positive regulation of transcription from ma polymerase promoter (go:004594]
negative regulation of pseudohyphal growth (go:2000221)
positive regulation of transcription from ra polymerase promoter by pheromones (go:0007329)
negative regulation of invasive growth in response to glucose limitation (go:2000218)
asparagine catabolic process (go:0006530)
negative regui ation of growth unicellular organism as thread of attached cells (go:0070785)
phosphate lon transport (go:0006817))
mitotic cytokinetic process (go:1902240)
negative regulation phosphorylation (go:0011336))
inorganic cation transmembrane transport (go:0098662)
chromosome condensation (go:0030261)
maturation of Isu-rrna from tricistronic mrne transcript (ssu-rma 5.8s rma Isu-rrna) (go:0000463)
hydrogen ion transmembrane transport (go:1902500)
prcton transport (go:0015992)
BioPrune
10
Figure 6: Yeast pathway analysis. We visualize the top-20 significant pathways for each method, showing the pathway z-score (x-axis) and indicate significant results after FWER correction (Bonfer- roni, p-value cutoff at .05) with
19

---

Strategy
Sparsity
Bal. Acc. (ChipSeq)
Bal. Acc. (TF Perturb)
MSE (10-3
None/Baseline
0.10%
49.87%
49.92%
4.84
Lo
34.43%
48.43%
49.28%
5,33
C-NODE
10.89%
50.04%
50.17%
4.87
PathReg
12.09%
50.11%
49.92%
5.35
PINN
0.17%
49.93%
50.01 %
5.77
DST
77.80%
49.92%
50.33%
5.18
IMP
83.22%
49.99%
48 48.45%
5,46
Iter. SynFlow
85.65%
49.57%
49.77%
5,41
SparseFlow
95.22%
49.89%
51.58%
5.38
BioPrune
94.69%
79.23%
64.50%
5.94
DASH V
97.18%
88.43%
66.79% %
5.27
PINN MP *,
95.01% %
55.39%
52.95%
6.09
Table 8: Balanced accuracies for experiments on yeast data. Balanced accuracy based on transcription factor binding ChIP-seq available for this data (23) and 2) A TF perturbation network created by Hackett et al. based on their TF perturbation experiments [21]. marks our suggested baselines and method, v marks methods that use prior information for sparsification.
20

---

A.4 Bone marrow data
Table 9: Results Hematopoesis data for the Erythroid lineage. We give sparsity of pruned model and test MSE on predicted gene expression dynamics. No reference gene regulatory network is available to compute the accuracy of the recovered network we hence resort reporting the average out-degree of nodes in the recovered network. DASH found an optimal X-value (0.80, 0.80). * marks our suggested baselines and method, v marks methods that use prior information for sparsification.
Strategy 0.25%
Sparsity
OutDeg
Test MSE (10 4)
None/Baseline
529
2.12
Lo
12.03%
518
2.14
C-NODE
1.33%
529
2.17
PathReg
13.94%
522
2.12
PINN
5.93%
529
2.14
DST
90.54%
328
2.23
IMP
73.00%
404
2.26
Iter. SynFlow
82.28%
350
2.18
SparseFlow
94.44%
123
2.12
BioPrune *,
87.09%
319
2.14
DASH
95.95%
54
2.12
PINN MP v
92.00%
218
2.14
heme signaling sumoylation intracellular rečeptors
transcriptional regulation of white adipocyte differentiatinn transcriptions regulation of granulopoiesis nuclear receptor transcription pathway
activation of anterior hox genes in hindbrain development during early embryogenesis
myogenesis
g0 and early gl
transcription of e2f targets under negative control by dream complex
runx1 regulates transcription genes invoivec in differentiation of hscs
signaling by retinoic acid
circadian clock
runx3 regulates wnt signaling
repression of wnt target genes
oncogene induced senescence
transcriptions regulation by ventx
dependent signaling in response to wnt mitochondrial iron sulfür cluster biogenesis
Sparse PathReg BloPrune (Durs) DASH (Ours)
2-score
Figure 7: Hematopoesis pathway analysis. We visualize the top-20 significantly enriched pathways on the Erythroid lincage. For each method (x-axis) we show the pathway z-score and indicate significant results after FWER correction (Bonferroni, p-value cutoff at .05) with
21

---

for Table 10: Prior-informed pruning on an MLP for bone marrow data. We compare sparsification strategies on PHOENIX base model and a simple 2-layer MLP base model with ELU activations. We tested on Erythroid lineage of the bone marrow data.
PHOENIX base model
MLP base model (with ELU)
Strategy Sparsity
Test MSE (10 (10-+)
Sparsity
Test MSE (10-4)
None
0.25%
2.12
0.00%
2.11
Lo
12.03%
2.14
8.41%
2.29
C-NODE
1.33%
2.17
2.55%
2.22
PathReg 13.94%
2.12
21.48%
2.15
DASH
95.95%
2.12
84.06%
2.13
22

---

B Supplementary methods
B.1 Synthetic data generation
The purpose of simulation based data is so that the the underlying dynamical system that produced the this gene expression was known. Do this end, we closely follow the steps outlined by the simulation pipeline provided by [27 to generate reliable synthetic time-series gene expression data from two ground truth gene regulatory networks G350 and Geo0 consisting of 350 and 690 genes, respectively.
The pipeline adapts SimulatorGRN 4 to generate from synthetic S. cerevisiae gene regulatory systems (SIM350 and SIM690, consisting of 350 and 690 genes respectively). For every noise setting {0%, 5%, 10%), ). the connectivity structure of each in silico system is used to synthesize 160 noisy expression trajectories for each gene across t E T = {0, 2, 2,3,75 7.9.. We split up the trajectories into training (88%), validation (6% for tuning A), and testing (6%). Since the average simulated expression value is z 0.5, adding Gaussian noise of (0, o²) using a E {0, 40 20
1 1 corresponds
roughly average noise levels of {0%, 5%, 10%).
B.2 Setup for model training
B.2.1
Model complexity
Since the number of genes i in each problem is different, the number of neurons m in PHOENIX's hidden layer is chosen to roughly scale with this it according the original paper [27]
SIM350: 350, 40
SIM690: = 690, m = 50
Bone marrow data: h 529, m = 50
Yeast data: h: 3551, 120
Breast cancer data: 11165, m 300
B.2.2 Initialization and optimizers
For initialization values for each of NN NN NNprode- NNEcombine. and NNIcombine as well as that of u.s we choose the default provided by the PHOENIX implementation (27. The ODESolver (dopri5) and optimizer (Adam) are also chosen as the PHOENIX defaults across all experiments.
B.2.3 Pruning details
We iterative pruning schedules that are initially very aggressive and then become much more gradual. We found this approach to achieve high sparsity without adversely affecting the training dynamics (and subsequently the validation performance).
SIM350: prune 70% epoch 3, and then 10% every 10 epochs
SIM690: prune 70% epoch 3, and then 10% every 10 epochs
Bone marrow data: prune 70% at epoch 3, and then 10% every 10 epochs
Yeast data: prune 90% at epoch 10, and then 10% every 20 epochs
Breast cancer data: prune 90% epoch 10, and then 10% every 20 epochs
Weight normalization for DASH pruning scores As described Section 3 the weight matrices of
PHOENIX need to be normalized ( (. ( | and ( in the formula for calculating
DASH pruning scores f2y, Pni Vx Vn. We perform the following normalizations:
For (!) Σ we simply normalize by taking elementwise absolute values of w-(!) and divid- ing all entries by the overall sum of absolute values.
23

---

For W I
we approach similarly, with the only modification that the weights are element-
wise exponentiated instead of clementwise absolute value, given that w (2) log-space.
IT operates on the
ForlU and UU we approach again similarly, with the important modification that the gene-specific multipliers (from Section [B.0.2) are row-wise multiplied into the weight matrices prior to normalization. This allows the effect of gene multipliers to appropriately be considered when performing pruning.
B.2.4 Learning rates
The learning rate used as the PHOENIX default of 10-3 We reduce the learning rate by 10% every 3 epochs, unless the validation set performance shows reasonable improvement. Importantly, we reset the learning rate back to 10 immediately after a pruning step is completed, thereby allowing the newly sparsified model to start learning with a higher learning rate.
B.2.5 Stopping criteria
We train for up to 500 epochs on an AWS c5. <5.9xlarge instance, where each epoch consisted of the entire training set being fed the model, preceded by any pruning step that prescribed by the pruning schedule. Training is terminated if the validation set performance fails to improve in 40 consecutive epochs. Upon training termination, we have obtained a model that has been iteratively sparsified to an extent that fails to improve the validation set performance. Hence this training procedure automatically finds an optimal sparsity level using the validation set.
B.3 Prior knowledge to obtain DASH pruning scores
As mentioned in Section 3 DASH can leverage prior matrices P and C' to inform its pruning score. We use the following our experiments:
SIM350 and SIM690:
for synthetic experiments choose P = 40 to be noisy/corrupted versions (see B. of the adjacency matrices of ground truth networks G350 and Geen to reflect that transcription factor binding to target genes can itself be noisy process in real life.
A A represents prior knowledge of an interaction existing between two genes, and represented interaction.
For C' we use the outer product coregulation. We again applied the corrupiionmmiscqeecicctttiii procedure from B.3.] so that C' also noisy.
4 (A*% T, to represent prior ledgledge of
Breast cancer data:
For the prior domain knowledge, we set P Wo. where Wo was a motif map derived from the human reference genome, for the breast tissue specifically, which we obtained through GRAND B. Wo is a binary matrix with Wo., E {0, 1} where I indicates a likely occurence of TF sequence motif in the promoter of the target gene, and hence indicating a putative interaction. More simply put, it indicates that whether there is (likely) binding interface for the protein close the target gene.
Based on information from the STRING database [58] we obtained protein-protein interaction matrix (PPI) which could use to operationalize our C' matrix, since PPI is a again a binary matrix that is suggestive of which transcription factors have combined (or coregulatory) effects. In nutshell, the STRING database is graph with proteins as vertices and knowledge about interactions between two proteins the graph specifying edges. We set an entry P.j to 1 if the experimental evidence score on the edge between protein i and j is larger than 0.6, and set P.j to otherwise.
Bone marrow data:
For the prior domain knowledge, we followed a similar strategy as the breast cancer analysis, and set P and C' based on the motif map and PPI matrix used in [61]. We
24

---

appropriately subsetted P and C' to only be limited to the A = 529 genes that were selected by the PathReg authors [T] in the analysis.
Yeast data:
For prior domain knowledge model we set P to reflect the regulatory network structure of motif map. The map is based on predicted binding sites for 204 yeast transcription factors (TFs) [23]. These data include 4360 genes with tandem promoters. 3551 1 of these genes are also covered on the yeast cell cycle gene expression array. 105 total TFs in this data set target the promoter of one of these 3551 genes. The motif map between these 105 TFs and 3551 target genes provides the adjacency matrix A of Os and 1s, representing whether not prior interaction is likely between TF and gene. We set C to be the PPI matrix used for the same data in the PANDA paper [20]. We appropriately subsetted C' to only be limited to the k = 3551 genes that were in the data.
B.3.1 Creating corrupted/misppeciiie prior models for synthetic data
For each noise level 0% E {0%, 5%, 10%] in our in silico experiments, we created a shuffled version of G350 (and similarly G690) where we shuffled 0%, of the edges by relocating those edges to new randomly chosen origin and destination genes within the network. This yielded the shuffled network G350 (and similarly G with corresponding adjacency matrix A 0%, We additionally performed sensitivity analyses using 0% f20%, 40%} to investigate the effect of even higher levels of prior corruption. We then used 4 obtain "corrupted" P and C described in B.3
B.4 Validation and testing
The choice of À = (Ai, A2) is important for optimally combining prior information with model weights. Hence we implement K K-fold cross validation approach choose Test set performance is measured as the mean squared error between predictions and held-out expression values in the test set.
B.5 Measuring biological alignment of sparsified models
To validate biological alignment trained and sparsified models, we extracted GRNs from each models (as explained in T0.44, and compared back to the validation networks. Specifically, once we extracted a GRN from the trained model, we looked at how well Os vs non-zeros in that network aligned with Os vs non-zeros in the validation network. Our comparison metric was balanced accuracy, which is the average of the true positive and true negative rates. The validation networks follows:
SIM350, SIM690: We used the ground truth networks G350 and G
Breast cancer data: We used ChIP-seq data from the MCF7 cell line (breast cancer) in the ReMap2018 database [10] to create validation network of TF-target interactions.
Bone marrow data: As also noted by [1. validation network was available, so resorted to the pathway analysis.
Yeast data: We used two kinds validation networks
ChIP-seq data 23 create network of Ta-tare interactions, and used this
validation network to test explainability. The targets of transcription factors in this ChIP-chip data were filtered using the criterion p < 0.001.
TF perturbation network created by Hackett et al., who fit dynamical systems to their TF perturbation experiments (T..
B.6 Strategy to potentially extend DASH to arbitrary number of layers
Supposing we only have access to prior knowledge the form of putative prior effect sizes between the inputs and outputs P E Roxn, Then, for an NN with L layers W1 E Im= Xn W2 E Rmmxxm ...., Wt. E Roxm,, we can adopt a strategy where we consider the pruning scores to be fixed for all but one layer.
25

---

Since the product wL W2 w E Ro represents the overall flow of information from inputs to outputs at epoch i, we surmise that 2 2. s2 E Roxa should reflect P. We can thus prune as follows:
1. Starting with the last layer, we fix the pruning scores of all other layers and compute as follows:
WT (1) Ar. P-IINVR (2 ( ... 12 (2
..
2. For the middle layers / E (1 - 1, - 2. 3,2], we do:
22 = (1-))) + plnv, (n) 2 (442)14))) P.PIIVV ( ... (2
3. The first layer can be pruned using:
2 = (1- + A PInVL (5 ... ...... ( 2n P
B.7
Processing steps for real data
B.7.1 Breast cancer
The original data set comes from cross-sectional breast cancer study (GEO accession GSE7390 [3] consisting of microarray expression values for 22000 genes from 198 breast cancer patients, that is sorted along a pscudotime axis. We note that the same data set was also ordered in pseudotime by [57] in the PROB paper. For consistency in pscudotime inference, we obtained the same version of this data that already preprocessed and sorted by PROB. We normalized the expression values to be between 0 and 1. We limited our analysis to the genes that had measurable expression and appeared in the aforementioned motif map and PPI matrices. This resulted in a pseudotrajectory of expression values for 11165 genes across 186 patients. We removed contiguous interval of expression across time points for testing (5%), and split up the remaining 178 time points into training (170, 90%) and validation for tuning Aprior (8, 5%).
B.7.2 Yeast
GPR files were downloaded from the Gene Expression Omnibus (accession GSE4987 44) and consisted of two dye-swap technical replicates measured every five minutes for 120 minutes. Each of two replicates were separately na-normalized using the maNorm () function in the marray library in A/Bioconductor [63]. The data were natch-corrected (29] using the ComBat () function in the sva library [33 and probe-sets mapping to the same gene were averaged, resulting expression values for 5088 genes across fifty conditions. Two samples (corresponding to the 105 minute time point) were excluded for data-quality reasons, as noted in the original publication, and genes without motif information were then removed, resulting in an expression data set containing 48 samples (24 time points each replicate) and 3551 genes.
B.7.3
Bone marrow
The data is originally from 41 (GEO accession code = GSE194122). The cleaning, preprocessing, and pseudotime analysis and was appropriately performed by [0 in the PathReg paper, and made publicly available, allowing us to access the processed version. Importantly, ①I split up the data into 3 different lineages (Erythroid, Monocyte, and B-Cell), and we fit a separate PHOENIX model on each lineage. The set contains 5 separate batches of data for each lincage. we used I for training (batch SID2), 1 for validation (batch SIDI) and 3 for testing (batches SIDI, S2D4, and S3D6).
B.8 Pathway analyses for breast cancer, yeast, and bone marrow datasets
We followed very closely the steps below from the Methods section of the PHOENIX paper 27 in order to compute pathway scores, with the only difference that we compute scores between different sparsification strategies Pru.
26

---

B.8.1 Gene influence scores
Given MPru a PHOENIX model trained on a dataset consisting of k genes, and sparsified using the pruning strategy Pru (for Pru E (DASH, IMP, C-NODE, ..., PathReg)), we performed perturbation analyses to compute gene influence scores ISrru.. We randomly generated 200 initial (t 0) expression vectors via i.i.d standard uniform sampling (o(0), E { Next, for each gene j in Mpzu. we created a perturbed version of these initial value vectors {g (0) } where only gene j was perturbed in each unperturbed vector of (r) 12000 We then fed both sets of initial values into Mrru to obtain two sets of predicted trajectories (tg(t), R her Rhhrrr r-
200 and {1ğ' (t),
R hee) } across set of time points T. We calculated influence the average absolute difference between the two sets of predictions, that represented how changes initial (t = 0) expression of gene j affected subsequent (t > 0) predicted expression of all other genes in the Pru-dimensional system
200
& (,(()), 02 (00)))
IS pra.j 200
T
IET ty0
B.8.2 Pathway influence scores
Having computed gene influence scores ISpru.j for each gene j in each dynamical system of dimension & genes sparsified with method Pru, translated these gene influence scores into pathway influence scores. We used the Reactome pathway data set, GO biological process terms, and GO molecular function terms from MSigDB (36), that map each biological pathway/process, to the genes that are involved in it. For each system sparsified by Pru, we obtained the pathway (p) influence scores prrpp) the sum of the influence scores of all genes involved in pathway p
PSeru.p
Σ ISpru.j. jEP
We statistically tested whether each pathway influence score is higher than expected by chance using empirical null distributions. We randomly permuted the gene influence scores across the genes to recompute "null" values PSpeapr For each pathway, we performed Q 1000 permutations to obtain
a null distribution (PSprupp-q 1g that can be compared to PSpru.p. We could then compute an empirical p-value as p q== Ipsp. p PPSmpp''' where I the indicator function. Finally, we used the mean (Ho(Peu.p) and variance (o (((ru.p)) of the null distribution (PSpra.p-g 18 to obtain and visualize pathway z-scores that are now comparable across pathways (p) and sparsification strategies (Pru)
19
PSPrup Mo(Pru.p)
1(pru.p)
O(Pru.p)
B.9 Implementation details for other sparsification strategies the PHOENIX architecture
B.9.1
Iterative magnitude pruning
As discussed in Section 3 IMP can operationalized as special case of DASH by setting
λ1 >2 0.
B.9.2 PINN
This is simply the PHOENIX model equipped with the prior-informed loss term. This loss-term in the original PHOENIX paper 27 is inspired by Physics-informed neural networks (PINNs).
B.9.3 PINN + MP
Once a PHOENIX model (trained including the prior-informed loss term, i.c. the PINN term) is fully trained (without any pruning), we inspect the trained model and pruned the lowest 9% of parameters in each of Wz, Wn. Uz. Un based on on the normalized weights (see [.2e B.2 O. We
then fine-tune (i.e retrain without training the pruned parameters) this p% sparsified model and
27

---

calculate its performance on the validation set. We repeat this process for a grid of values for p E {0.50, 0.75, 0.83, 0.87, 0.90, 0.92, 0.95, 0.97, 0.99]. The validation set can then inform the best value of We repeated this entire procedure 3 times, so that we could apply the standard enror rule [24 and choose the optimal p as the sparsest fine-tuned model whose validation MSE is within standard error of lowest obtained average validation MSE.
B.9.4 Penalty based methods
C-NODE, PathReg, and To implementations were obtained from the code associated with the PathReg paper [. We adapted the code so that the base NN architecture was exactly that of the PHOENIX model, including implementation
the gene-speeific multipliers (from Section B.10.)). Finally, we tuned the relevant parameters Åo and λ in the objective function using the validation set.
The code for DST was obtained from:
B.10 A brief overview of PHOENIX NeuralODE model
The following adapted from 27 and provided here for the reader's convenience.
B.10.1 Neural ordinary differential equations
EeuralODEs 8 learn dynamical systems by parameterizing the underlying derivatives with neural
networks: (* dt = f(g(t),t) t) = (N) t). Given an initial condition (tu),, the output (tt;) at any given time-point now be approximated using numerical ODE solver of adaptive step size:
g gto)) + (Ne dt.
A loss function L ((((h) gft) NNe(g(t), dt) is then optimized for θ via back propagation, using the adjoint sensitivity method to carry the backpropagation through the integration steps of the ODESolver.
B.10.2 PHOENIX overview
PHOENIX models gene expression dynamics using NeuralODEs. Notably, for an expression vector of genes, PHOENIX models both additive and multiplicative regulatory effects using two parallel linear layers with m neurons each: NN NNsuma (with weights Wz _ and biases by ") and NN prods (Wn E Rmxr, bu E R"".. Here, NN and NN are equipped with activation functions that model the Hill equation
1+ x 0.5 and (n(x) = log (0)(z)+1).
0.5
The Hill equation is a classical formula in biochemistry that models molecular binding in dependence of concentration. This results in outputs of the two parallel layers
(g)g = Wzor(g(t) + by and en (g)) = exp o (Wnon (g(t) bn). As shown above, (x) yields the output of NNprods in the log space and is subsequently exponenti- ated in en to represent multiplicative effects in the linear space. The outputs es (g())) and (n))(()) are then separately fed into two more parallel linear layers NNEcombine (with weights Us 12 RR and NN Icombiiee (U E Brxm), respectively. The outputs of NNEcombine and NN are summed obtain
cu(g(t)) = (Czc(((t)) + (n))(g(t)..
Finally, PHOENIX includes specific multipliers IR" for modeling steady states of genes that do not exhibit any temporal variation (de-()) 0, Vt). Accordingly, the output for each gene in (t)) is multiplied with ReLU(x;) in the final estimate for the local derivative
(N) = ReLU(u) @ (cu(g(t) gg(t)
Although PHOENIX achieves some sparsity in its weight matrices (Wx. WT., Us, Un) without any external sparsification strategy, the achieved sparsity level is, however, most 12% (see Figure 2 and Tables 4
28

---

B.10.3 Prior knowledge incorporation in base PHOENIX model itself
PHOENIX has the option to promote the NeuralODE to flexibly align with structural domain knowledge, while still explaining the observed gene expression data. This is operationalized via modified loss function
loss based matching observed gene expression data
Lmdd (g(11) = ==((()())) ; (to) + I (1)) dt
loss based matching domain-knowwlee
+ +1 (1- (P (g(4)) NNo(g(()).))
that incorporates the effect of any user-provided prior model P', using tuning parameter т, and the original loss function T.x Ø). PHOENIX implements P as simple linear model (7) A - where is the adjacency matrix of likely connectivity structure based on prior domain knowledge (such experimentally validated interactions) with Ai E {+1, -1, 03 representing an activating, repressive, or no prior interaction, respectively.
For synthetic experiments, we used the simple linear model: P (-y) = A " % where we chose A be noisy/corrupted versions of the adjacency matrices of ground truth networks G350 and Geeo (details in B.3.1b. We set activating and repressive edges in 4% to 1, while "no interaction" was represented using
B.10.4 Algorithm for efficiently retrieving encoded GRN from trained PHOENIX model
We start with PHOENIX's prediction for the local derivative given gene expression vector g() e R" an r-gene system:
dg(t) ReLU(v) W Wu(cz(g(()) ec(((())) s(1]. where
dl
(g)g())) Wzdz(g(t) + by and en(g(t)) exp (Winon(g(t) + bn)
trained PHOENIX model encodes interactions between genes primarily within the gene-specific multipliers IR", and the weight parameters from its neural network blocks Wn,, Wx Ruxr and W. u E 22222mm This inspired an efficient means of projecting the estimated dynamical system down gene regulatory network (GRN) G, In particular a matrix D _ is calculated, where Dij approximated the absolute contribution of gene j to the derivative of gene i's expression
W WT
D = W.
Gene-specific multipliers applied, before adapting the marginal attribution approach described by Hackett et al. (21) This resulted the dynamics matrix D where Di₃ was scaled according to the relative contribution of gene to the rate of change in gene i's expression:
D
D = bu
B.11 Subjecting PHOENIX to DASH pruning
With its simple yet powerful architecture, PHOENIX provides an ideal base setting for applying and testing the merits of the discussed neural network sparsification strategies (Section[2,, including DASH. We discuss the DASH implementation here and provide implementation details for other sparsification strategies in Appendix B.9
For DASH, we follow the steps the L = case in Section Note that in PHOENIX we have k = since we are modeling how set of k genes (inputs) affects each others derivatives (i.e. outputs). We apply the steps from Section 3 on each parallel member of the first (Wx, Wn) and second {Us, 3n} layers. For the first layer, we leverage prior knowledge of transcription
29

---

factor coregulation (often available in the form of protein-proteni interaction matrices) to formulate C' R This then used calculate pruning scores Jzg Un for Wx. Wn. Similarly, for Uz Un in the second layer, we utilize P E IR"* which are easily obtainable motif map matrices encoding prior knowledge of transcription factor binding sites around genes to calculate pruning scores E II
Algorithm 1 Computing 22 (
4
(t)
(t) IT
E
Inputs: weights ( (
_
(t) U priors C, P: epoch I;
previous scores 2
g S (t-1), tuning λ1, A2
(t- 1)
Normalize w
(1) to get W (0)
(see Appendix B.2)
Similarly obtain |W(())
II JUS and (
If
Initialize 22 and 2 randomly with values from standard Gaussian
Else
0 = (1 A1) (1)
A
Plnvz ( c
0 = (1 (1-x)) W (
A Plnvz (2(t-1)T)
1)T c
1 = (1 u2) + X9
p. Pinv (2)))
1 n = (1 U + A P Plnv R (1
B.12 Ablation study using plain MLP instead of PHOENIX the base model
We believe that DASH should remain performant even when using base model that is different from PHOENIX. To be more explicit, the base model would give new expression for NN (g()),
in B. 10.2.
For PHOENIX as base model we have: NNo(g((),) = ReLU(u) @ [oula(t) -90])
For alternative base model we have: (N))g(t), = alternative base model(g(t))-g())
In our ablation experiments, with used a simple two-layer MLP with ELU activation functions the alternative base model. The choice of ELU activation is motivated by the PathReg paper [I]. We chose the number of hidden neurons such that the total number of parameters was comparable between the MLP and the PHOENIX base models. We tested few sparsification strategies against DASH on this new base model, for both synthetic data and the bone marrow data.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explain the new concept of prior-informed pruning (Fig. 1) in Section 3, introduce the state-of-the-ar model of gene regulatory dynamics in Section 4 and show in Extensive Experiments in Sec. 5 and App. A that it finds more meaningful and domain relevant models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide short discussion in Sec. 6, pointing to potential new direction of future work applying DASH different domains, and provide critical analysis in the Experiments (see e.g. closing remarks of synthetic data study).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
31

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There are no theorems nor proofs in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide discussion of synthetic data generation (App. B.1), setup of training (App. B.2), and information about how prior knowledge was defined (App. B.3). We further provide information on which reference gold standard was used for evaluation of the inferred gene regulatory network (App. B.5), any pre- rrepprocessing steps for the real world data (App. B.7) and how analyses were carried out (App. B.8). We further discuss all necessary details on how different pruning methods were implemented (App. B.9) and how regulatory networks were extracted (App. B014.).. We provide an implementation of DASH supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
32

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code and data via GitHub: https://githi: com/ Quackenbushhaab/D.... DASH and reference where all data can be found in the text (note that all real data is publicly available through the respective original authors).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
cublic/guides CoodeSubmissionool.
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //nips /nips.cccpublll. http/uublccggguueee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See questions above.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide error ranges for all synthetic data experiments and provide statisti- cal significance (multiple test corrected) for pathway analyses.
Guidelines:
33

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: All experiments were carried out on the same machine. As these are compara- bly small-scale experiments (i.e., network parameters in the thousands) most standard hardware will easily reproduce these experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that http:///......lecccccccccoooaaa.
didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We follow the NeurIPS Ethics Guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
34

---

Justification: We do not see direct harmful application of our suggested approach, it is foundational line of research and our suggested upplication is in molecular biology. As such, it will rather bring positive societal impact by improving understanding of health and disease and thereby improving therapy design.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: We properly cite the work for the compared methods and all data used within the scope of this work. We also checked that each license applicable. But we do not list each license and terms explicitly.
Guidelines:
35

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The primary contribution of this work is foundational, yet do provide code and data-generating scripts as supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
36

---

All data used within the experiments was from public sources and databases, which have undergone reviews ensure no risks are involved prior their respective publication.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
37


Reviews:
[{"review_id": "pSxODbi2My", "reviewer": "Reviewer_GCM5", "summary": "The paper presents DASH (Domain-Aware Sparsity Heuristic), a new framework for pruning neural network models by incorporating domain-specific knowledge. The primary goal is to improve the interpretability and biological relevance of models used for gene regulatory network (GRN) inference. Traditional pruning methods often fail to reflect biologically meaningful structures, leading to less interpretable models.\n\n### Key Contributions:\n- Introduction of DASH: A pruning method that uses both learned weights and prior domain knowledge to iteratively score and prune network parameters, ensuring the resulting models are both sparse and biologically meaningful.\n- Improved Model Interpretability: By guiding pruning with structural information about gene interactions, DASH produces models that align better with biological insights compared to traditional methods.\n- Experimental Validation:\nOn synthetic data, DASH outperformed general pruning methods in accurately recovering the underlying GRN.\nOn real-world gene expression data, DASH identified biologically relevant pathways that other methods missed.\n- Robustness to Noise: The framework maintained model robustness even in noisy data environments, showcasing its practical utility.", "questions": "### 1. Format and Numerical Examples of $P$:\nIt appears that using prior knowledge alone (i.e., the \"BioPrune\" baseline) can perform reasonably well for both synthetic and real datasets. However, the description of the domain knowledge used is not entirely clear. It seems that different sets of domain knowledge are employed for different datasets. While we understand the generic form of $P$, can you provide numerical examples for both synthetic and real datasets to illustrate the exact format of $P$?\n\n### 2. Parameter Tuning\nHow are the lambda values $\\lambda$ determined for each dataset? Are they manually tuned, or is there an automated process for selecting these values? Can you provide the results and details on the validation process for tuning these parameters?\n\n### 3. Generalizability (to other neural network, to other domain)\nThe paper demonstrates the effectiveness of DASH on gene regulatory networks. How generalizable is this approach to other domains or types of neural networks (CNN, transformer)? Also, what if there are activation functions in the current used models (MLPs)? Have there been any preliminary tests or considerations for applying DASH to fields such as physics, material science, or other areas of computational biology?\n\n### 4. Comparative Analysis\nWhile DASH is compared with several pruning methods, are there any other recent state-of-the-art techniques or methods that should be included in the comparison? How does DASH stand against the very latest advancements in neural network sparsification?", "limitations": "The paper does discuss certain limitations, but more details could be added to enhance transparency and improve the assessment of the work:\n\n**Scope of Application**: The generalizability of DASH to other domains outside gene regulatory networks isn't thoroughly explored. Adding more discussion on the potential limitations when applying DASH to different types of neural networks or scientific fields could be beneficial.\n\n**Bias and Fairness**: The datasets used in biological research can sometimes contain biases, such as underrepresentation of certain populations. Discussing how the method handles such biases and ensuring that the models do not reinforce existing disparities is important. Suggesting ways to incorporate fairness checks and balance the representation in datasets would strengthen the ethical considerations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "### Originality\nThe paper introduces DASH (Domain-Aware Sparsity Heuristic), which uniquely integrates domain-specific knowledge into neural network pruning. This approach is innovative as it combines traditional pruning methods with biological insights to enhance model interpretability, a critical need in scientific research.\n\n### Quality\nThe methodology is robust, with thorough experimentation on both synthetic and real-world datasets. The authors present a detailed comparison with existing pruning methods, demonstrating DASH's superior performance in recovering biologically meaningful structures. The experiments are well-documented, ensuring reproducibility.\n\n### Clarity\nThe paper is well-organized, with clear explanations of the proposed method and its implementation. The use of visual aids such as figures and tables helps in understanding the results and the effectiveness of DASH. However, simplifying some of the technical details could further enhance accessibility.\n\n### Significance\nThe contributions of this paper are significant, especially for computational biology. By improving the interpretability of neural networks in gene regulatory dynamics, DASH provides valuable insights that can aid in understanding complex biological processes. This approach has the potential to be applied to other domains, making it a versatile tool for scientific research.\n\nThese strengths highlight the paper's contribution to advancing the field of neural network pruning by incorporating domain-specific knowledge, leading to more interpretable and meaningful models.", "weaknesses": "### Testing Set Size\nOne notable limitation of the paper is the relatively small size of the testing set, which comprises only 6% of the total data. This raises concerns about the generalizability and robustness of the reported results. A small testing set can lead to overfitting and may not adequately capture the model's performance on unseen data, especially in cases where the performance differences between methods are quite small.\n\n### Complexity of the Methodology\nWhile the methodology is detailed and thorough, it may appear overly complex for some readers. Simplifying certain aspects or providing more intuitive explanations could improve accessibility and comprehension.\n\n### Biological Validation\nThe paper could further substantiate the claim that DASH improves biological relevance by involving domain experts and providing concrete examples.\n\nSuggestions for Improvement:\n\n- **Expert Validation**: Include validation from domain experts to verify the biological significance of the results.\n- **Concrete Examples**: Provide specific examples where DASH's results align with known biological mechanisms or lead to new hypotheses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Test set size**: For synthetic data, we know the *ground truth generative model* and evaluate on that. For breast cancer, we use 6\\% of data to evaluate the MSE, which we picked as data is really scarce and we need sufficient number of samples to train the model. We do, however, have data of an *independent biological experiment* (the ChipSeq data) to evaluate the inferred model in terms of reconstructed GRN, hence the performance is also generalizable. For the yeast cell cycle data, we do test on an entire biological replicate (i.e., have one replicate for training, one for testing). We do see that this information might got lost in the details and will clarify in the revised manuscript.\n\n**Complexity of methodology**: The score is based on a basic reasoning that also underlies GRN inference techniques like [3], which argues that proxies of TF-TF interactions ($P$) or gene-gene interactions ($C$) are most readily available. The DASH pruning score aligns these proxies $P$ and $C$ with corresponding functions of the edges in the neural network. We have also provided more mathematical motivation below to Reviewer #4. We are happy to provide even more intuition on other aspects that the reviewer wants more clarity on. \n\n**Biological validation**: In terms of expert validation, one of our co-authors has deep expertise in molecular biology, and they have helped in the biological analysis of the GRNs inferred by DASH. We further provide the example of the following pathways: \"... *TP53 activity* and *FOXO267 mediated cell death*, both of which are highly relevant in cancer [ 37 , 24 ]\". Unfortunately, due to the strong space constraint, we had to present further biological validation in the appendix (see Section A). Furthermore, note that yeast cell cycle as well as breast cancer results were validated using ChIP-seq, which is an *independent experimental validation of biological plausibility*. As an additional result, we found an interesting biological insight, which we provide in the general rebuttal response due to character limit.\n\n**Numerical examples of $P$**: For synthetic data, we use a corrupted prior based on the data-generating model, which we elaborate on in App B.3 and B3.1. For real data, we use general information of transcription factor binding to gene promoter regions as prior information, which can be computed from binding motif matches with the corresponding genome (human respectively yeast). The result is a matching score that can be thresholded to get the {0,1} matrix encoding which (TF-encoding) gene has a relationship with which other gene. We follow the approach of Guebila et al. [1] to get matrix P. As prior C, we use the STRING database [2], which gives a general (i.e., not tissue-specific) graph of protein-protein interaction. Here, we use the interactions based on experimental evidence only and employ a cutoff of .6 to get a binary adjacency matrix. We will add the additional information to the discussion of priors in Appendix B.3.\n\n**Parameter tuning**: As mentioned in Appendix B4, the $\\lambda$ values are determined using a $K$-fold cross validation approach. Specifically, we have automated a process that fits multiple models to the same dataset each with a different set of lambda values chosen from a grid. This automated grid-search is used to optimize the $\\lambda$ values, based on predicted MSE on the validation set.\n\n**Generalizability**: We focused here on the considerably large field of gene regulatory networks, which has many applications in the biomedical domain. That said, we anticipate that DASH could be applicable to other core science domains, given the task's neural network structure follows a similar MLP layout and we have domain knowledge of similar structure (like matrices about input-output relationships $P$ and output-output relationships $C$).  In principle, the general definition of the score does not depend on the activation functions. The alignment of prior information with the pruning can be seen as an alignment of the prior with the amount of information that flows through the specific edges. Regarding CNNs and transformers, the application of DASH might be less straight-forward and requires further thinking on which part of the network (connections through filters, parts of attention heads) should be aligned with the prior. This is not directly evident, as such components are not directly interpretable in this context and are not necessarily associated with genes. We would be happy to add a detailed discussion about this aspect.\n\n**Comparative analysis**: We compared DASH to 8 other strategies for neural network sparsification, and additionally BioPrune and PINN+MP which are our own suggested strong baselines for comparison against DASH. Some of these eight strategies are classic benchmarks for comparison in the pruning literature, including IMP, which is considered as the standard baseline in the field, while others such as PathReg have been recently proposed for this exact problem setting. The PINN based method for inducing sparsity in the baseline PHOENIX model is as recent as 2024. We would appreciate specific references to relevant recent sparsification methods that include prior knowledge as we are not aware of any further ones.\n\n**Bias**: This is a great point, and we would be happy to discuss this as part of ethical considerations.\n\n[1] MB Guebila et al. GRAND: a database of gene regulatory network models across human conditions. Nucleic Acids Res. 2022\n[2] D Szklarczyk et al. The STRING database in 2023: protein-protein association networks and functional enrichment analyses for any sequenced genome of interest.  Nucleic Acids Res. 2023\n[3] D Weighill et al. Gene Regulatory Network Inference as Relaxed Graph Matching, AAAI, 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mm94PdeddK", "reviewer": "Reviewer_YHXc", "summary": "The proposed DASH method underscores the importance of interpretability in network pruning for biological discoveries, emphasizing the need for alignment with domain knowledge. Using both synthetic and real data, DASH demonstrates superior performance beyond baselines and offers insights into biological systems.", "questions": "1. In Figure 3, what is the percentage of true relationships comparing DASH and PINN + MP?\n2. It’s not surprising that incorporating the known domain knowledge enhance the interpretability and learned more meaningful dynamics. Moreover, how does the learned gene regulatory dynamics itself can benefit the downstream disease related outcome predictions?\n3. The paper mentioned that DASH has the better quality of inferred (new) knowledge. Is there any discussion or literatures to support the inferred “new” knowledge?\n4. How about making lambda a learnable parameter to mimic the complex and dynamic biological system?", "limitations": "The authors clearly addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The DASH’s ability to integrate domain-specific information makes the resulting models more robust to noise which is usually a challenge in complex biological data.", "weaknesses": "Depending on the size and complexity of the domain knowledge, DASH might be hard to apply to large-scale or highly complex networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Large scale networks**: For our particular domain, this problem was addressed by the design of the PHOENIX architecture, which scales up to large and complex networks commensurate to the whole human genome (on the order of $10^4$ genes/dimensions). Here we demonstrate that DASH works effectively on the PHOENIX architecture applied to such large scale datasets (e.g. the breast cancer dataset covers 11165 genes).\n\n**Regarding Figure 3**: For the noise levels of 0%, 5%, and 10%, the percentage of true relationships captured by DASH are 98\\%, 95\\%, and 88\\% respectively, while the corresponding numbers for PINN+MP are 98\\%, 98\\% and 90\\%. However, this is not a good quantitative measure of the how well each method recovers the ground truth, since PINN+MP recovers a lot of spurious features. This is shown in additional Figure 1 (see rebuttal PDF) where we zoomed in to just these two methods and increased the contrast.  A better quantitative measure of alignment is the mean squared error (MSE) between the inferred and the ground truth relationships. We calculate this and include it in additional Figure 1, which clearly shows that DASH outperforms PINN+MP.  \n\n**Integration of domain knowledge for downstream disease related outcome tasks**: We agree that incorporating domain knowledge leads to more meaningful learned dynamics, which is why we anticipate our approach DASH to find many relevant practical applications. In Genomics, also in the biomedical domain, the goal is often not *prediction* but *inference*, which is also the case here: We would like to learn more about the inherent structure of the disease in terms of the complex gene regulations. By learning about these structures, such as learning about specific genes that strongly affect other genes in a specific cancer, we can better understand the disease process and ultimately can design better treatments. More concretely, we can for example try to understand the effect of a drug on the gene regulatory network by investigating the change in gene regulatory dynamics between cells exposed to the drug versus those that were not exposed, which is what the authors of PHOENIX -- the model we use as basis for our experiments -- did in their original work. We also added an additional result, where we found an interesting biological insight, which we provide in the general rebuttal response due to character limit. In particular, we show how the insights derived from the GRN can be used to generate a potential new therapeutic approach.\n\n\n\n**\"New\" knowledge from DASH**: For both breast cancer as well as yeast cell cycle, we use gold standard ChIP-seq data (a biological experiment to locate binding of TFs to the genome) to validate the biology of the inferred GRN. I.e., ChIP-seq experiments on the particular cells gives us a gold standard GRN for validation and we additionally note that this data has not been used for the construction of the prior. For higher-level knowledge, we have included a biological pathway analysis and found that the most relevant pathways identified in our model correspond to key paths in breast cancer progression, yeast cell cycle progression, or, respectively, bone marrow hematopoesis that align with the known biology of these processes.  More concretely, we provide the example of the following pathways: \"... *TP53 activity* and *FOXO267 mediated cell death*, both of which are highly relevant in cancer [ 37 , 24 ]\". In response to your question, we further investigated the discovered biology focusing on the Heme signaling pathway, which was uniquely identified by our method. We provide more details on these interesting findings in the general author rebuttal due to space constraints.\n\n**A learnable $\\lambda$**: Here $\\lambda$ is a hyperparameter of DASH. As mentioned in Appendix B4, the $\\lambda$ values are determined using a $K$-fold cross validation approach. Specifically, we have automated a process that fits multiple models to the same dataset each with a different set of lambda values chosen from a grid. This automated grid-search is used to optimize the $\\lambda$ values, based on predicted MSE on the validation set. This means that $\\lambda$ adapts to the complexity of the data. Learning $\\lambda$ in a differentiable manner instead would involve differentiating a complicated loss that is defined on cross-validation data, which is usually not efficient.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ZiJykkLzs", "reviewer": "Reviewer_m1mG", "summary": "Gene regulatory network inference is an important, but difficult problem.\n\nThe manuscript explores a novel approach to build domain knowledge into a general NODE model for this problem via pruning. The approach could work in other areas.", "questions": "In Fig 2, what does it mean for DASH to have 2x fewer parameters than BioPrune but achieve greater GRN accuracy? Isn’t BioPrune simply using the GRN?\n\nIn Fig 2, what is ”Base model”? \n\nIt would be helpful to understand why PHOENIX with the same GRN prior information, but using regularization instead of pruning, falls behind the DASH pruning strategy.\n\nI’m not really sure what I’m supposed to be able to understand from Figure 3. I can’t see any of the specific matrix entries well enough to compare them.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Gene regulatory network inference is an important, but difficult problem.\n\nThe manuscript explores a novel approach to build domain knowledge into a general NODE model for this problem via pruning. The approach could work in other areas.", "weaknesses": "It’s very hard to match real data biases for the prior GRN when studying simulated data. I’m struggling to understand whether the noise % refers to this GRN prior or the expression data. 5% noise for the prior seems completely unrealistic and would render these experiments irrelevant to real data analysis.\n\nAnalyzing real gene expression data, the authors use priors based on TF motifs mapped to promoter sequences, which enable DASH to identify genes with true TF ChIP binding with apparently high accuracy. Regulatory network inference method evaluations generally find that genes with nearby TF binding events do not correspond well to genes whose expression changes upon TF perturbation. Evaluation with TF perturbation data is the gold standard here and would make for a more compelling DASH evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Noise levels**: The noise was applied to both the expression data as well as the GRN prior, which we further describe in Appendix B1 and B3. We tested on three different noise levels (0\\%, 5\\%, 10\\%). We included results from the 5\\% setting in the main paper and the results for the remaining noise levels are in A1.\nGenerally, we find that DASH is more robust to noise than its competitors and the performance gap increases with the noise level.\nThis insight is also supported by the fact that DASH identifies more meaningful biology on real world data, which is indeed very noisy.\n\n**Evaluation with TF perturbation data**: This is a great observation and we fully agree. However, this is prohibitively expensive and difficult to generate for time-course data and is hence not available up to our knowledge, especially on the genome-scale ($>10^4$ genes). We thus resorted to the next best thing, which is TF ChIP-seq data. If we could obtain perturbation data in the future, we would be eager to extend DASH to it. \n\n**DASH vs BioPrune**: It is correct that BioPrune is simply using the prior GRN to calculate pruning scores, unlike DASH which takes *both* the prior GRN and the learned neural network weights into account when calculating pruning scores. \nHowever, the achieved sparsity is determined by the validation set. Thus, BioPrune can still contain edges that are not in the prior. \nThe pruning process in concretely described in Appendix B.2.5, where we describe how each training epoch consisted of the entire training set being fed to the model, preceded by any pruning step that is prescribed by the pruning schedule. Training is terminated if the validation set performance fails to improve in 40 consecutive epochs. Upon training termination, we have obtained a model that has been iteratively sparsified to an extent that fails to improve the validation set performance. We discovered in our experiments that DASH's combination of data-based information (from the learned weights) and prior information (from the prior domain knowledge or GRN), leads to this achieved sparsity being much better than that of BioPrune. When we compare the GRN encoded by this sparse model structure back to the ground truth GRN, we see that the sparser model obtained by DASH encodes a much more accurate GRN than the denser model obtained by BioPrune. \n\n**Base model**: This is the PHOENIX base architecture without any form of prior regularization. It is a fully connected two layer MLP  with activation functions resembling Hill-like kinetics and  gene specific multipliers for improved trainability. This base model is then subjected to the different sparsification strategies being benchmarked.\n\n**Why prior-informed regularization falls behind prior-informed pruning**: This is because a pruning-based strategy (such as DASH) *explicitly* sparsifies the neural network by setting weights to zero, while regularization takes an indirect route and *implicitly* encourages the neural network parameters towards zero. The former leads to better sparsification in pruning based strategies, which is similar to the traditional pruning literature, where explicit pruning such as IMP, SNIP, SynFlow, etc usually outperform implicit $L_p$-based approaches.\n\n**Regarding Figure 3**: The figure aids a qualitative comparison, where the overall structure of the matrix recovered by each method should be compared to the ground truth matrix (leftmost). It becomes evident that DASH as well as PINN+MP are much more aligned with the ground truth. A good quantitative measure of alignment here is the mean squared error (MSE) between the inferred and the ground truth relationships. We calculate this and include it in Additional Figure 1 (see rebuttal PDF), which clearly shows that DASH outperforms PINN+MP.  We truncated Additional Figure 1 to only DASH and PINN+MP and increased the contrast to visualize the difference between these two, as they can appear to be very similar in Figure 3 of the paper. We will add this quantitative measure to all sparsification strategies in Figure 3 in the final version if the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RfL1pgHyLw", "reviewer": "Reviewer_QJVK", "summary": "The authors propose a network pruning approach which is guided by prior biological domain knowledge, which they call Domain-Aware Sparsity Heuristic (DASH). They aim to obtain highly sparse networks, which align with known biology and gene regulatory dynamics. To do so they propose computing pruning scores which combine learned weights with prior domain knowledge, balanced by a tunable parameter $\\lambda$. The authors apply DASH to PHOENIX, a neuralODE designed to model gene regulatory dynamics and conduct experiments on four datasets, one synthetic and three real world. They also compare DASH to other existing pruning techniques, such as $L_0$, PathReg and SparseFlow. The experiments compare DASH to these techniques in terms of sparsity, accuracy in recovering known biological relationships and MSE of predicted gene expression values, as well as pathway analysis.", "questions": "In addition to the questions and concerns I raised above, I have the following questions/comments:\n\n- For clarity, you apply all the pruning  methods mentioned - $L_0$, C-NODE, PathReg, DST, IMP, SynFlow, SparseFlow - to PHOENIX?\n    \n- What's the difference between PHOENIX with biological regularisation and BioPrune?\n    \n- In Results, you say PINN+MP and BioPrune are your models, yet they both correspond to post-hoc pruning if I understand correctly. Therefore, neither of them have undergone pruning with DASH. Yet, both perform strongly, with PINN+MP outperforming DASH on MSE in Table 1 and seemingly doing a better job at reconstructing ground truth relationships in Figure 3. Why do you think that is?\n    \n- The main difference between models in Table 2 seems to be the Balanced Accuracy, but is it not expected for DASH to recover similar levels of sparsity to BioPrune given you are \"forcing\" convergence towards $C$ in your model and hence achieving a higher Balanced Accuracy?\n    \n- If one of the takeaways of DASH is the reduced training time and/or memory usage it would be good to see training/inference time or GFLOPs.\n    \n- Can this be used to generate new biological knowledge? Some discussion on this topic would enrich the conclusions of this paper in my opinion.", "limitations": "I do not think the potential limitations of DASH are adequately discussed, see my comments above.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The problem addressed in this paper is scientifically very relevant and topical. How to extract real biological insight from gene expression information, in particular as it pertains to gene regulatory networks in an open and important problem. Using domain knowledge priors to constrain the problem and guide pruning of a neural network is a logical approach to this question. While using prior domain knowledge to regularise a neural network is a well studied area, this paper focuses on how to introduce these constraints to continuous-depth NeuralODE models. The authors conduct extensive experiments on both synthetic and real-world datasets, as well as comparing against other well established pruning methods. I think this is relevant and important research, but in my opinion there are four main issues to address in this paper: the mathematical formulation for DASH is very confusing, there is an over emphasis of results obtained on synthetic data, as well as a reliance on a metric (balanced accuracy) which might potentially lead to inflating results, and a lack of clarity about potential confounding effects between DASH/PHOENIX. I will detail each of these concerns below.", "weaknesses": "I think this is relevant and important research, but in my opinion there are four main issues to address in this paper: the mathematical formulation for DASH is very confusing, there is an over emphasis of results obtained on synthetic data, as well as a reliance on a metric (balanced accuracy) which might potentially lead to inflating results, and a lack of clarity about potential confounding effects between DASH/PHOENIX. I will detail each of these concerns below. \n\n1 - Mathematical formulation of DASH, in the case of L=2 (at the top of p.5 - I have no numbering on the PDF here for some reason): \n\n  - In line 139 you define your weight matrix as  $W \\in \\mathbb{R}^{k \\times r}$, where $k$ is your input corresponding to your genes - yet a paragraph later you transpose your notation and start using $W_1\\in \\mathbb{R}^{m\\times k}$, $W_2\\in \\mathbb{R}^{r\\times m}$, which is confusing.\n   - You then define an input-input matrix $C \\in \\mathbb{R}^{n\\times n}$ which should be $\\mathbb{R}^{k\\times k}$ as you defined your inputs a few lines above, you follow by saying that $W_1$ has $n$ inputs and that the matrix product $\\Omega_1^{(t)^T} \\Omega_1^{(t)}  \\in \\mathbb{R}^{n\\times n}$ -  this doesn't make sense dimensionally. \n   - You posit $\\Omega_1^{(t)^T} \\Omega_1^{(t)}  \\in \\mathbb{R}^{n\\times n}$  approximates to $C$ without first clearly defining what $\\Omega_1^{(t)}$ is and without clearly enunciating why this is most likely a good approximation. \n   - You then define a recurrent relation between $\\Omega_1^{(t)}$ and $\\Omega_1^{(t-1)}$, which needs an initial condition to be well defined. Furthermore, this temporal dependency is not present in the L=1 case. \n   - Why are you using the left and right pseudoinverses on $\\Omega_1^{(t-1)}$ and $\\Omega_1^t$ respectively? It would be good to take time to explain this approach in more detail, as this is the central premise of this paper. You don't need to define what the left and right pseudoinverse of a matrix is. You do need to explain why you are using them here and to what end. \n   - The product of $W_2^{(t)} \\cdot W_1^{(t)}$ should be $\\in \\mathbb{R}^{k\\times r}$, not $\\mathbb{R}^{o\\times n}$\n       \n\nThe definition of DASH is the central premise of the paper, yet there are many notation errors, variables which are not properly defined, and most importantly only vaguely motivates why the proposed substitution makes sense. I suggest you introduce all the notation you will use, even if it seems trivial, and be more careful with errors in notation. It would be great if you expanded on this paragraph by giving more intuitions behind the physical reasoning, as well as explaining the mathematical steps described. \n\n2 - Over emphasis of results obtained on synthetic data:\n\nA significant portion of results (Table 1, Figure 2 and 3) and discussion focus on synthetic data. Although useful for initial validation and exploration, an in depth examination of the three real-world datasets studied would be more relevant. The data generation process likely oversimplifies the complexities of real regulatory networks, which could lead to an unrealistic assessment of the model's performance.  The results obtained on the real-world datasets are not as compelling as those obtained on the synthetic dataset suggesting DASH is not generalising as well as it could and might be fitting to characteristics of the synthetic dataset. Expanding on the results obtained for the real-world datasets would help address these concerns. For example, the results presented in A.3 Table 6 for the bone marrow data differ significantly from the Sparisity and MSE results obtained for $L_0$, C-NODE and PathReg in the PathReg paper. I would be interested to know more about these differences. Could it be coming from using PHOENIX as a baseline, rather than the NN model used in PathReg?  In my opinion the results on the bone marrow dataset should be in the main body of the work, as it reproduces an experiment from the PathReg paper, permitting a more direct comparison between the methods. \n\n3/4 - Balanced accuracy to validate biological alignment \\& confounding effects between DASH/PHOENIX\n\nThe balanced accuracy for biological alignment metric is not clearly defined in the main body of the article, yet is one of the main results in Table 1, 2 and 6, where the authors argue that because DASH is obtaining higher balanced accuracy scores it is better able to recover true biology. They point towards B.5, where they say \"To validate biological alignment of trained and sparsified models, we extracted GRNs from each models (as explained in B.10.4), and compared back to the validation networks.\" In B.10.4 they describe retrieving GRN from the PHOENIX model. This lack of clarity regarding how the biological alignment is measured makes it difficult to evaluate these results. Is it possible that the extraction process itself could be biased towards producing results that align with the prior knowledge used in DASH? This could artificially inflate the balanced accuracy scores for DASH and BioPrune. Could it be there is a confounding effect in the ablation study coming from applying the sparsification methods only to the PHOENIX model? To check this the results would need to be expanded to include a different SOTA neuralODE model to verify DASH works across different models and is not a result of the DASH/PHOENIX combination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Mathematical formulation**:  We apologize for the typos in dimensionality. Below we provide a brief note that includes all the corrected notation, definitions of key variables, and motivations behind the mathematical steps, especially pseudo-inverses. \n\n*DASH for $L=1$.*\nFor a single layer NN, with $k$ input and $r$ output neurons and corresponding weight matrix $\\boldsymbol{W}$ should be $\\in \\mathbb{R}^{r\\times k}$, and we compute pruning scores $\\boldsymbol{\\Omega} \\in \\mathbb{R}^{r\\times k}$ using domain knowledge $\\boldsymbol{P} \\in \\mathbb{R}^{r\\times k}$. At any epoch $t$, we simply use $\\boldsymbol{P}$ as the prior-based portion of $\\boldsymbol{\\Omega^{(t)}}$, the pruning scores at that epoch. \n\n*DASH for $L=2$.*\nWe are sorry for the confusion, during a revision of the paper we changed the notation inconsistently. The dimensionalities you provided are correct. The prior should be $C\\in \\mathbb{R}^{k\\times k},$ where $k$ is the number of genes. Similarly $\\boldsymbol{{\\Omega^{(t)}_1}^{\\intercal} \\Omega_1^{(t)}} \\in \\mathbb{R}^{k\\times k}$. Now, since $\\boldsymbol{W_1^{(t)}}\\in \\mathbb{R}^{m\\times k}$ learns how the $k$ inputs are encoded by $m$ hidden neurons, we surmise that the matrix product $\\boldsymbol{{\\Omega^{(t)}_1}^{\\intercal} \\Omega_1^{(t)}} \\in \\mathbb{R}^{k\\times k}$ should approximately align with our prior knowledge of how the inputs co-vary, i.e. with $\\boldsymbol{C}$. Since solving $\\boldsymbol{{\\Omega^{(t)}_1}^{\\intercal} \\Omega_1^{(t)}} = C$ is not feasible, we initialize $\\boldsymbol{\\Omega_1^{(0)}}$ as all 1s, and solve  $\\boldsymbol{{\\Omega^{(t-1)}_1}^{\\intercal} \\Omega_1^{(t)}} = C$ instead. This is what motivates using the pseudoinverse. \n\n**Synthetic vs real world results**: We agree that the synthetic data is only one part of the story, yet, it highlights distinct differences between the methods in a setting with available ground truth. If methods already fail to perform well on this \"oversimplified\" data, or show specific biases in terms of recovered structure, then such experiments are valuable. Moreover, we disagree that the real world results are \"not as compelling\", we get close to 90\\% accuracy in network recovery, which is much better than existing work (see Table 2). Naturally, there is a sparsity difference to the PathReg paper as the neural network architecture is different, thus different sparsity levels are required, as you correctly pointed out. We do recover a ranking of methods similar to the PathReg paper, which speaks for consistency. We are happy to dedicate more space to bone marrow results with the additional page in case of acceptance.\n\n**Balanced accuracy**: Thank you for pointing that out, we will make this more concise and available in the beginning of the experiment section. In brief, the balanced accuracy measures whether an edge is correctly reconstructed weighted by the sparsity of the ground truth graph. The extraction is the same for all models, as each use the base PHOENIX architecture. The metric is computed with respect to a ground truth (in case of synthetic data) or gold standard (in case of real data). The gold standard ChIP-seq experiments are *not* included in and are distinct from the prior.\n\n**DASH+PHOENIX confounding**: We believe that DASH should remain performant even when using an base model that is different from the PHOENIX. Hence we performed additional experiments (Table 1 in rebuttal PDF), with a simple two-layer MLP with ELU activation function as base model. The choice of ELU activation is motivated by the PathReg paper. We test a few sparsification strategies on this new base model applied to both synthetic data and the bone marrow data. We found DASH to still be performant. \n\n**Questions about PHOENIX**: We apply all the sparsification strategies to the PHOENIX *base model*, that is the PHOENIX model without the biological regularization, a two-layer MLP with activation functions that resemble Hill kinetics. BioPrune is a pruning-based strategy that uses prior knowledge of the GRN to *explicitly* sparsify the neural network by setting weights to zero. Biological regularization takes an indirect route and *implicitly* encourages the neural network parameters towards zero through a penalty-term in the regularizer. \n\n**PINN+MP vs DASH**: PINN+MP and BioPrune are novel models, as they have not been proposed in this context and we primarily discuss them as baselines compared to DASH.  On synthetic data, PINN+MP is indeed a strong contender, with an MSE *comparable* to DASH (within error margin), yet deliver much less accurate representation of the underlying gene regulatory system (cf. balanced accuracy). Regarding Fig 3, the difference is barely visible in the small image. We provide a high-res version (Fig 1 in rebuttal PDF) where we also display the error between the inferred and true relationship. DASH outperforms PINN+MP which recovers lots of spurious structure.\n\n**Table 2**: Yes, DASH and BioPrune do encourage the alignment of the model with the *prior knowledge*. But, the balanced accuracy is measured by comparing the sparsified model to ChipSeq *validation data* which is experimentally independent of the priors in question. \n\n**Training time**: This is not a primary focus of our work, as runtime is generally negligible in comparison with applications in vision or language, e.g. DASH on PHOENIX takes approximately 40 minutes on the breast cancer data. The main motivation for pruning here is interpretability and alignment of sparsified models with known biology.\n\n**New knowledge**: In principle, this approach can be used to generate new knowledge. We here focused on insights we can validate, which naturally means known knowledge. As a response to the question, we added a finding on Heme signaling based on new insights and its potential role in therapeutic design, which we will also add to the discussion. Due to the character limit, please refer to the general rebuttal for the details of this finding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iaIkNjUwr3", "reviewer": "Reviewer_pC8f", "summary": "The authors proposed an interesting model called DASH. The model can recover the gene interact relationships in a sparsity way and make a high accuracy.", "questions": "Can the knowledge be updated by the model? \nCan you model and simulate the different activities of the gene regulatory? E.g. activate the genes or deactivate the genes. \nHow to align the parameter n with \\Omiga in DASH for H=2? What is its biological meaning and align with the prior knowledge? \nHow to measure the accuracy? Do you consider whether the reconstructed relations exist in biology but have not been discovered? E.g. regulatory network rewiring in the cells\nHow to do the pathway analysis? How to get the enriched pathways?\n\nYou use Fig. 1 and Fig. 3 but Figure 2. Please ensure they are in the same form.", "limitations": "The authors don't set a limitation section in the paper. The authors proposed a method that simulates the bio system. However, the biology system is more complex. More things can be discussion in the future.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors proposed an interesting method to model the cells.\n\nThe insights are straightforward for biology, and the regulatory network should be sparse and accurate.", "weaknesses": "The paper is not well organized. E.g., The explanation of the equations and experiments. (see questions)\n\nMore biological background should be included for better understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**More biological background**: Thank you for the suggestion. With the additional space, we will add further introductory background including a simple figure of how proteins and genes are interacting, and how such an interaction is represented in the prior matrices, and further explanations on the biological mechanism of gene regulation and its impact on disease, in particular cancer, where these regulatory networks are perturbed leading to the particular diseases, which makes GRN analysis so relevant. If you have further specific requests on what needs elucidation, we would appreciate your feedback.\n\n**Can knowledge be updated by the model**: Yes, definitely. While the prior knowledge is a great starting point, DASH sparsifies based on *both* the prior knowledge and the patterns it learns from the data itself. If there are new patterns in the data that are not present in the prior knowledge, then the final model will have learned updated relationships on top of the prior domain knowledge. \n\n**Simulating effect of activating or deactivating genes**: Yes, this is something the PHOENIX model itself (i.e. the base model on which we benchmark the different sparsification strategies) is able to do. We can intervene on an input and set it to zero and analyze the effect on the predicted dynamics. \n\n**Calculating $\\Omega$**: We are sorry for the confusion, during a revision of the paper we changed the notation inconsistently. The prior should be $C\\in \\mathbb{R}^{k\\times k},$ where $k$ is the number of genes. Similarly $\\boldsymbol{{\\Omega^{(t)}_1}^{\\intercal} \\Omega_1^{(t)}} \\in \\mathbb{R}^{k\\times k}$. So the product of the score matrix reflecting the first layer should align with the prior input-input relationship matrix $\\boldsymbol{C}$. Similarly, $\\boldsymbol{{\\Omega^{(t)}_1} \\Omega_2^{(t)}} \\in \\mathbb{R}^{r\\times k}$ should align the prior output-input relationship matrix $\\boldsymbol{P}$. The score is based on a basic reasoning that also underlies GRN inference techniques like [1], which argues that proxies of TF-TF interactions ($P$) or gene-gene interactions ($C$) are most readily available. The DASH pruning score aligns these proxies $P$ and $C$ with corresponding functions of the edges in the neural network. We have also provided more mathematical motivation below to Reviewer #4.\n\n**Finding new/undiscovered biological relations**: Such a finding would be ideal and our model has been designed for this purpose.\nInspired by your and other reviewer's question, we further analysed the obtained networks. Focusing on the Heme signaling pathway, which was uniquely identified by our suggested approaches. It turns out that this signaling pathway is highly relevant in breast cancer, we then proceeded to extract interaction partners from our model, i.e., the factors most highly affecting the Heme signaling molecules' dynamics, that could potentially be used as a new drug target. We provide further information on this finding in the general rebuttal response. We hope to discover further relations in the future, when we apply this approach in the field. \n\n\n\n**Pathway analysis**: As described in Appendix B8,  we first take a trained and sparsified model and compute gene influence scores, which we then use to compute pathway influence scores via permutation tests. The $z$-scores of the permutation tests are a measure of pathway enrichment.\n\n**Labelling consistency** : Thank you for pointing this out. We will ensure consistency in a revised version.\n\n[1] D Weighill et al. Gene Regulatory Network Inference as Relaxed Graph Matching, AAAI, 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their service and the provided constructive feedback. We are confident that we addressed the remaining concerns. In particular, we
- clarified the validation setting of GRNs, which stemmed from ChIP-seq experiments that were *independent* of the prior information,
- added additional results on a "simple" MLP architecture as base model (see rebuttal PDF),
- added an additional example of biological knowledge that can be extracted from our networks and sketch how it can be further used (see below),
- made mathematical notation consistent throughout the manuscript,
- added further explanations on the biological background, the relation of pruning scores and the application, and specifics on the training and validation process.

We report an additional finding on Heme signaling, which is a pathway uniquely identified as relevant in our approaches (cf. App. Fig. 5). Heme as a signaling molecule has key roles in the gene regulatory system [1], and turns out to have an anti-tumor role in breast cancer specifically [2]. Subsequent approaches pharmaceutically targeting Heme signaling showed success [3], with one of the key regulators affected being Bach1. To suggest further targets for, e.g., combination treatment, we hence examined the top-5 regulatory factors in terms of weights in our estimated gene regulatory dynamics. These factors include PBX1 and FOXM1, for which a drug repurposing of existing compounds, such as [4], could lead to a potential new treatment for this specific cancer.

[1] SM Mense, L Zhang, Heme: a versatile signaling molecule controlling the activities of diverse regulators ranging from transcription factors to MAP kinases.  Cell Res. 2006
[2] NA Gandini et al. Heme Oxygenase-1 Has an Antitumor Role in Breast Cancer.  Antioxid Redox Signal. 2019
[3] P Kaur et al. Activated heme synthesis regulates glycolysis and oxidative metabolism in breast and ovarian cancer cells. PlosOne 2021
[4] YA Shen et al. Development of small molecule inhibitors targeting PBX1 transcription signaling as a novel cancer therapeutic strategy. iScience 2021

Meta-review: The scores from reviewers are 5, 5, 4, 3, 5. The reviews make many valuable points, but the authors have responded very well, so as the area chair, I believe that the numerical scores deserve to be higher, and this submission should be accepted.

The authors' responses include additional experiments with positive results, and a novel biological finding that is plausible and may have clinical utility.

Here is a high level question for the authors to discuss in the final version. They write "While pruning can simplify deep neural network architectures ... in the context of gene regulatory network inference, state-of-the-art techniques struggle with biologically meaningful structure learning." There are two different meanings for the word "network" here: the DNN versus the actual biological mechanisms. An ideal algorithm would output the actual regulatory structure, not just a pruned DNN that can answer questions about this. A DNN is an indirect representation of the actual mechanisms. The question is, to what extent can the literal regulatory structure be represented and discovered directly?

Another high-level suggestion: Mention in the abstract, and discuss more in the text, applicability of the DASH method outside molecular biology, expanding lines 61, 62 and 326, 327. In which domains does or does not prior knowledge tend to be of the type required by DASH?
Decision: Accept (poster)