Title: Bilevel Network Learning via Hierarchically Structured Sparsity
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural network

network estimation

hierarchical selection layers

sparse regularization
Abstract: Accurate network estimation serves as the cornerstone for understanding complex systems across scientific domains, from decoding gene regulatory networks in systems biology to identifying social relationship patterns in computational sociology. Modern applications demand methods that simultaneously address two critical challenges: capturing nonlinear dependencies between variables and reconstructing inherent hierarchical structures where higher-level entities coordinate lower-level components (e.g., functional pathways organizing gene clusters). Traditional Gaussian graphical models fundamentally fail in these aspects due to their restrictive linear assumptions and flat network representations. We propose NNBLNet, a neural network-based learning framework for bi-level network inference. The core innovation lies in hierarchical selection layers that enforce structural consistency between high-level  coordinator groups and their constituent low-level connections via adaptive sparsity constraints. This architecture is integrated with a compositional neural network architecture that learn cross-level association patterns through constrained nonlinear transformations, explicitly preserving hierarchical dependencies while overcoming the representational limitations of linear methods. Crucially, we establish formal theoretical guarantees for the consistent recovery of both high-level connections and their internal low-level structures under general statistical regimes. Extensive validation demonstrates NNBLNet's effectiveness across synthetic and real-world scenarios, achieving superior F1 scores compared to competitive methods and particularly beneficial for complex systems analysis through its interpretable bi-level structure discovery.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, authors propose DeepBLNet, a novel deep learning framework for bi-level network estimation. This method captures nonlinear dependencies and reconstructs inherent hierarchical structures in the data. By integrating hierarchical selection layers and compositional deep architectures, the method identifies both high-level group and low-level variable dependencies while overcoming the limitations of traditional Gaussian and linear models. The paper provides theoretical guarantees for consistent recovery of hierarchical structures and demonstrates superior empirical performance on synthetic and real-world datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- This is a novel, original deep learning approach to bi-level network estimation.\n- The paper provides clear motivation for the work, and specifies the use cases (high dimensional, non-linear) for which it would be most useful.\n- Theoretical guarantees are convincing for the case considered of Gaussian noise. The introduced notion of bi-level consistency appears to be a useful new metric for the field, especially for high-dimensional cases.\n- Empirical results on synthetic and real-world datasets show this is a valuable tool for non-linear, high-dimensional settings. Authors are transparent with the caveat of associated longer runtimes. \n\n**Weaknesses**\n- The approximation theory is rigorous but dense, which is unfortunate because of how useful it could be for practitioners. A more intuitive explanation of the implications (for example, what network depth and width are needed for given data) would improve accessibility.\n- Similarly, the methodology section is not particularly accessible to someone not familiar with network estimation -- Figure 1 is helpful, but would be even more useful if better leveraged in Section 2. As concepts are introduced and steps are outlined, could we follow along in the Figure? Or introduce smaller figures?\n- The work would benefit from additional transparency around model complexity and how that scales (see Q1), practical interpretability (Q2), and effects of the Gaussian noise assumption (Q3).\n\nQUESTIONS:\nHere are my questions about the work. Upon receiving satisfying responses to these questions (either in the form of additional analysis/results or explanation as to why the answer isn't important) as well as some assurance relevant information will be included in the updated manuscript, I will revise my score.\n\n1. **Scalability:** How does the method scale with dataset size, specifically in terms of model size? I think it would be valuable to understand how the cost grows for large-scale cases where this work would be most helpful. Authors provide runtimes but do not provide explicit computational complexity or memory requirements.\n\n2. **Interpretability:** Authors claim improved interpretability due to bi-level structure, and provide some insight into how this works in Appendix A3. However, it remains mostly unclear to me how easily domain experts can extract actionable insights from the learned networks. How could one leverage DeepBLNet's so-called interpretability? Can you provide examples for the datasets considered?\n\n3. **Gaussian noise assumption:** How does the assumption of Gaussian noise affect the usefulness theoretical guarantees? Are there relevant examples of datasets/cases where this assumption doesn't hold? How can we understand the value of the method in those cases?\n\nLIMITATIONS:\nAuthors do include a limitations section. I have remaining concerns about model complexity (see Q1) as well as the significance of the Gaussian noise assumption (Q3).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a deep learning framework for bi-level network inference via node-wise regression for predicting each variable using all others. The authors introduce hierarchical selection layers with adaptive sparsity penalties to capture both group-level and variable-level network structure.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths: Significance and Originality\n\nThe hierarchical selection mechanism via a group-level layer and a variable-level layer is conceptually appealing and applying neural networks with sparsity penalties to the variable selection setting (particularly to the hierarchical one) appears novel. The theoretical analysis provides formal guarantees and the problem formulation is clear.\n\n## Weaknesses (Quality and Clarity)\n\n- I could not find Stage 1 of the two-stage algorithm from Algorithm 1 in the code. Instead, adaptive weights are computed from random initialization.\n- The experimental evaluation lacks proper validation methodology. Neither the paper nor code describes train/validation/test splits or cross-validation for selecting the super-sensitive sparsity hyperparameters in the baselines, as well as in the proposed method.\n- Baseline choice is hard to interpret: the paper does not discuss its baselines in the related-work section, and the methods mentioned there, i.e., Shan et al. (2020) [22] and Cheng et al. (2017) [4], are not included in the experiments. Comparing against these baselines, or discussing the used baselines in the related work, would strengthen the paper.\n- Placing the related-work section in the appendix is problematic, because clear positioning within the field is central to a scientific paper. The discussion there is quite high-level, mentioning only three competitors without sufficient detail. The authors should include the foundational variable selection work of Meinshausen & Bühlmann (2006), and also cite this when they introduce their variable selection setting, and they should better emphasize their novelty by discussing which aspects have appeared elsewhere. From my (shallow) search, applying neural networks to this variable selection setting appears novel, which would strengthen the paper's contribution if properly positioned in the main text.\n\n### Minor:\nThe architecture uses only 3 layers total, so calling it \"deep\" learning (as in the title) seems overstated. Additionally, referring to a standard linear layer as a \"core innovation\" (line 105) is a bit off.\n\n## Conclusion:\nWhile I appreciate the novel idea and the theoretical contributions look promising, the problems stated above need to be addressed before I can give a positive rating.\n\nQUESTIONS:\nSee Strengths And Weaknesses section\n\nLIMITATIONS:\nThe limitations discussion is embedded within the conclusion rather than treated separately, and should be expanded.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a bi-level network estimation model that learns dependencies between variables at both the cluster level and the individual node level, connecting the two via a selection-consistent mechanism. The method outperforms baseline approaches on real-world and synthetic networks with nonlinear signals, and achieves comparable results on synthetic networks with linear signals.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe idea of network estimation from a multiscale perspective is novel and compelling, especially considering that many real-world networks exhibit hierarchical or community structures. While multiscale modeling is well established in fields such as PDE-based dynamics, it is less explored in network estimation, making this approach interesting.\n\n**Weaknesses**\n\n1. As acknowledged by the authors, the proposed method relies on known node labels that reflect the community structure. This assumption could limit the method’s applicability. It would be helpful if the authors discussed typical real-world scenarios where such labels are available and justified their practicality. In addition, it is not entirely clear how much of the performance gain is attributable to the inter-class sparsity information provided by the labels, versus the bi-level architectural design of the proposed method. See the Questions section for further discussion.\n\n2. How the parameters $\\zeta$ and $\\gamma$ are chosen should be discussed, as they directly controls the level of sparsity, and can reflect strong prior knowledge of the network. For the network estimation problem, it seems there is no such a validation set in determine these parameters. Also it will be helpful if the dependency on these parameters can be represented.\n\nQUESTIONS:\n1. *Clarifying the Utility of the Bi-Level Architecture*. The paper uses DeepNet as a baseline to demonstrate the advantage of the bi-level architecture. However, if I understand correctly, DeepNet does not utilize the label information, which raises a question: to what extent does the observed improvement stem from the architecture itself, versus the use of additional label information?  This comparison will be more convincing if the authors willing to consider the following. Suppose the inter-cluster connections are sparse, thereby naively predicting them to be absent will only cause a small decrease in accuracy. Then, evaluate DeepNet where training is done within each cluster. In this case, DeepNet can leverage the label information to a partial extend. If DeepBLNet still outperforms this modified DeepNet, it would provide stronger evidence for the value of the bi-level design rather than just label availability.  Additionally, reporting the within-cluster and between-cluster edge densities (including synthetic networks) would help clarify how much signal is available at each level of structure, and how much the model is leveraging inter- vs. intra-cluster information.\n\n2. *Group level sparsity*. In Equation (3), the Lasso penalty on theta implicitly assumes a priori that the inter-group connections are sparse. However, this may not hold in all settings. In some cases, θ could be dense, but with lower (or higher) average weights compared to within-group connections—for instance, in the contextual stochastic block model. While sparsity is a reasonable assumption at the individual edge level, it is not necessarily justified at the group level. The paper would benefit from a clearer discussion of when and why this sparsity assumption is appropriate. This is also related to the last question of whether a prior knowledge on sparse inter-group connections contribute mostly to the performance improvement.  \n\n3. *Scale-free terminology*. Referring to a network with only 10 nodes and a maximum degree of 4 as \"scale-free\" is somewhat misleading. The concept of a scale-free network typically involves a wide range of scales (i.e., node degrees spanning multiple orders of magnitude). With such a small network, it is difficult to meaningfully assess whether the degree distribution follows a power-law or exhibits true scale-free behavior.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DeepBLNet for recovering bilevel graphical structures from data.\nThe method is designed to learn both a high-level adjacency structure and a low-level structure simultaneously, from observed data.\nThe authors model this task via a bilevel architectural decomposition.\nThe paper provides a consistency theorem that claims the proposed architecture can recover the true bilevel structure asymptotically under certain assumptions.\nExperimental validation is primarily conducted on synthetic datasets with known structure.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n- Novel Problem Formulation: The paper addresses a new variant of structure learning: recovering bilevel graph structures.\n- Methodological Design: The proposed DeepBLNet is a well-designed neural network architecture that handles group-level and intra-group adjacency jointly\n- Theoretical Guarantee: The authors establish a consistency theorem for structure recovery. While based on idealized assumptions, this provides a valuable theoretical anchor.\n- Modular Framework: The approach is likelihood-model agnostic (Gaussian, Ising, etc.), making it flexible across domains.\n\nCons:\n- The consistency result is solid in isolation, but it depends on strong assumptions: perfect optimizer, sufficient model expressiveness, asymptotic data. There's no discussion of finite-sample behavior, generalization, or network capacity.\n- While the paper includes some experiments on real-world datasets (e.g., neuroimaging and gene expression), the evaluations are mostly structural and unsupervised. There is limited benchmarking against other models or external validation of whether the recovered graphs are meaningful for real tasks.\n- The paper lacks an architectural ablation study. In particular, it would be useful to understand whether the bilevel separation (group-level and intra-group adjacency prediction) actually improves performance, compared to a flat or shared model. Some ablation or alternative design comparison would clarify the necessity of the proposed structure.\n\nQUESTIONS:\n1. How sensitive is your model to group assignment errors or group size imbalance?\n2. What is the inductive bias that makes DeepBLNet better than training a flat structure learner on a large graph?\n3. Do you plan to apply this method in real application domains, and if so, which ones?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}