HGNet: Scalable Foundation Model for Automated Knowledge Graph Generation from Scientific Literature
Authors: Devvrat Joshi
Islem Rekik
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Graphs
Representation Learning
Graph Neural Networks
Geometric Deep Learning
Scientific Text Mining
Primary area: foundation or frontier models, including LLMs
TL;DR: The first lightweight (<1B parameters), hierarchy-aware framework for building high-quality knowledge graphs from scientific research papers, significantly outperforming all state-of-the-art models in entity and relation extraction.

Abstract: Automated knowledge graph (KG) construction is essential for navigating the rapidly expanding body of scientific literature. However, existing approaches face persistent challenges: they struggle to recognize long multi-word entities, often fail to generalize across domains, and typically overlook the hierarchical and logically constrained nature of scientific knowledge. While general-purpose large language models (LLMs) offer some adaptability, they are computationally expensive and yield inconsistent accuracy on specialized, domain-heavy tasks such as scientific knowledge graph construction. As a result, current KGs are shallow and inconsistent, limiting their utility for exploration and synthesis. We propose a two-stage framework for scalable, zero-shot scientific KG construction. The first stage, Z-NERD, introduces (i) Orthogonal Semantic Decomposition (OSD), which promotes domain-agnostic entity recognition by isolating semantic “turns” in text, and (ii) a Multi-Scale TCQK attention mechanism that captures coherent multi-word entities through n-gram–aware attention heads. The second stage, HGNet, performs relation extraction with hierarchy-aware message passing, explicitly modeling parent, child, and peer relations. To enforce global consistency, we introduce two complementary objectives: a Differentiable Hierarchy Loss to discourage cycles and shortcut edges, and a Continuum Abstraction Field (CAF) Loss that embeds abstraction levels along a learnable axis in Euclidean space. To the best of our knowledge, this is the first approach to formalize hierarchical abstraction as a continuous property within standard Euclidean embeddings, offering a simpler and more interpretable alternative to hyperbolic methods. To address data scarcity, we also release SPHERE, a large-scale, multi-domain benchmark for hierarchical relation extraction. Our framework establishes a new state of the art on benchmarks such as SciERC, SciER and SPHERE benchmarks, improving named entity recognition (NER) by 8.08\% and relation extraction (RE) by 5.99\% on the official out-of-distrubtion test sets. In zero-shot settings, the gains are even more pronounced, with improvements of 10.76\% for NER and 26.2\% for RE, marking a significant step toward reliable and scalable scientific knowledge graph construction.

Published as a conference paper at ICLR 2026
HGNET: SCALABLE FOUNDATION MODEL FOR AUTO- MATED KNOWLEDGE GRAPH GENERATION FROM SCI-
ENTIFIC LITERATURE
Devvrat Joshi Islem Rekik
BASIRA Lab, Imperial-X (I-X) and Department of Computing Imperial College London, London, United Kingdom (devvrat. joshi24, i.rekik)e (enkikkeemmerrr... al acc.kk
ABSTRACT
Automated knowledge graph (KG) construction is essential for navigating the rapidly expanding body of scientifie literature. However, existing approaches face persistent challenges: they struggle to recognize long multi-word entities, often fail generalize across domains, and typically overlook the hierarchical and log- ically constrained nature of scientific knowledge. While veneral-purposs large language models (LLMs) offer some adaptability, they are computationally ex- pensive and yield inconsistent accuracy on specialized, domain- tasks such as scientific knowledge graph construction. As result, current KGs are shallow and inconsistent, limiting their utility for exploration and synthesis. We propose a two-stage framework for scalable, zero-shot scientific KG construction. The first stage, Z-NERD, introduces (i) Orthogonal Semantic Decomposition (OSD), which promotes domain-agnosten entity recognition by isolating semantic "turns" in text, and (ii) Multi-Scale TCQK attention mechanism that captures coher- ent multi-word entities through n-gram n-gram-aware aware attention heads. The second stage, HGNet, performs relation extraction with hierarchy-aware message passing, ex- plicitly modeling parent, child, and peer relations. To enforce global consistency, we introduce two complementary objectives: Differentiable Hierarchy Loss discourage cycles and shortcut edges, and a Continuum Abstraction Field (CAF) Loss that embeds abstraction levels along learnable axis in Fuclidean space. To the best of our knowledge, this is the first approach to formalize hierarchical abstraction as a continuous property within standard Euclidean embeddings, offer- ing simpler and more interpretable alternative to hyperbolic methods. To address data scarcity, we also release SPHERT large-scale, multi-domain benchmark for hierarchical relation extraction. Our framework establishes new state of the art on benchmarks such as SciERC, SciER and SPHERE benchmarks, improving named entity recognition (NER) by 8.08% and relation extraction (RE) by 5.99% on the official out-of-distribation test sets. In zero-shot settings, the gains are even more pronounced, with improvements of 10.76% for NER and 26.2% for RE, marking a significant step toward reliable and scalable scientific knowledge graph construction.
INTRODUCTION
The exponential growth of scientific literature has created overwhelming challenge: the pace of publication now far exceeds human capacity for manual review and synthesis Taylor et al. (2022). Automated systems that can distill unstructured text into structured, machine-readable representa- tions are therefore essential. Knowledge Graphs (KGs) offer compelling solution. representing entities such as methods, datasets, or concepts as nodes and their semantic counections edges Wang et al. (002a)). Yet, constructing high-quality KGs from dense, jargon-rich scientific text re-
Code and data available at htcp::///gtttbb..cmmaaaaaaaaaa// 'Our benchmark dataset is available at https::/github.ccmm//

---

Published as a conference paper at ICLR 2026
mains difficult, complex terminology, long multi- word entities, and layered hierarchical structures introduce challenges that current approaches fail to resolve.
Scientific KG construction is constrained by four interdependent challenges that limit both accuracy and scalability. The first two concern node identification. Many scientific concepts are expressed as long multi-word phrases, such as "in situ transmission electron microscopy", which must be recognized as coherent units. This problem of multi-word entity recognition remains unresolved be- cause most sate-offtheeart models treat token boundaries as incidental rather than explicit objectives Zhou et al. (2024): Zaratiana et al. (023). ) A second challenge is domain generalization: systems trained on one discipline must adapt new fields without extensive retraining. Supervised models often collapse out of distribution, while large language models (LLMs) with more than 10 billion parameters offer broader adaptability but are computationally expensive, making them impractical for routine KG construction. In contrast, our proposed model is lightweight, with only ~300 million parameters. Unlike general- LLMMs which require billions of parameters to achieve gener- alization, HGNet matches the computational efficiency of specialized baselines while offering the robust zero-shot capabilities of a foundation model.
Once entities are identified, the next task is to establish edges between them, introducing two fur- ther challenges. Scientific knowledge is hierarchical, for instance, "Deep Learning" " is a subfield of "Machine Learning". Capturing such relationships requires hierarchy
relation modeling Bai et al. 2021), , yet conventional models are largely hierarchy-blind, relying on shallow co-occurrence statisties rather than deeper conceptual structures. Beyond hierarchy, graphs must also be logically consistent: contradictions such as declaring A part of B and B part of A undermine integrity. Large language models, while capable of performing both NER and RE in single framework, are again prohibitively expensive and yield inconsistent results on specialized, hierarchical scientific knowl- edge (Refer table of /Zhang et al. (2024). Ensuring globally consistent structures therefore essential, but current methods lack mechanisms to guarantee that the graph forms valid Directed Acyelic Graph (DAG) Chami et al. (202).. Reliable KG construction thus requires not only accurate entity recognition but also principled modeling of relational and structural dependencies.
To the best of our knowledge, we introduce the first end-to-end system designed to address all four challenges by discovering latent hierarchical structures directly from text. Our framework oper- ates two stages. Stage one employs Z-NERD, zero-shot recognizer that ensures robust domain generalization via Orthogonal Semantic Decomposition (OSD) and captures complex entities with a Multi- TCQK attention mechanism. Stage two applies HGNer (Hierarchy Graph Network), which builds a latent probabilistic graph, preserves hierarchical dependencies through specialized message- message-passing, and enforces structural integrity via two objectives: Differentiable Hierarchy Loss and Continuum Abstraction Field (CAF) Loss. To enable rigorous evaluation and mitigate data scarcity, we contribute SPHERE a large-scale, multi-domain benchmark. Across datasets such as SciERC, SciER and SPHERE, our framework achieves new state-of-the-arr results: average gains of 8.08% in NER and 5.99% in RE, with even larger improvements in zero-shot settings (10.76% for NER and 26.2% for RE). Collectively, these contributions establish the first principled, empirically validated solution for building robust, high-quality scientific KGs at scale. Our main contributions include:
We propose a novel, domain-agnostie NER model that significantly outperforms all state-of-the-art baselines on the most challenging scientific benchmarks (Refer table D). Its core innovations are the Multi-Scale TCQK mechanism, which enables coherent recognition of multi- word entities by dedicating attention heads to n-gram patterns, and Orthogonal Semantic De- composition (OSD), a new technique for zero-shot generalization that identifies domain- invariatt "semantic turn" signals.
Z-NERD:
Hierarchy Graph Network (HGNet): We introduce GNN architecture for relation extraction that establishes a new -fute-of-the-art by a significant margin on complex hierarchical benchmarks (Refer table 2, 3 4. It learns and reasons over a latent, probabilistic conceptual graph and, unlike standard GNNs, uses specialized message-passing channels (parent-to-child, child-to-parent, and peer-to-peer) to preserve the directional flow of hierarchical information.
A Geometric Theory of Abstraction: We introduce a novel paradigm for representing hierar- chical knowledge. We the first to formalize abstraction as an intrinsic geometric property of standard Euclidean space, realized through a learnable Abstraction Field Vector that creates a universal axis of generality. This approach, enforced by Continuum Abstraction Field (CAF)
2

---

Published as a conference paper at ICLR 2026
Loss, offers more direct and interpretable alternative complex methods like hyperbolic em- beddings.
To address the critical bottleneck of data scarcity, we created and release SPHERE, the first large-scale, multi-domain benchmark specifically designed for hierarchical re- lation extraction. Generated via novel methodology, it contains over million paragraphs and 111,000 annotated relations, enabling more robust training and evaluation of complex KG con- struction models.
The SPHERE Dataset:
RELATED WORKS
The task of constructing a Knowledge Graph (KG) from scientific literature involves two primary sub-tasks: Named Entity Recognition (NER) to identify conceptual nodes, and Relation Extraction (RE) to identify the semantic edges between them. This section situates our work within existing paradigms for these tasks, highlighting the persistent gaps that motivate proposed framework.
2.1 ENTITY RECOGNITION IN SCIENTIFIC TEXT
High- High-performance scientific NER has been dominated by supervised transformer models such as SciBERT (Bltagy et al. (2019) and BioBERT et al. pre-trained on large scientific corpora and fine-tuned on task-specific data. This paradigm achieves state-of-the-art performance on in- domain benchmarks and has been scaled to foundation models like BioMedLM et al. (2024), yet faces a critical architectural limitation. The ability capture complex, multi-word entities (e.g., "in situ transmission electron microscopy") arises only as an emergent property of contextual embeddings rather than a dedicated feature, often resulting in fragmented or incomplete recognition. Our Z-NERD framework addresses this gap through the Multi-Scale TCQK mechanism, which intrinsically modifies attention to force heads to specialize in n-gram patterns of varying lengths, offering principled, structural solution.
second limitation poor domain generalization: supervised models degrade sharply on out-of- domain text. Zero-shot methods such GLiNER Zaratiana al. (2023) and UniversalNER Zhou ctt al. (2024) reformulate the task as span matching, while general-purpol LLMs like GPT-4 Ope- nAl (2025) show impressive but inconsistent zero-shot performance Zhang et al. (024). Yet these approaches still depend on surface semantics or world knowledge. By contrast, our Orthogonal Se- mantic Decomposition (OSD) trains the model detect domain-agnostic semantic turns-points where new concepts are introduced, shifting focus from vocabulary to discourse structure. This enables Z-NERD to achieve robust zero-shot performance beyond the reach of semantic matching.
2.2 RELATION EXTRACTION AND HIERARCHICAL MODELING
Relation extraction (RE) has evolved from localized, sentence-level models to corpus-level systems capable of multi-hop hop reasoning across documents. Early neural approaches relied pipeline chitectures, but error propagation soon motivated joint models that simultancously extract entities and relations Zhong & Chen (2021); Yamada et al. (2020); Yan al. (2023). Benchmarks such as Sci SERC| Luan et al. (2018) and SciER ZZhng et al. (2024) have been instrumental in driv- ing progress, enabling transformer-basee methods that achieve state-of-the-art performance on fine- grained scientific relations. However, these methods remain confined to sentence-level reasoning and fail capture the long-range dependencies and cross-sentence evidence chains that are central to scientific literature.
To address this limitation, recent work has shifted toward cross-document relation extraction, em- ploying graph neural networks (GNNs) and multi-hop retrieval to link entity mentions across docu- ments and aggregate distributed evidence Wang al. (022b): Lu al. (202)). Yet such methods typically rely on surface features like co-occurrence or syntactic proximity, conflating textual adja- cency with genuine conceptual relatedness and yielding noisy graphs. Meanwhile, hierarchy-aware approaches such as hierarchical attention Han et al. (2018) and reinforcement learning frameworks Takanobu et al. (2019) show promise but are tailored to shallow taxonomies, limiting their applica- bility the deep, nested, and implicit hierarchies of scientific knowledge. We therefore introduce HGNet, the first GNN architecture explicitly designed for hierarchical relation extraction in scientific literature. HGNet builds latent conceptual graph and leverages parent, child, and peer message-
3

---

Published as a conference paper at ICLR 2026
passing channels to model the directional flow of information, disentangling textual proximity from conceptual hierarchy and enabling the capture of both local and global dependencies while preserv- ing the layered structure of scientific knowledge.
2.3 GEOMETRIC AND LOGICAL REPRESENTATIONS OF HIERARCHY
A key challenge in learning hierarchical structures is ensuring they are both logically and geomet- rically sound. Our HGNet captures directional information flow and disentangles textual proximity from conceptual hierarchy, but still requires principled embedding space for global consistency. To address this, we introduce geometric perspective: instead of merely extracting relations, we learn hierarchy representation that respects logical constraints and abstraction levels. While hyperbolic geometry is often used for low-distortion tree embeddings Nickel Kicla (017)). our approach de- fines new paradigm, learning globally consistent abstraction ordering directly in Euclidean space. This is achieved via the Continuum Abstraction Field (CAF) Loss, which orients the embedding space along learnable universal "axis of abstraction." Simpler and more interpretable, this prior integrates with our Differentiable Hierarchy Loss, enforcing logical constraints such as acyclicity. Together, these losses ensure the learned KG is both geometrically organized and logically coherent.
METHODOLOGY
Our framework consists of unified, co-trained architecture utilizing a shared SciBERT encoder. First, Z-NERD processes raw scientific text to identify and extract entity mentions. Second, HGNet takes the contextualized entity embeddings from this shared encoder as input, which maintains the document-level context, and learns their hierarchical and peer relationships, constructing globally consistent knowledge graph.
3.1 Z-NERD: ZERO-SHOT ENTITY RECOGNITION
Z-NERD is an efficient tagging model that addresses two key challenges in NER: recognizing multi- word entities and generalizing to new domains. Its architecture first applies Orthogonal Semantic Decomposition the input embeddings to extract agnosic features, then feeds these en- riched representations into a transformer encoder modified with our Multi-Scale TCQK mechanism.
3.1.1 DOMAIN GENERALIZATION VIA ORTHIOGONAL SEMANTIC DECOMPOSITION (OSD)
To overcome domain overfitting, a model must learn to recognize abstract linguistic patterns rather than memorizing domain-specific vocabulary. This requires identifying features that are invariant across different scientific fields.
We therefore hypothesize that Hypothesis 3.1 robust domain generalization can be achieved by training a model to rely on features that explicitly isolate the introduction of new semantic concepts, rather than simply tracking the overall semantic flow. By providing the model with "semantic turn" signal (a measure of how much the meaning deviates from the preceding context), we can make it sensitive to the underlying logical structure of the text instead of overfitting to vocabulary.
We achieve this by decomposing the change vector between consecutive word embeddings, AE, = Frexi, F into orthogonal components. The sustaining component is the projection of this change onto the previous word's embedding, representing elaboration. The divergent component, which is orthogonal to the previous word's direction, captures the introduction of new concept.
AEC- Exstt-11 Exexte-1
Usustaining =
E|exte_ ||2
(1)
Udivergent, AE Usustaining,
We concatenate the divergent vector Udivergent, with the original contextual embedding Eust, (Re- fer figure (). This enriched representation provides the model with the domain-invarion signal of conceptual shifts necessary for robust zero-shot generalization.
4

---

Published as a conference paper at ICLR 2026
3.1.2 COHERENT MULTI- WORD ENTITIES VIA
TCQK ATTENTION
Standard self-attention mechanisms lack a strong architectural bias for word adjacency, often failing to identify the precise boundaries of long entities. This leads to fragmented predictions and an incomplete understanding of complex concepts.
Our guiding hypothesis is that Hypothesis 3.2 robust, variable-length entity detection can be achieved by designing a self- attention mechanism where different heads are architecturally special- ized to capture n-gram pafterns of different lengths. By fusing the global reach of attention with the local sequence awareness of convolutions at multiple scales, the model can learn to recognize single tokens, short phrases, and long entities in parallel.
We introduce the Multi-Scale Temporal Convolutional Queries & Keys (TCQK) mechanism to re- alize this. Before computing attention scores, we modify the Query (Q) and Key (K) vectors using ID convolutions. We partition the H attention heads into G groups, assigning each group a con- volutional kernel Cg with a specific size Kig (e.g., 3, For each head h in group g. we compute:
Q A = Cg Koonv. = C.(A))
(3)
This modification intrinsically alters the self-attention mechanism, compelling different heads to specialize in n-gram patterns of varying lengths. This allows the model to capture both short acronyms and long chemical names as single, coherent concepts. Note: We apply Multi-Scale TCQK mechanism over the concatenated embeddings from orthogonal semantic decomposition. (Refer figure 6 for more details)
3.2 HGNET: HIERARCHY GRAPH NETWORK
Given the entities extracted by Z-NERD, the goal of the Hierarchy Graph Network (HGNet) is to estimate the conditional distribution P(Tweal D, GK), where each local relation triplet (start entity, relation, end entity) is constrained by global Hierarchical Knowledge Graph (HKG) Gx. The input entities (h,,, h2)) are the contextualized output embeddings from the SciBERT encoder, ensuring the document-level context is maintained for relationship prediction. a standard procedure for efficiency in SOTA RE models. Since 9K is unobserved, HGNet must jointly infer its structure and leverage for reasoning. The model organized around three core components, each grounded in a specific hypothesis about hierarchical consistency. (For complete visual overview of the architecture, refer Fig. 7 in the Appendix.)
3.2.
PROBABILISTIC HIERARCHICAL MESSAGE PASSING
Traditional Graph Neural Networks (GNNs) fundamentally "hierarchy-blind"" They operate on a single, undifferentiated graph, propagating messages uniformly across all connections. This ap- proach flawed it cannot distinguish between information flowing "up" from specific child, "down" from an abstract parent, or "sideways" from a peer, thereby corrupting the learned repre- sentations. To address this, our work is guided by the hypothesis that Hypothesis 3.3 GNN can preserve and leverage hierarchical structure if its message-passing architecture is explicitly de- signed to respect it. By creating distinct, parallel channels for information flowing along different axes of the hierarchy, the model can learn specialized, context-aware update functions, leading to richer and more robust entity embeddings.
To realize this, our architecture operates on a probabilistic graph where relations treated as learnable variables. First, Latent Relation Predictor (MLP) estimates the probability distribution over relation types {parent-of,, peer-of, no-edge} for every pair of entity nodes (u, v):
Pav softmax (ettmaxxMMM(((() (MLP((h.]]hh]]]]
(4)
These probabilities serve as soft edge weights for a three-channel message passing scheme. For given node at layer k, we compute aggregated messages, each with separate, learnable weight matrix (W up: W doww: W peer) capture the unique semantics of each relational direction:
1. Parental (Upstream) Aggregation: m.
parents
Evey
Pparent (Wap 1e (K)
2. Child (Downstream) Aggregation: m.chlldrnn
children
Euey prarent (Waonthie
5

---

Published as a conference paper at ICLR 2026
3. Peer Aggregation: m Evey Pav (Wreart)) (K))
Finally, these three context-specific messages are concatenated with the node's previous state and passed through an update MLP to produce the final, structure-awar embedding for the next layer:
h (k+1)
UpdateMLP((1g (
m m
children
(5)
3.2.2 LOGICAL COHERENCE VIA DIFFERENTIABLE HIERARCHY LOss (DHL)
A critical challenge in learning a latent graph is that, without explicit constraints, the model has no incentive to ensure its structure is globally coherent. During training. it might predict logically im- possible structures, such as cycles (c.g.. A is part of B, and B is part of A) shortcuts that skip hierarchical levels (e.g., mistaking a grandparent for a parent). These structural inconsistencies cor- rupt the message- message-passing process and lead to semantically invalid graphs. We therefore hypothesize that Hypothesis 3.4
can enforce a logically sound latent hierarchy by explicitly and differen- tiably penalizing these structural impossibilities. particular, by introducing a composite loss that punishes cycles and invalid shortcuts, we guide the model toward a parameter space where the latent graph forms valid Directed Acyclic Graph (DAG) with a strict parent-child hierarchy.
This is achieved with the Differentiable Hierarchy Loss (eiierarhy)) regularizer operating on the predicted parent-of adjacency matrix, Aparent. a weighted sum two components:
Ctieracchy Aecycli Cacyetic +
Aseparation Ausparationneeepratese
(6)
The first component an Acyclicity Loss, which uses the trace of a matrix exponential to differen- tiably ensure the graph is DAG (Refer appendix A..]] for accelerated calculation using Krylov's subspace) Here, is the number of nodes (entities) in the graph. For proof of why this function pushes our graph structure be DAG, refer appendix A.1I
Lacyelic tr(eAppon Aponn O d
The second component is Hierarchical Separation Loss, which penalizes shortcut edges that skip intermediate hierarchical levels (Refer appendix A.7 for efficient computation). Formally, it is de- fined as:
Escparation
(A2 (Apseem)us uw (Aparent) Jww
20,20
Here, (Ap counts the number of length-2 paths from node u to node and the elementwise product with (Aparent we selects only direct edges that skip intermediate node. This encourages the model to maintain a strict parent-child hierarchy by discouraging shortcuts.
3.2.3 GEOMETRIC COHERENCE VIA CONTINUUM ABSTRACTION FIELD (CAF) Loss
model's embedding space typically geometrically "flat," lacking an intrinsic structure for ab- straction. While model might learn that "RNN" and "LSTM" are related, it fails to encode that an RNN is a more general concept, leaving embeddings as a disorganized cloud of points. Our approach is founded on the hypothesis that Hypothesis 3.5 hierarchical understanding is fun- damenial geometric property of the embedding space. By organizing all concepts along single, universal "axis of abstraction," the model can embed hierarchical information directly into the vec- tor representations, making the abstraction level of concept intrinsic property of its learned embedding.
We introduce the Continuum Abstraction Field (CAF) Loss (Leat) impose this geometric structure. introduces a learnable unit vector, the Abstraction Field Vector Wabs. that defines this universal axis (Refer appendix A.5 for for more details on the unit abstraction field vector). An entity o's abstrac- tion score is defined as its projection onto this axis: Date = h Wabe- This abstraction score is continuous, real-valued number, ensuring the model learns fluid continuum rather than limited

---

Published as a conference paper at ICLR 2026
number of fixed, discrete levels. The composite loss, Leat Lranking
V/1 Canchor + "/2 Pregressions
shapes this structure using three distinct objectives:
Ranking Component: Enforces relative parent-child ordering with margin 6.
Lranking
max(0, (he - hp) wabs + 6)
(9)
ppart-of
(e.p)
Anchoring Component: Pins known root (V,) and leaf (Vi) nodes scores of and 0.
Lanchor
Σ (he,
Wabs 1)² +
Σ VEV.
(h.,
Wabs 0)2
(10)
V.E..
Regression Component: Pulls predicted scores towards ground-truth topologiaal depth scores (w), which are derived for all benchmarks by performing a topological sort on the ground truth hierarchical relations.
Eregression
E(hh, Wabs)
((he
Mlopo ())
(11)
Varain
DEVenn
Note that while the margin å in Cranking could theoretically limit the number of discrete levels to 1/6, this constraint effectively relaxed by Lregression which acts as the dominant global anchor pulling each embedding toward its true topological depth. The model thus learns continuous spectrum of abstraction rather than discrete levels (see Appendix A.6 for empirical evidence).
This transforms abstraction from a simple regression target into an organizing principle of the entire embedding space.
3.2.4 FINAL RELATION PREDICTION
The relations (parent-of, peer-of, no-edge) described in Section 3.2.1 are an internal mechanism used solely for structure regulation. The h(++1) embedding produced by HGNet represents the final, optimized structure-aware representation. The extraction of the actual, task-specific (head, relation, tail) triplets then performed by a standard downstream classification head (the same type utilized by models such HGERE or PL-Marker) operating tation. This head takes the structure-awar (A++1) embedding as input and predicts the full set of fine-grained relations required by the benchmarks. The loss from this external task, CRE, constitutes the primary task objective of the entire framework.
this refined represen-
3.2.5 COHERENT ARCHITECTURE AND JOINT OPTIMIZATION
While Sections 3-2.1-3. define the modular components of HGNet, the system operates as single, unified framework, where all elements are simultaneously optimized in one end-to-end for- ward pass. This co-training mechanism ensures the learned structure is globally consistent, logi- cally sound, and geometrically coherent. The entity embeddings (h,,h..) are the contextualized outputs from the shared SciBERT encoder of the Z-NERD stage. The Latent Relation Predictor estimates the initial probability distribution Pue over relations, which immediately initiates two par- allel paths: Logical Regularization and Message Passing. The predicted parent matrix (A AArrea) feeds directly into the Differentiable Hierarchy Loss (Chierurchyy, which penalizes structural errors like cy- cles (Lacyetie) and shortcut edges (Lseparation). Concurrently, the probabilities P.. are used as soft edge weights to guide the three-channel Probabilistic Message Passing GNN, which produces enhanced, structure-aware entity embeddings (h (1(()))).
These final embeddings h. (k+1) are then used to compute the Continuum Abstraction Field (CAF) Loss (Leat). This loss enforces geometric ordering, shaping the embedding space along the universal axis of abstraction. The embeddings task-specific (head, relation, tail) triplets (CRE). The total loss for the model is a weighted composite sum of the primary task objective and the two structural regularizers:
also passed to final classification head, which predicts the
Limal LRE XI Litierarchy Da
(12)
7

---

Published as a conference paper at ICLR 2026
This joint optimization is the core of HGNet: Qmierrchy forces the graph structure be logically sound, while Lear forces the node embeddings to be geometrically sound.
VALIDATION OF STRUCTURAL LOSSES
The efficacy enforeing structural and geometric coherence confirmed through targeted ablation studies. Ablating the Differentiable Hierarchy Loss (DHL) led to notable drop in performance, confirming the necessity penalizing logical inconsistencies like cycles and shorteut edges. Sim- ilarly, removing the Continuum Abstraction Field (CAF) Loss resulted in significant degradation in Rel+ FI score, validating that embedding generality as an intrinsic geometric property critical for hierarchical reasoning. (For detailed empirical results, refer to the section
Refer figure for detailed visualization of all components of HGNet. Refer appendix AA.12 for short proof sketch of HGNet a generalized attention mechanism. For short discussion convergence of HGNet, refer appendix A.13
4 EXPERIMENTS
In this section, we present a comprehensive empirical evaluation of our proposed Z-NERD and HGNet frameworks. We first detail the experimental setup, then present the main performance results against strong baselines, and finally conduct series of ablation studies and analyses to validate our core hypotheses.
4.1 EXPERIMENTAL SETUP
Datasets We evaluate our models on diverse of scientific information extraction benchmarks. This includes four established datasets: SciERC Luan et al. (2018), SciER /Zhag et al. (2024), BioRED, and SemEval-2017 Task 10 Augenstein al. (2017). These datasets span multiple sci- entific domains, feature complex entity and relation types, and are standard benchmarks for this task. For fair comparison we report all the metrics on the Out of Distribution official test sets. To address the scarcity of large-scale annotated data, we also introduce SPHERE, new, large-scale dataset created via a novel LLM-based generate-ande andtte-ta---n methodology. SPHERE contains four distinct scientific domains (Computer Science, Physies, Biology, and Material Science), enabling robust evaluation of both in-domain and zero-shot performance.
Evaluation Metrics For Named Entity Recognition (NER), we report the standard micro Fl score. For the more complex end-to-end Relation Extraction (RE) task, we use the strict Rel+ Fl metric Zhong & Chen (202)), which requires the model to correctly predict the boundaries and types for both entities in relation, well as the relation type itself.
Baselines Our frameworks are benchmarked against comprehensive suite of strong models. For NER, we compare Z-NERD against state-of-the-ar supervised models (SciBERT, PL-Marker, HGERE), powerful specialized model (UniversalNERR-) and several general-purpose LLMs in zero-shot setting. For RE, we compare HGNet against top-performing end-to-end supervised models (PL-Marker, HGERE), standard GNN architectures (GCN, GAT), and LLMs. table I for ref- erences). Additional experiments comparing HGNet against Hyperbolic Baselines and Few-Shot CoT LLMs detailed in A.8 and A.9 respectively.
For implementation details, hyperparameters and SPHERE dataset generation process, refer appen-
dices A.2 and A.3.1
4.2
MAIN RESULTS
Z-NERD for Entity Recognition As shown in Table our Z-NERD framework sets new state- of-the-art art across all benchmark datasets, achieving an 8.08% average F1 improvement over previous supervised models. The gains even higher in the zero-shot SPHERE domains, with 10.76% average improvement. In contrast, general general-purpose LLMs evaluated directly in zero-shot mode without task-specific fine tuning failed to produce meaningful results, mainly due to difficulties in
8

---

Published as a conference paper at ICLR 2026
identifying multi-word word entity boundaries. These LLMs are also much larger, highlighting 2-NER'ss efficiency at under 1B parameters.
HGNet for Relation Extraction The central goal of HGNet is to learn a globally coherent rep- resentation of scientific knowledge that respects its inherent hierarchical structure. We divide the relations in each dataset into two classes, hicrarchical and peer, and report the macro FI for these two classes separately. As shown in Table 2 and 4 HGNet consistently outperforms all bascline 2
models, with an average improvement of 5.99 % on the benchmark datasets and 26.20% % on the zero- shot SPHERE dataset. This demonstrates a distinct advantage on datasets characterized by complex hierarchical relations, driven by its hierarchy-awerr aware multi-channel message- message-passing architecture.
4.3 ABLATION STUDIES AND ANALYSIS
Analysis of Z-NERD Architecture To validate our architectural contributions to Z-NERD, performed targeted ablation studies. First, removing the Multi-Scale TCQK mechanism sults in a severe degradation of performance across every dataset. This sharp decline confirms Hypothesis 3.1 validating that standard attention mechanisms are ill-equipped to handle the co- herent identification of complex, muldi-word entities and that explicit architectural bias for n- gram patterns is fundamental to success. Second, removing the features from Orthogonal Semantic Decomposition (OSD) also leads to consistent drop in F1 scores. The true significance of this component becomes most apparent in the zero-shot domain generalization task, where the perfor- mance drop is particularly pronounced. This provides compelling evidence for Hypothesis 3.2, confirming that isolating "semantic turns" is the key to learning abstract, domain-agnostic patterns for robust generalization. (Refer table I For evidence of how Orthogonal Semantic Decomposition affects the learned embeddings to improve zero-shot inference, refer appendix A..4 A.4
Analysis of HGNet Architecture The superior performance of HGNet is driven by its unique design, which we validate through ablations. The model's overall strong performance across all datasets, particularly those with deep hierarchies, provides strong empirical support for Hypothesis 3.3 This confirms that explicitly hierarchy-aware GNN architecture with special- ized parent, child, and peer message-passing channels produces richer and more accurate entity rep- resentations than standard GNNs. Furthermore, removing the Continuum Abstraction Field (CAF) Loss resulted in significant degradation in Rel+ Fl score, validating Hypothesis 3.5 by demon- strating that embedding generality hierarchical reasoning. Similarly, ablating the Differentiable Hierarchy Loss also led to a notable drop in performance, which confirms Hypothesis 3.4 and underscores the necessity of enforcing logical constraints like acyclicity. (Refer table and 3) For learned abstraction score analysis and qualitative error analysis, refer appendices A. and A.6
intrinsic geometric property of the space is critical for
Table 1:
Fl scores (%) of different models on NER benchmarks. SPHERE: CS, Physics, Bio, MS report both supervised (Sup) and zero-shot (ZS) results. OSD refers orthogonal semantic decomposition. Z-NERD w/o OSD and w/o TCQK refer to ablations.
Models
SCERC
SaER
BORED
SemEval
CS
Physics Z5
Bio
MS
Z5
Sup
ZS
Sup
Z8
Supertiod
ScIBERT Ye al
2022
67.32
70.71
89.15
69.14
68.19
37.02
71.90
61
73.33
68.45
67.29
57.14
PL Marke Tan
2013
70.32
74,01
86.41
47.69
68.64
56.39
72.83
60.51
75.
66.17
66.72
57.92
HGERE Van 20233
75.92
81.19
89.43
48.25
69.82
58.95
72.46
60.67
76.42
68.51
67.24
58.03
UniversunvER 76 Zhou
2024
66.09
73.13
88.46
47,60
00M
Zem-Stoot
Mama 3.76 70h Louvron al
A023
46.20
49.57
$4.82
30.15
OOM
Wehn3-32h Quhen e 2025
41.63
46.52
31.71
26.48
OOM
lama- 3.1 Iimmn redyton
2023
33.96
31,21
33.58
21.70
0OM
Proposed Apuroach (Z-NERD)
NERD w/o TCQK
73.43
75.12
84.43
47.85
68.47
59.35
74.92
61.74
73.92
68.30
69.48
57.73
Z NERD wio OSD
74,39
80.27
90.12
$0.95
76.93
62.01
76.68
65.17
82.40
73.29
78.24
63:15
Z-ERR
78.84
82.71
91.05
52.26
80.47
69.52
82,39
73.19
84.35
74,21
83.96
72.28
CONCLUSION AND FUTURE WORK
We present a novel two-stage framework for automated knowledge graph construction in the sci- entific domain. The first stage, Z-NERD, combines Orthogonal Semantic Decomposition with Multi-Scale TCQK attention mechanism for robust, domain-agnostic recognition of complex enti- ties. The second stage, HGNet, employs probabilistic graphical model with specialized message-

---

Published as a conference paper at ICLR 2026
Table 2: Rel+ Fl scores (%) of different models on SciERC, SciER, BioRED, and SemEval 2017 for two relation types (Hierarchical and Peer), along with overall Fl across all relation types. For BioRED, which only has Peer relations, the overall score equals the peer score. The same entity prediction method is used across models for fair comparison. All values are rounded to two decimals.
Models
SciERC Poor
SciER Poer
BioRED Overall
SomEval 2017 Peer
Hier.
Overall
Hier.
Overall
Hier.
Overall
Supervised Models
PL-Marker Yo al. 2022
35,00
44.97
41.63
40.25
61.84
56.78
29.87
32.96
43.40
37.19
HGERE Yan al.
37.72
47.35
43.86
43.79
64.35
58.47
32.39
33.81
45.73
38.63
PURE PhRE & Chen 2021
34.39
38.46
36.78
38.53
56.21
49.35
29.41
28.94
41.35
34,92
Zero-Shot 7.7M
Models
GPT33.5 Turbo Ye et 2023
14,97
15.02
14.98
8.35
8.91
8.58
6.36
16.30
17.13
16.74
openailent-nes 120m
2023
19.68
21.27
20.45
27.93
27.52
27. 64.6
7.15
23.59
26.16
23.88
laaa 3,3- v. Jouvron el
2023
22.15
22.53
22.39
23.97
25,06
24.59
7.29
23.65
25.38
24.12
gwen/gwen3-33 3-2benn33333--- Oween
2015
16.57
19.33
18.20
24.02
24.45
24.28
6.71
20.92
21.38
21.09
1lama-3.1-B- 1-Sb-instamlf Teavion of
2023
13.30
14.27
13,92
17.15
17.69
17.43
5.48
14.11
14.46
14.24
Superiised
GNN-bbaed Models
GCN
40.13
48.78
45.62
47.37
63.89
$7.35
31,93
34,08
45.92
38.96
GCN w/o Zom.
38.46
48.51
44.98
46.85
64.22
56.89
32.28
32.82
45.72
37.99
GAT
40.37
49.1
46.21
47.35
64.29
57.64
32.40
34.47
46.19
39.25
GAT w/o Coen
38.96
49.25
45.48
47,03
64.23
$7.30
32.74
33.52
45.88
38.43
Proposed Appraacess
HG wio Rpat
48.52
55.37
51.68
59.10
68.93
62.79
34.31
42.16
49.42
45.05
HGGNet LosF HGNet
42.70
47.33
54.75
58.67
33.09
38.58
50.96
55.41
53.19
62.36
67.02
65.38
33.85
45.37
50.64
47.03
Table 3: Rel+ F1 scores (%) on SPHERE dataset variants (Computer Science, Physics, Biology, Material Science) for two relation types (Hier,, Peer). We also report overall F1 across all relation types. All models use the same entity prediction method for fair comparison. Subscripts indicate improvement over SOTA model HGERE.
Models
Comp. Sci. Peer
Physies Peer
Biology Poer
Mat. Sci.
Hier.
All
Hier.
All
Hier.
Hier.
Peer
All
Supervised Models
PL-Marker Ye et al.
2022
51.98
57.04
55.29
50.22
56.48
53.51
52.35
53.76
53.03
52.96
53.27
53.12
HGERE Yan el
2023
54.20
59.86
57.93
53.17
58.90
56.28
54.52
56.47
55.21
55.84
55.86
55.43
Proposed Approaches
HGNet (ours) w/o CpHL w/o Cear
77.40
81,36
79.51
76.93
83.47
80,60
82.53
84.29
83.74
81.91
85.64
83.65
73.62
74.83
77
74.01
75.30
74.66
79.15
78.64
78.90
77.43
76.92
77.28
67.14
65.89
66.50
64.51
66.24
65.96
75.17
73.29
74.13
75.95
77.38
76.32
Table Zero-shot Rel+ F1 (%) when trained on Physics+Biology and evaluated on Comp. Sci. and Mat. Sci. datasets. Due to the expensive nature of these experiments, we only tested our zero-shot performance on the best performing sfate-of-the-an models in our previous experiments.
Models
Comp. Sci.
Mat. Sci.
Hier.
Peer
All
Hier.
Peer
All
PL-Marker Ye et al. (2022)
28.72
28.41
28.56
33.10
34.22
33.85
HGERE Yan et al. (2023
29.93
29.63
29.81
36.27
39.41
37.97
HGNet (ours)
59.36
64.07
62.60
69.92
71.33
70.62
passing channels, regularized by Differentiable Hierarchy and Continuum Abstraction Field losses. The latter introduces learnable Abstraction Field Vector, ensuring logical coherence and geometric structuring around a universal abstraction axis. We also introduce SPHERE, a large-scale bench- mark for scientific KG construction. Experiments show gains of up to 10.76% for NER and 26.2% for RE in zero-shot scenarios, validating our hypotheses and the efficacy of a structurally-aware approach to knowledge extraction.
Future work could extend this framework to incorporate multimodal information from figures and tables, and explore its application in dynamic, continuously updated knowledge graphs that reflect the real-time evolution of scientific fields. Additionally, syntactic filtering based on dependency parsing could be integrated as a preprocessing step to prune unlikely entity pairs, further enhanc- ing relation extraction precision (Joshi & Rekik 2025). Furthermore, leveraging these structured KGs for downstream reasoning tasks, such as automated hypothesis generation, presents an exciting avenue for further research.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, all source code for the Z-NERD and HGNet models and the newly introduced SPHERE dataset publicly available https://githi:: //gttubbg com/ We have provided comprehensive details of our experimental setup, includ- ing datasets, evaluation metrics (Section 4.Ip. implementation, software/hardware configurations, and training hyperparameters (Appendix A.2) The methodology for generating the SPHERE dataset further detailed in Appendix A.3.I and all baseline models are described in Section 4.1 facili-
tate fair comparison.
ACKNOWLEDGEMENTS
We are grateful to the Computing Support Group (CSG) at Imperial College London for manag- ing the GPU cluster used in our experiments. We also thank the anonymous reviewers for their constructive feedback, which significantly strengthened this manuscript.

---

Published as a conference paper at ICLR 2026
REFERENCES
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. SemEval 2017 task 10: SciencelE extracting keyphrases and relations from scientific publica- tions. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens (eds.), Proceedings of the 11th International Workshop on Semantic Eval- uation (SemEval-2017), pp. 546-555, Vancouver, Canada, August 2017. Association for Com- putational Linguistics. doi: 10. 18633/11/117-2091 URL pnttps://ace ac aenlllnnhooooo
S17 S17-2091/
Haim Avron and Sivan Toledo. Randomized algorithms for the trace of implicit symmetric positive semi-definite matrix. Journal of the ACM, 58(2):1-34, 2011. doi: 10.1 10.1145/9 1944349.
Yushi Bai, Zhitao Ying, Hongyu Ren, and Jure Leskovec. Modeling heterogeneous hierarchies with relation-specific hyperbolic cones. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text, 2019. URL
Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Rox- ana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, and Christopher D. Man- ning. Biomedlm: A 2.7b parameter language model trained on biomedical text, 2024. URL https://arxtt
httgg/bs/2443..0 18421
Ines Chami, Rex Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks, 2019. /tL//ttpss///axxi http:///aaaii..oo/aaa/////////// httgg//bs/199/ 12933
Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low- dimensional hyperbolic knowledge graph embeddings, 2020. URL htttss://arxiv.orrg
abs/2005.00543
Xu Han, Pengfei Yu, Zhiyuan Liu, Maosong Sun, and Peng Li. Hierarchical relation extraction with coarse-to-fine grained attention. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2236-2245, Brussels, Belgium, October-November 2018. Association for Com- putational Linguistics. doi: 10.18653/v//D18- 1247. URL bttps://allatthol... org/ D18-1247/
Devvrat Joshi and Islem Rekik. Dependency parsing-based syntactic enhancement of relation ex- traction scientific texts. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.). Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 24888 24897. Suzhou, China, November 2025. Association for Computational Lin- guistics. ISBN 979-8-89176-335-7 doi: 10.186531 1/2025. findings-emmlp.13555 URL htRpst::: //clannthology.
Findinggs emn 1p. 13547
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim. Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomed- ical text mining. Bioinformatics, 36(4): 1240, 09 2019. ISSN 1367-4803. doi: 10w.:..193/bbbe (0.1003tiioiffrmaaissooozzz URL http:htts://// htt.:////........oooooooooaaaaa. bt2682
Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. Multi-hop ev- idence retrieval for cross-document relation extraction. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 10336-10355 tics. doi: 10. 10.18653/v1/2023. 1ttp:/3////222....nnnnng-go/ aclinng--cll667. URL ht stttps: http://aaaaattoooooooo////////// findi Findings-een acl parrings-acc..co
Toronto, Canada. July 2023. Association for Computational Linguis-
6577
12

---

Published as a conference paper at ICLR 2026
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Ellen Riloff, David Chi- ang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceedings of the 2018 Conference Em- pirical Methods in Natural Language Processing, pp- 3219-3232, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/11/D18-115 1360. URL
13607
Maximilian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions, 2017. URL httppp:://ariii.or///aaaaa//////
08039
OpenAL. Chatgpt. version).
2025. Large language model (March 2025
Qwen,
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL URLhhtts::// /ttp:://aaxivi org/ abs/ /2//2412..11 15115
Yousef Saad. Iterative Methods for Sparse Linear Systems. SIAM, 2 edition, 2003. doi: 10.1 137/1.
9780898718003.
Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. A hierarchical framework for relation extraction with reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):7072-7079 Jul. 2019. doi: 10.1609/ http://aaaai.v3...33333333 v3300..30117777. URL https:
//ojs. aaa.j.aaaa...og//ddee..
Ross Taylor, Marcin Kardas, Guillem Cucurull. Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science, 2022. (ttts:/ts:://att
www. absg/ab////eee /2111. 09085
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL httpt::tt//......... /ttp:///axvvv.vo http://aawwv..ogaa///////aa 2tbbs/2302..33 13971
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. arXiv preprint 151iv:551..0336 2015.
Chenglin Wang, Yucheng Zhou, Guodong Long, Xiaodong Wang. and Xiaowei Xu. Unsupervised knowledge graph construction and event-centric knowledge infusion for scientific nli, 2022a. URL /httts://aaxxtt
org/ /bs/
Fengqi Wang, Fei Li, Hao Fei, Jingye Li, Shengqiong Wu, Fangfang Su, Wenxuan Shi, Donghong Ji, and Bo Cai. Entity- centeed cross-document relation extraction. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9871-9888 Abu Dhabi, United Arab Emirates, Decem- ber 2022b. Association for Computational Linguistics. doi: 10. 10.18665/1120222.mmpmmmaa URL http:::://aaaaalthaii ogy org 2022. wwn iimmannnn n. 6717
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. Luke: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6442-6454, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/w1/0200
mmilp-main.523. URL htt.ittss//aaaaattt.oooo
org/2020 emnlp main. 523
Zhaohui Yan, Songlin Yang, Wei Liu, and Kewei Tu. Joint entity and relation extraction with span pruning and hypergraph neural networks. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 7512 -526, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/v 1/t//2/eeel2ommmmeeeeeaa emnlp-main.466) 467. URL ht httpp::: http://aaaaantttg..ooooo lemn 1 ma n. 4677
13

---

Published as a conference paper at ICLR 2026
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. Packed levitated marker for entity and relation extraction. In Smaranda Muresan, Preslay Nakov, and Aline Villavicencio (eds.). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4904-4917, Dublin, Ireland, May 2022. Association for Computational Linguis- tics. doi: 111666////022222lloooooo hol-iong.337)
URL https: //ap::://cclll
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models, 2023. URL https://
wrxiv.org/abs//200
Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thienry Charnois. Gliner: Generalist model for named entity recognition using bidirectional transformer, 2023. Utl.pttss://arxxv..rrg
08526
Qi Zhang, Zhijia Chen, Huitong Pan, Cornelia Caragea, Longin Jan Latecki, and Eduard Dragut. SciER: An entity and relation extraction dataset for datasets, methods, and tasks in scien- tific documents. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceed- ings of the 2024 Conference on Empirical Methods Natural Language Processing, pp- 15083-13100 Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10. 10..8853//v1// 1/201/.222222222 h/2024.emmlpmmmma..... emnlp-mminnnn...m. 726. URL emr ma in. 7267
Zexuan Zhong and Danqi Chen. A frustratingly easy approach for entity and relation extraction. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 50-61, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 12202/ maacl-main.5. URL https: /ttps://aclantholooge orq/ 2021. naacl-main.3
Wenxuan Zhou, Sheng Zhang, Gu, Muhao Chen, and Hoifung Poon. Universalner: Targeted distillation from large language models for open named entity recognition, 2024. URL https: //ar wtt.aa//e....... 03279
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 STATEMENT ON THE USE OF LARGE LANGUAGE MODELS (LLMS)
In adherence to the ICLR 2026 policy, we disclose the use of Large Language Models (LLMs) in the preparation of this manuscript and our research methodology.
Role Dataset Generation As detailed in Appendix A.3..1 LLMs (specifically, a mixture of models from the GPT and Gemini families) were programmatically used to generate and self-annotate the SPHERE dataset, which was crucial for training and evaluating our proposed models. The entire process, from KG scaffolding sentence generation and annotation, was designed and supervised by the authors to ensure the quality and validity of the dataset.
component of research. They were
Role Manuscript Preparation Beyond their role in the research itself, LLMSs were also utilized tools to aid in the preparation of this paper in the following ways:
Writing and Polishing: We used LLMs (c.g.. GPT-4) as advanced writing assistants. Their use was primarily focused on improving the clarity, precision, and readability the text, This included tasks such rephrasing sentences for better flow, correcting grammatical errors, ensuring consistent terminology, and polishing the overall prose. The core scientific ideas, arguments, and the structure of the paper were conceived and written entirely by the authors.
Literature Retrieval and Discovery: LLMs were used as supplementary tool to augment our traditional literature review process. We used them to summarize abstracts known papers and to help identify potential related work based on keyword and concept queries. This assisted in broadening our search, but the final selection, critical reading, analysis, and citation of all literature were performed by the authors to ensure academic rigor.
Continuum of Abrtraction
Conseivatoe
Delnn
Nelsk Resin
Coimolegy
Places
Rail
Nirphysk
Wiaanable Plysks
Necia Posia
Nucker Fullon
Queruee Oxtis
Add water ryyiil Canpaationll Pisiia Kaaes Myyo
Hileaa Scarce
Accooore
Figure 1: Continuous axis abstraction for topics physics.
A.2 IMPLEMENTATION DETAILS
Hardware and Software All experiments were conducted on a pirfo-performance computing clus- ter equipped with NVIDIA A30 24GB GPUs. Our frameworks were implemented using PyTorch 2.1 and the Hugging Face Transformers library. For baseline models, we used their official public implementations and recommended hyperheraaapaaaaaaaa hyperparameters to ensure fair comparison.
Training Hyperparameters To ensure reproducibility, we detail the key hyperparameters for our proposed models in Table We used the AdamW W optimizer for all training runs and employed linear learning rate scheduler with a warm-up phase. The optimal hyperparameters were determined via grid search on the validation sets of the respective datasets.
15

---

Published as a conference paper at ICLR 2026
Table Key hyperparameters for Z-NERD and HGNet.
Hyperparameter
Z-NERD
HGNet
Encoder Base Model
SciBERT-base
SciBERT-bar
Learning Rate
x 10 5
1 10
Batch Size
16
8
Optimizer
Adam W 0.1
Adam W
Dropout Rate
0.2
Max Sequence Length
512
512
TCQK Kernel Sizes
[1, 3, 5, 7] N/A
N/A
HGNet Layers
3
CAF Loss Margin (6)
N/A
0.5
CAF Weights (71, 72)
N/A
(1.0, 0.5)
DHL Weights (Aacyelic; Aseparation)
N/A
(1.0, 0.1)
A.3
SPHERE DATASET
A.3.1
GENERATION METHODOLOGY
The SPHERE (Scientific Multidomain Large Entity and Relation Extraction) dataset was created to overcome the critical bottleneck of data scarcity in scientific RE. We employed novel, three-phase generate-anddamnottte methodology driven by a Large Language Model (in our case, mixture of GPT (OpenAI) and Gemini (Google DeepMind) models).
Phase 1: Programmatic KG Scaffolding. We first constructed a ground-truit knowledge graph to serve as a structured backbone. This was done by prompting the L.L with a high- level field (c.g.. "Computer Science") and asking it to recursively expand it into more gran- ular, interconnected sub-fields, methods, and concepts. This foundational step produced deep and logically consistent taxonomy of over 40,000 entities across four domains before any text generated.
2. Phase 2: Throughhrouhhtt Sentence Generation. With the KG as scaffold, high- throughput pipeline generated annotated sentences. This involved sampling small, con- textually related sets of concepts from the graph (e.g., a parent, child, and peer concept) and prompting the LLM, acting as an expert technical writer, to compose a long, complex, academic-style paragraph describing their relationships.
3. Phase 3: LLM Self-Annotation. The newly created sentences
immediately passed back to the same L.LM for self-annotation within the original context. The model performed Named Entity Recognition and Relation Extraction, linking the identified concepts back to their permanent IDs in the ground-truth KG. We observed that the LLM's annotation performance is drastically higher on text has generated itself, enabling the creation of large-scale (10,000 documents, 11 111,000 relations), high-qualin corpus.
A.3.2 STRUCTURAL COMPLEXITY AND SCALE ANALYSIS
To validate the necessity of SPHERE foundation benchmark, we compare its structural proper- ties against existing gold-standard datasets Table
Scale and Diversity. Existing benchmarks like SciERC and BioRED constrained by the high cost of human annotation, typically limited to roughly 500 abstracts and a single domain. In contrast, SPHERE leverages the generative scaffolding approach scale 10,000 documents across four distinct domains (Computer Science, Physics, Biology. Material Science). This scale is critical for pre-training "foundation" extraction models that can generalize zero-shot.
Taxonomic Depth. Most standard datasets utilize "flat" entity ontologies (e.g., broad categories like Method or Material). SPHERE, being generated from a deep Knowledge Graph scaffold, con- tains nested hierarchical definitions (e.g., Adam Optimizer Stochastic Optimization Optimiza-
16

---

Published as a conference paper at ICLR 2026
tion Method). This distinct structural depth forces models to learn fin-grained hierarchical reason- ing (tested via HGNet) rather than simple surface-level pattern matching.
Structural Consistency and Global Scope. A critical distinction of SPHERE is the scope of its graph topology. Standard benchmarks like SciERC are annotated at the document level, meaning the hierarchical relationships are locally inferred and often inconsistent (e.g., an entity may be root in one document but a leaf in another). In contrast, SPHERE is generated from a Global Knowledge Graph Scaffold containing over 40,000 entities. This ensures that the hierarchical position of concept remains globally consistent across the entire corpus, preventing the "inflated structure" or hallucinated loops often associated with unconstrained LLM generation.
Table 6: Comparison of SPHERE against standard scientific IE benchmarks. A key distinction is Graph Scope: standard datasets define hierarchies locally within isolated documents (fragmented), whereas SPHERE is generated from single Global Knowledge Graph, ensuring hierarchical sistency across the entire corpus.
Dataset
Domain
Does
Relations
Graph Scope
Hierarchy Source
SciERC
CS (AI)
44.6k
Local (Doc-Level)
Inferred from Text Inferred from Text Inferred from Text
BioRED
Biomed
600
388k
Local (Doc-Level)
SciER
CS
106
122
Local (Doc-Level)
SPHERE
Domains
10,000
111,000
Global (Corpus-Level)
Pre-defined Scaffold
The fidelity of the SPHERE dataset is evidenced by its surprising zero-shot efficacy. When trained only on SPHERE, our model generalizes to the aaman-annotated SciERC and SciER benchmarks with scores of 46.55% and 59.17% respectively, outperforming the previous fully supervised state- of-the-art (HGERB). This confirms that SPHERE faithfully models the complex entity-relation de- pendencies of scientific text, validating our constrained generation pipeline.
Table 7: Performance comparison on SciERC and SciER datasets.
Metric
Training Source
SciERC (Test)
SciER (Test)
HGERE Yan et al. 2023) HGERE (Zero-shot transfer)
Full Supervised Training
43.86%
56.28%
SPHERE-CS Training Only
25.62%
28.34%
HGNet (Zero-shot transfer)
SPHERE-CS Training Only
46.55%
59.17%
Manual Quality Audit. To quantitatively assess the fidelity of the SPHERE dataset and ensure minimal hallucination, we conducted a manual verification study on a randomly sampled subset of the corpus. We analyzed 1,000 entity spans and 500 relation triples against the ground-truth topological scaffold. The audit yielded an entity precision of 96.5% (measuring correct boundary and type) and relation precision of 94.2% (measuring correct edge classification). These high precision scores confirm that our constrained "generate-from-grapp pipeline effectively enforces structural consistency while maintaining textual fluency.
A.4 VISUAL EVIDENCE FOR ORTHOGONAL SEMANTIC DECOMPOSITION
To further validate the premise of OSD. Figures 2 illustrate the average Orthogonal Semantic Ve- locity Norm for tokens at entity boundaries versus non non-entity tokens. The plots provide compelling visual support for Hypothesis 3.2 A clear and substantial gap emerges between the high velocity norms of boundary tokens and the low norms of non-entity tokens. This demonstrates that our en- gineered feature effectively captures the sharp "semantic turns" that occur when a new concept is introduced, providing robust, domain indicator of entity boundaries.
A.5 GEOMETRIC REALIZATION VIA AN ABSTRACTION FIELD
Instead of treating abstraction score as an external label to be predicted, our central hypothesis is that the abstraction score should be intrinsic geometric property of the learned embedding space
17

---

Published as a conference paper at ICLR 2026
Average Onthogonal Semanticc velectt Norm Endiyy Boundaries vs, Mon Entities
Austage Orthogora Semantic Welstt Bounderie Man-Ertttie
Oriigine wwweeeeeeeeeee wichy
arch
Figure 2: Average Orthogonal Semantic Velocity for tokens at entity boundaries "Start'/'End vs. 'Non-entity' tokens for SPHERE-CS (left) and SciER (right). The clear separation provides visual evidence for Hypothesis 3.1.
itself. We propose that the entire high-dimensional space can be oriented along single, universal direction that represents continuum from specificity to generality. We formalize this concept as the Abstraction Field Vector.
Definition A.1 (Abstraction Field Vector) a learnable unit vector Wads E RR. This vector de- fines the primary axis of abstraction within the embedding space. The predicted abstraction score, Hata(V)) for any concept y with embedding h, is then simply its orthogonal projection onto this vector:
(v) == h Wabs
(13)
Justification for a Single Universal Axis: The choice to model abstraction with a single unit vector is a deliberate application of simplicity and method for imposing a strong, beneficial inductive bias. While one could model abstraction using multiple orthogonal vectors or more complex non-linear function, such approaches would implicitly assume the existence of multiple, independent "types" of abstraction. Our formulation, by contrast, hypothesizes that the dominant organizing principle of a scientific knowledge hierarchy constraint forces the model to discover the most salient and universal axis of abstraction that is con- sistent across all entities, rather than overfitting spurious, domain -specfff hierarchical patterns. This mirrors findings in other areas of representation learning, where simple linear axes have been shown to capture profound semantic relationships (e.g., the famous 'king man + woman' analogy in word embeddings). By reducing abstraction to single, interpretable dimension, ensure the learned geometric structure not only robust but also directly analyzable. The empirical success of this method across multiple domains serves strong validation for this simplifying, yet powerful, geometric assumption.
single, primary dimension of generality versus specificity. This
This formulation is powerful because it transforms the abstract notion of "generality" into concrete, measurable geometric arrangement. A concept's position along this axis directly reflects its level of abstraction. This approach ensures that the learned hierarchy is not afterthought but the primary organizing principle of the entire embedding space, making the learned representations globally coherent and interpretable. Refer figure [ for visualization of continuum of abstraction in physics domain.
A.6 LEARNED ABSTRACTION SCORE ANALYSIS
To qualitatively assess the geometric structure learned by HGNet, we visualized the distribution of the final abstraction scores for entities within each domain of the SPHERE dataset, as shown in Figure 3 Based on the programmatic, recursive generation of the underlying knowledge graph, the ideal distribution would exhibit an exponential decay, with high density of concrete entities at low abstraction scores and a progressively smaller number of entities at higher levels of abstraction. The analysis reveals distinct, domain-speciff patterns that reflect the inherent structure learned from each field.
18

---

Published as a conference paper at ICLR 2026
The Computer Science domain (d) aligns most closely with this expected pattern, showing clear concentration of entities at lower abstraction values and a long tail of increasingly abstract concepts. In contrast, the Material Science data (a) shows distribution heavily clustered at lower scores, while the Biology data (c) displays more gradual decline, likely reflecting a flatter hierarchy in its source text. It is crucial note, however, that even the Computer Science distribution is not perfect match for the ground-truth hierarchy. The visible deviations from an ideal curve highlight that some concepts
still misplaced along the abstraction axis. These imperfections in the learned geometric structure are precisely what lead to non-perfect Rel+ Fl score, highlighting the tight coupling between representational geometry and task performance.
Distribution Abstraction Scores for Material Science Dataset
Distribution of Abstraction Scores for Physics Dataset
1200
of
0,2
00
02
0.6
10
(a) Material Science Dataset
(b) Physics Dataset
Distribution Abstraction Blology
Distribution
Computer Dataset
1400
1.00
1200
1500
Murbbr breket
1250
13:0
750
60
Abstrastoo
(c) Biology Dataset
(d) Computer Science Dataset
Distribution abstraction scores across different scientific domains, showing distinct patterns for each field.
Figure 3:
A.7 SCALABLE ACYCLICITY REGULARIZATION VIA KRYLOV SUBSPACE METHODS
potential computational bottleneck in our framework is the Differentiable Hierarchy Loss (Eq. 7), which involves the calculation of matrix exponential. For graph with entities, the parent- of adjacency matrix Aurent is of size n n. A direct computation of the matrix exponential, (appeea parent Aurem., scales with time complexity of O(n²), which can become prohibitive for the large-scale knowledge graphs targeted by our work.
exp(A
To ensure the scalability of our approach, this term can be efficiently approximated using Krylov subspace methods (Saad, 2003). Instead of explicitly forming the dense matrix exponential, these iterative methods approximate its action on a vector by projecting the matrix onto low- dimensional Krylov subspace, Km, of dimension <
The computational cost of this approach is dominated by two steps. First, the construction of an orthonormal basis for the subspace, typically via the Arnoldi iteration, requires matrix-vector products. Since the learned adjacency matrix Apmen is inherently sparse, with a number of non- zero entries denoted by nnz (. this step has complexity of O(m nnz anz(Aparem)) Second, the exponential of the small m x m projected matrix is computed directly, which incurs a cost of (m3))
19

---

Published as a conference paper at ICLR 2026
Therefore, the total time complexity for the approximation is (m--nn puremt)) + Furthermore, to compute the trace required by our loss function, Krylov methods is combined with stochastic trace estimators, Hutchinson method (Avron & Toledo 2011), to approximate (()))(A)) without ever forming the full matrix.
Hierarchical Separation Loss The second component, the Hierarchical Separation Loss (Eq. 8), is defined as Exeparation = Lu.ar (A ) (Aparem)un A direct computation would first involve squaring the matrix A parent. an operation that, even for sparse matrices, can be costly as the resulting matrix purent may be significantly denser. However, we can reinterpret this loss as a sum over specific graph structures. The term (A represents the sum of weights of all paths of length two from entity u to w. The loss thus penalizes the existence of direct "shortcut" edge (u, w) when such two-step paths exist. This structure allows for far more efficient calculation. Instead of matrix multiplication, we can compute the sum by iterating through all 2-paths in the graph. An efficient algorithm involves iterating through each node v and considering all pairs of its incoming edges (u, v) and outgoing edges (v, w). For each such 2-path + t w, we perform sparse lookup to check for the existence of the direct edge (u, w) The total complexity of this approach is approximately O(Evev in in-degrec(v) out-degree (D)), which is directly proportional to the local sparsity of the graph and avoids the costly formation of A parent
This analysis demonstrates that both components of the Differentiable Hierarchy Loss can be com- puted efficiently, ensuring that the enforcement of globally consistent DAG structure remains com- putationally feasible even for knowledge graphs containing thousands of entities.
A.8 EXTENDED GEOMETRIC BASELINE ANALYSIS
To validate the efficacy of the Continuum Abstraction Field (CAF) against non- -on-lucldd ap- proaches, we compare HGNet against two strong geometric baselines using the same SciBERT backbone:
HGCN (Hyperbolic GCN) Chami et al. (2019): Maps embeddings to the Poincaré ball manifold, theoretically optimized for hierarchical trees.
Order-Embeddiin Vendrov al. (2015): Enforces partial order constraints via cone geometry (E = (0, =)
Table 8 presents the results. HGNet outperforms both baselines. We observe that HGCN requires extensive tuning of the Riemannian Adam optimizer and often struggles with "Peer" relations that violate strict tree geometries, whereas HGNet's Euclidean CAF objective remains stable and accu- rate.
Table 8: Comparison against Geometric Baselines (Rel+ F1 %). HGNet outperforms non-Euclidean methods on both standard and large-scale hierarchical datasets.
Model
Geometry
SciERC (Overall)
SPHERE-CS (Overall)
HGCN Chami et al. 2019
Hyperbolic (D") Cone (RT)
45.82
66.35
Order-Ember Endeddnmbed Vendrov et al. 2015) HGNet (Ours)
44,27
67.97
Fuclidean + CAF
53.19
79.51
A.9 FEW-SHOT LLM EVALUATION
To ensure a fair comparison regarding reasoning capabilities, we evaluated Llama-3-88 using a 3- Shot Chain-of-Thouulh (CoT) strategy. We provided the model with three context-response pairs demonstrating step-by-step relation extraction before querying the target sentence.
As shown in Table 9 while CoT provides notable performance boost over the zero-shot setting (+5.73% on SciERC), the model still significantly underperforms compared to HGNet. Qualitative error analysis reveals that while CoT helps identifying relation types, the LLM continues to strug- gle with precise entity boundary detection (c.g., including determiners or punctuation the span), which is penalized by the strict Rel+ metric.
20

---

Published as a conference paper at ICLR 2026
Table 9: Impact of Prompting Strategy on Llama-3-8B Performance (Rel+ F1 %).
Model
Prompting Strategy
SciERC
SciER
Zero-Shot 3-Shot CoT
13.72
14,95
Llama-3-8B
19.45
25.18
HGNet
Supervised
53.19
62.36
A.10 QUALITATIVE ERROR ANALYSIS
CORRECTED ERROR: PREVENTING HIERARCHICAL SHORTCUTS
A significant advantage of HGNet is its ability to maintain a strict, multi-level hierarchy by penal- izing "shortcut" edges that skip intermediate levels. This corrects errors where local model might conflate grandparent relationship with direct parent one. Consider a biology paper discussing genetics:
Sentence 1: The SRY gene is responsible for encoding the Testis-determining factor protein.
Sentence 2: A conserved motif within the Testin-det factor protein is the High- mobility group (HMG) box, which binds to DNA.
From these sentences, correct hierarchy is established: (HMG box - Testis-determiing factor protein + SRY gene). However, another sentence might state: "The DNA-binding function of the SRY gene is conferred by its HMG box." A local model, seeing this direct functional link, could incorrectly infer a direct compositional relation: (HMG box, Part-Of, SRY gene). This creates a flawed, flattened hierarchy.
HGNet corrects this error. Its Hierarchical Separation Loss (Loepaattinn) is explicitly designed to prevent this. Once the model identifies the valid two-step path from "HMG box" to "SRY gene", the loss function penalizes the prediction of direct edge between them. This forces the model to respect the intermediate entity ("Testis-determining factor protein"), ensuring the final graph accurately reflects the nested biological structure.
Robustness to Non-Hierarchical Structures. Here, we explain how HGNet behaves when the underlying structure is not strict tree (e.g., multiple inheritance or cross-links). We observe that the Peer pessaae-passing channel is critical in these scenarios. In cases of multiple inheritance (e.g., "Reinforcement Learning" being a child of both "Machine Learning" and "Control Theory"), HGNet successfully assigns high probability to both parent edges because the DAG constraint (Lacyetse) permits multiple parents, only forbidding cycles. However, we note a failure mode in "loopy" citations where definitions circular (A defines B. B defines A). In such rare cases, the acyelicity loss forces the model to arbitrarily break the loop, potentially dropping a valid semantic link.
A.11 JUSTIFICATION FOR THE DIFFERENTIABLE ACYCLICITY Loss
To enforce a Directed Acyelic Graph (DAG) structure, we require differentiable function that pe- nalizes the presence of cycles within the graph represented by the learned adjacency matrix Aparent- Our loss function is built upon a well-established connection between the algebraic properties of graph's adjacency matrix and its topological structure.
The foundation of this approach lies in the observation that the number of distinct walks of length k from node i to node is given by the entry (i,j) of the matrix power A Consequently, cycle, which is a walk from node back to itself, is captured by the diagonal entries. The sum of these diagonal elements, the trace tr( Ar(A*) )), therefore counts the total number of cycles of length k across the entire graph.
A graph is DAG if and only if it contains cycles of any length k 2 1, which implies that tr( Ar(A)) = 0 for all & 2 1. To aggregate this condition over all possible cycle lengths into single, smooth function, leverage the matrix exponential, defined by its Taylor series
21

---

Published as a conference paper at ICLR 2026
exp(A)= kk= A Due to the linearity of the trace operator, we have:
(r(At) tr([) + tr( + tr(A2)
tr(exp(A))
k!
2!
For a graph with d nodes that is a perfect DAG, all trace terms for l 2 1 vanish, causing the expression to simplify elegantly tr(exp(A)) tr(I) d.
Based on this property, our loss function, Lazyelic tr tr(exp( A purent)) - d, is formulated. This ob- jective function non-negative and equals zero only when the graph perfectly acyclic. By mini- mizing this loss during training, computation made efficient by modern numerical libraries such as PyTorch's 'torch.linal corresponding graph structure satisfies the DAG constraint.
eexp', we guide the model to learn an adjacency matrix A parent whose
A.12 PERSPECTIVE: HGNET AS A GENERALIZED ATTENTION MECHANISM
At its core, the self-attention mechanism, which powers modern Transformers, can be understood as form of message passing on a fully connected graph. Each token in sequence acts as node, and it updates its representation by aggregating information from every other token. This very powerful, as it allows the model to capture long-range dependencies. However, it is also a brute- force approach. It operates under the assumption that any token could relevant to any other, leading to two major limitations:
1. Computational Inefficiency: The number of connections grows quadratically with the sequence length, making computationally expensive long documents.
2. Semantic Noise:
In a scientific document, the relationship between the vast majority of token pairs meaningless. Forcing token like "LSTM" to attend every instance of "the" or "is" introduces significant noise and forces the model to expend capacity learning to ignore these irrelevant connections.
The fundamental insight of our work is that we can create far more powerful and efficient reason- ing mechanism by moving from a dense, token-level graph to sparse, entity-level graph. Instead of every word attending to every other word, we want key scientific concepts to attend only other relevant scientific concepts. By "skipping the middle tokens" and operating directly on the meaning- ful entities, we can focus the model's capacity on learning the true global structure of knowledge. Our Hierarchical GNN is the formal embodiment of this principle, representing more advanced and generalized form of attention.
PROOF SKETCH: FROM FULL ATTENTION TO STRUCTURED, HIERARCHICAL ATTENTION
To prove this, let us first formulate the standard self-attention mechanism in the language of Graph Neural Networks.
1. Self-Attention as a GNN on a Fully Connected Graph embedding h; in self-attention layer is:
The update rule for a single token
h
Σ jeVa
asy (h,
(14)
where Vau is the set of all tokens in the sequence, and æı is the attention weight between token and token j. This is precisely a GNN pessagg-passing step where the graph is fully connected, meaning every token is neighbor of every other token. The message from node to node i is its transformed value, mj--i = h, Wv. and the aggregation is a weighted sum, with attention scores serving as the weights. This is powerful but unstructured mechanism. It treats all potential connections as cqually plausible a priori.
2. The Hierarchical GNN as a Generalized, Structured Attention Our Hierarchical GNN intro- duces powerful inductive bias by replacing the fully connected graph with sparse, semantically
22

---

Published as a conference paper at ICLR 2026
meaningful graph, one based the learned hierarchy. The update rule for an entity embedding h.
is:
h (--1) 1)
UpdateMLPP(h (k) m
m
children m
(15)
Let's analyze one of these components, the message from parents:
m parents
parent
paperree h he)
(K))
(16)
va
This is also an attention mechanism, but with three crucial generalizations. First, through Graph Sparsification, the aggregation is no longer over all possible nodes Van but is instead over a small, semantically relevant subset, A/Parcuuts This prunes the vast majority of noisy, irrelevant connec- the relationships that truly matter and directly addressing
tions, focusing the model's attention both the computational and semantic noise problems. Second, instead of single, monolithic atten- tion mechanism, our GNN employs Multi- Multi-Channel Attention with multiple, specialized channels. learns separate projection matrices (W parent, Wehita Weer)) and attention mechanisms for each type of hierarchical relationship, allowing the model learn different "types" of attention. For example, learning to "inherit" abstract properties from parents while "aggregating" specific evidence from children. Third, through Entify-Level Reasoning, the nodes in our graph are not tokens but aggre- gated entity concepts representing stable ideas across the entire corpus. This provides much more robust and global context for reasoning than the ephemeral, documenntsppccfiic context of individual tokens.
The standard self-attention mechanism is a special case of our Hierarchical GNN framework under specific set of simplifying assumptions. These assumptions that the graph is fully connected (N, = Vau for all v), that there is only one message- passing channel (c.g., only a "peer" channel), and that the nodes represent tokens, not global entities. By relaxing these assumptions, our Hierarchical GNN generalizes the attention mechanism to operate on sparse, structured, multi-channel graph of global concepts. This not merely an incremental improvement; it is a fundamental shift from brute-force pattern matching to structured, hierarchical reasoning. It allows the model to capture the kind radial, layered knowledge depicted in the conceptual image 4 of the scientific domain, making it a far more powerful and efficient architecture for understanding complex, interconnected information.
Conclusion of Proof
Tapckig
MAR MARrSEN
kaice Deliciann
Trokegey
Wat
LED SOE
DRFAA TE EXNEECT
LED dege
QUREAT TRQNISEDE
OPE AeNiio
OF
Figure 4: Example of physics domain hierarchical knowledge graph. Hierarchy radially extends outward.
A.13 DISCUSSION ON CONVERGENCE OF HGNET
Given that HGNet is probabilistic model, it's important to understand why its predicted proba- bilities converge toward a consistent graph structure rather than fluctuating randomly. The primary
23

---

Published as a conference paper at ICLR 2026
HGNet Structural Loss Functions (Computer Science)
Asyckiilly Loes
Separation
CAT Loes
0.6
Loss Valuoe
0.5
0.3
10
15
17
Training Epochs
Figure Loss plot of HGNet on computer science SPHERE dataset.
reason is that the model is not initialized from scratch. Instead, it uses standard MLP classifier to generate the initial edge probabilities between entities.
As demonstrated by the baseline models in our experiments, which rely on an MLP for classification, these approaches are reasonably effective, often achieving Rel+ scores exceeding 30-40% on their By using this starting point, HGNet's probabilistic message-passin well-informed "draft" of the graph. This process is far more efficient than random initialization; it's like solving a jigsaw puzzle where significant portion of the pieces already in their approximate correct locations, allowing the model to focus on refining the details rather than building the entire structure from scratch. Refer loss plot 5
begins with
A.14 COMPUTATIONAL COMPLEXITY AND EFFICIENCY ANALYSIS
We conduct a comprehensive analysis of parameter efficiency, computational cost (FLOPs), and inference throughput to validate our lightweight claims.
Efficiency Generalization Landscape. Table 10 benchmarks HGNet General-purpose LLMs, Specialized SOTA methods (PL-Marker, HGERE), and lightweight Graph Neural Networks (GCN, GAT), HGNet occupies unique "sweet spot": it matches the generalization of LLMs while maintaining the throughput of specialized models.
Component- Wise Parameter Breakdown. Table IT details the parameter distribution of the full HGNet pipeline. We employ a two-stage architecture (Z-NERD and HGNet) where decoupling im- plies the worst case parameter setting. Notably, the Z-NERD stage architecturally heavier (42.4M trainable params) due to the Multi-Scale TCQK mechanism, which employs 8 parallel convolutional heads with wide projection matrices (dproj = 2048) to capture dense n-gram contexts. The HGNet stage utilizes lighter, structure-awee GNN (31.6M trainable params) reason over the sparse entity graph.
Training Overhead. Structural losses (DHL/CAF) are not computed during inference. During training. the Krylov subspace approximation reduces the exact matrix exponential calculation time from ~ 150ms ~m2ms per batch, rendering the overhead negligible (< 5% total training time).
24

---

Published as a conference paper at ICLR 2026
reittiity provides explanatory priniples many argue quantum mechanics employed geometry
SOIBERT
Orthogonal Semantic Decompositon
beddidding
&E
Eun, Eirn.::
Emma Foreet Feo.. F
Caverreen AE Vaunsiiiing
Novel
TOQK
Projecter
Convosved
PenoolzionKomel
Final Embedding
Altention Scores convolutions
TTg
MLP Classiffer
TTag
O-fag
General rdlatiiity proviees povvisttpoooviess explanatory principlos that many researchers argue riguequantum mechanic erpploydd refine, within the regomeeetyy spocotimm
Theory
Theory
Concept
Figure 6: Main figure explaining the proposed Z-NERD algorithm. For TCQK, multi-head for each convolution has been shown as single head for simplicity. B refers to begin entity, I refers to inside entity and O refers outside entity.
25

---

Published as a conference paper at ICLR 2026
Latent Relation Predictor
Entlly Embeddingg
gradients backprop
Hlerarchical Loss
I Laamabia
MIPP
Novel
C Caterrcty Excyete
Cupmettee
PEATTS
Apanet
Wep
W.sw
Waswn
Hiororchy
UpdateM
LP
Loes
Apen
Wawe
Poer
my
Probabiliistic Message Passing
.... hierarchy
hierarchty
peers
pears acyclicity
Acyclic
Separation
Custa:
Figure 7: Main figure of proposed HGNet illustrating all proposed components. For clarity, we omit Fregression since simply a regression loss applied over the graph topology, similar nature to standard losses such mean squared error or binary cross entropy.
26

---

Published as a conference paper at ICLR 2026
Table 10: Efficiency Landscape on SciERC (A30 GPU, Batch=8). GFLOPs are estimated per input instance. HGNet (Full Pipeline) provides superior throughput compared pipeline-based SOTA models despite robust parameter capacity.
Model
Params
GFLOPs
Speed (doc/s)
Mem (GB)
Zero-Shot Gen.
Large Language Models
Llama-3-70B
70B
140k >> 16k
00.
OOM 16.0+
High Moderate
Llama-3-8B
~BB
444
Specialized SOTA
PL-Marker
220M
44.0
12.4
7.2
Low
HGERE
2200
22.5
14.1
9.5
Low
Graph Baselines
SciBERT+GCN
1110
22.0
48.2
6.1
Low
SciBERT+GAT
1100
22.1
46.8
Low
Proposed
HGNet
293M
44.7
14.6
High
Table 11: Detailed parameter breakdown of the HGNet framework (Two-Stage Configuration, worst case parameters). The Z-NERD stage incorporates high-capacity TCQK attention (8 Heads) to resolve complex morti-word boundaries, while HGNet utilizes specialized cessage--asssigg layers.
Stage
Component
Params (M)
% of Total
Specialized SciBERT Encoder Multi-Scale TCQK (8 Heads)
109.5
37.4%
Stage Z-NERD
42.4
14.5%
Specialized SeiBERT Encoder
109.5
37.4%
Stage 2: HGNet
Hierarchical GNN Layers
31.6
10.7%
Total
Full Two-Stage Pipeline
~99300
100%
27


Reviews:
[{"review_id": "zkGqHy1ryO", "reviewer": "Reviewer_NQq1", "summary": "The paper tackles the problem of generating KGs from scientific literature, focusing on four challenges that require more care than in general purpose KG construction: \n(i) careful extraction of multi-word scientific entities, \n(ii) domain generalization for this task of scientific NER, \n(iii) hierarchical relation extraction, and \n(iv) logical consistency in relation extraction (e.g. avoiding cycles).\n\nTheir contributions are: \n(i) a novel methodology for NER that looks for signals of semantic shifts to more robustly extract entities across domains and that incorporates convolutional filtering at different scales to specifically look for entities with different lengths, \n(ii) a novel methodology for RE that uses an explicitly hierarchical GNN to learn entity embeddings, a differentiable loss to encourage the KG to be a DAG, and losses that explicitly encourage the entity embeddings to encode hierachical relationships along a specific axis, and\n(iii) a novel large multi-domain scientific KG benchmark data set", "questions": "- In equation 7, what is d?\n- In section 3.2.3 by pinning roots and leaves to 0 and 1, and requiring a delta margin between parents and children, doesn't this undesirably limit the number of levels of abstraction you could have to 1/delta?\n- where do you get ground truth topological depth scores in (11)?\n- In HGNet you have a message passing system with latent relational predictors, the differentiable hierarchy loss, and the hierarchical separation loss, as well as the continuum abstraction loss -- how do these pieces fit together and how are they co-trained? \n- The description of HGNet in section 3.2 only mentions taking the entities from the NER stage as input, but the scientific documents must be used in predicting the relationship between two entities. This is not mentioned anywhere in section 3.2. E.g. the latent relational predictor only take the entity embeddings as input.\n- The description of HGNet in section 3.2 only mentions extracting relations of the type {parent-of, peer-of, no-edge}. Where in the KG generation does one extract the usual <head, relation, tail> triplets?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The proposed ideas are well presented in isolation, e.g. the several hypotheses that the authors present to justify their system design seem reasonable. \n- The ideas seem original in the area of scientific KG generation to me, and some are likely to be influential, especially the NER system and the use of the differentiable loss to ensure a logically consistent KG", "weaknesses": "- The architecture of the system is not sufficiently described in the main body of the paper to understand or reproduce the system. E.g. sections 3.1 and 3.2 that provide the methodologies for the NER and RE do not mention a specific architecture; rather they mention losses and modifications to attention heads. One must read the appendix to begin to understand the system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uMF0vcVWwG", "reviewer": "Reviewer_NX6E", "summary": "In this study, the authors proposed a two-stage framework for scalable, zero-shot scientific KG construction to resolve and mitgate the challenges of knowledge graph (KG) construction from massive literature data (especially for the challenge of long multi-word entity recognition. \nThe evaluation results showed that the proposed model improved the entity recognition and relationship extraction reliablly.", "questions": "How fast is the proposed model for processing massive literature data?\nHow to build complete sets of graphs of abstract accross diverse domains? Which can be important for learning/understanding the domain knowledge.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear and appropriate model design to resolve the specific challenges in KG construction from literature data\nSolid experimental evaluation with reliable improvement.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "igoxd2mIG0", "reviewer": "Reviewer_GYoP", "summary": "The paper proposes a two‑stage framework for scientific KG construction: Z‑NERD for zero‑/low‑shot entity recognition via Orthogonal Semantic Decomposition (OSD) and a Multi‑Scale TCQK attention that specializes heads to different n‑gram lengths, and HGNet for hierarchy‑aware relation extraction with three message‑passing channels (parent/child/peer) and two global regularizers: a Differentiable Hierarchy Loss (acyclicity + shortcut penalties) and a Continuum Abstraction Field (CAF) that learns a single Euclidean “axis of abstraction.” Results show new SOTA on SciERC/SciER/BioRED and on a new multi‑domain SPHERE benchmark; Table 1 (p. 8) and Tables 2–4 (p. 9) report average gains of +8.08 F1 (NER) and +5.99 F1 (RE), with larger zero‑shot gains (+10.76 / +26.2). Figures 5–6 (pp. 20–21) illustrate Z‑NERD/ HGNet; Appendix A.7 details scalable acyclicity via Krylov methods.", "questions": "- How does HGNet behave if the true structure is not strictly hierarchical (e.g., multiple inheritance, cross‑links)? Any failure cases beyond A.8?\n- SPHERE validation: What fraction was human‑audited? Provide inter‑annotator agreement and a human‑written subset to test robustness to LLM‑style.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- TCQK introduces architectural inductive bias for multi‑word entities; CAF imposes a simple, interpretable Euclidean ordering of abstraction; DHL neatly encodes DAG and anti‑shortcut constraints. \n- Solid ablations (Tables 1–3) and complexity notes for DHL \n- Clean factorization of challenges (entity coherence, domain generalization, hierarchy, global consistency) and matching components.\n- Consistent gains on SciERC/SciER and large zero‑shot improvements on SPHERE", "weaknesses": "SPHERE is generated and self‑annotated by LLMs; may encode stylistic biases, inflated structure, or task leakage that favors the proposed inductive biases. Though Limited human validation is described. (App. A.3.)  \n-  Large LLMs are evaluated “as is” (several OOM), and there is no strong hyperbolic/order‑embedding baseline for hierarchy—weakening the “simpler & better than hyperbolic” claim.\n- CAF relies on anchors and topological depths—procedure unclear for standard datasets; OSD’s learning objective (beyond feature concatenation) is under‑specified.\n- No runtime/throughput or memory comparisons despite “lightweight” claims; parameter counts not tabulated per component. Report wall‑clock, FLOPs, and memory vs. PL‑Marker/HGERE and vs. GCN/GAT; quantify DHL/CAF overhead and benefits from the Krylov approximation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)