GraphMERT: Efficient and Scalable Distillation of Reliable Knowledge Graphs from Unstructured Data
Authors: Margarita Belova
Jiaxin Xiao
Shikhar Tuli
Niraj Jha
Venue: Accepted by TMLR ()

Abstract: Researchers have pursued neurosymbolic artificial intelligence (AI) applications for nearly three decades. Thus, a marriage of the two components can lead to rapid advancements in AI. Yet, the field has not realized this promise since most neurosymbolic AI frameworks fail to scale. In addition, the implicit representations and approximate reasoning of purely neural approaches limit interpretability and trust. Knowledge graphs (KGs), a gold-standard representation of explicit semantic knowledge, can address the symbolic side. However, automatically deriving reliable KGs from text corpora has remained an open problem. We address the above challenges by introducing GraphMERT, a tiny graphical encoder-only model that distills high-quality KGs from unstructured text corpora and its own internal representations. Together, GraphMERT and its equivalent KG form a modular neurosymbolic stack: neural learning of abstractions; symbolic KGs for verifiable reasoning. GraphMERT + KG is the first efficient and scalable neurosymbolic model to achieve state-of-the-art benchmark accuracy along with superior symbolic representations relative to baselines. More concretely, we target reliable domain-specific KGs that are both (1) factual (with provenance) and (2) valid (ontology-consistent relations with domain-appropriate semantics). When an off-the-shelf large language model (LLM), e.g., Qwen3-32B, generates domain-specific KGs, it falls short on the reliability front due to prompt sensitivity, shallow domain expertise, and hallucinated relations. Thus, practitioners should avoid employing LLM-generated KGs in high-stakes domains, e.g., medicine, law, business, education, etc. On text obtained from PubMed papers related to diabetes, our KG extraction pipeline with a small 80M-parameter GraphMERT yields a KG with a 69.8% FActScore; a 32B-parameter baseline LLM yields a KG that achieves only a 40.2% FActScore. The GraphMERT-extracted KG also achieves a significantly higher ValidityScore of 68.7%, compared to an LLM-generated baseline (43.0%), demonstrating its ability to preserve ontology alignment. KG cleaning further improves factuality, with GraphMERT reaching 76.9% FActScore, compared to 55.6% for the LLM baseline. GraphMERT can then treat the augmented KG as the seed KG and refine it further. Finally, human experts can edit and audit the extracted KGs, further increasing their reliability. This is nearly impossible with purely neural representations. Hence, GraphMERT enables efficient, scalable, transparent (interpretable and explainable), attributable (with provenance), accountable (with governance), editable, auditable, and continually improvable state-of-the-art neurosymbolic AI. The code is available at https://github.com/jha-lab/graphmert_umls

Shikhar Tuli Published in Transactions on Machine Learning Research (02/2026)
GraphMERT: Efficient and Scalable Distillation of Reliable Knowledge Graphs from Unstructured Data
Margarita Belova
margorita.heldprinnetonncm
Deportment of Electrical and Computer Engineering, Princeton University
Jiaxin Xiao
jr08000@princeetnn.eee
Department of Electrical and Computer Engineering, Princeton University
stulf@nokummii..rincctoo..cm
Department of Electrical and Computer Engineering, Princeton University
Niraj K. Jha
jha@princeton.edu edu
Department of Electrical and Computer Engineering, Princeton University
Reviewed on OpenReview: https: // openreview. net/ forun? id- taXSdDhuge
Abstract
Researchers have pursued neurosymbolic artificial intelligence (AI) applications for nearly three decades. A marriage the neural and symbolic components can lead to rapid ad- vancements in AI. Yet, the field has not realized this promise since most neurosymbolic AI frameworks fail scale. In addition, the implicit representations and approximate reason- ing of purely neural approaches limit interpretability and trust. Knowledge graphs (KGs), a gold-standard reprosentation of explicit semantic knowledge, can address the symbolic side of the problem. However, aulomatically deriving reliable KGs from text corpora remains open problem. We address the above challenges by introducing GRAPHMERT, tiny graph- ical encoder-only model that distills high-quality KGs from unstructured text corpora and its own internal representations. Together, GRAPHMERT and its equivalent KG form a modu- lar neurosymbolic stack: neural learning of abstractions; symbolic KGs for verifiable reason- ing. GRAPHMERT + KG the first efficient and scalable neurosymbolic model to achieve state-of-the-art benchmark accuracy along with superior symbolic representations relative to baselines. More concretely, we target reliable domain-specifie KGs that both (1) fac- tual (with provenance) and (2) valid mattology-conisstenn relations with domain-appropriat semanties). When an off-the-shelf large language model (LLM), e.g., Qwwe3-332B, generates domain-specifi, KGs, it falls short on the reliability front due to prompt sensitivity, shallow domain expertise, and hallucinated relations. Thus, practitioners should avoid employing LLM-generatod KGs in high-stakes domains, c.g., medicine, law, business, education, etc. On text. obtained from PubMed papers related to diabetes, our KG extraction pipeline with -ar-maraeeter
small 80M- Mar-ppareettrr GRAPHMERT yields KG with 69.8% FActScore; baseline LLM yields KG that achieves only 40.2% FActScore. The GRAPIMERTT- extracted KG also achieves a significantly higher ValidityScore of 68.7%, compared to an LLM-generated baseline (43.0%), demonstrating its ability to preserve ontology alignment. KG cleaning further improves factuality, with GRAPIMERTT reaching 76.9% FActScore, compared to 55.6% for the LLM baseline. GRAPHMERT can then treat the augmented KG as the seed KG and refine further. Finally, human experts can edit and audit the extracted KGs, further increasing their reliability. This is nearly impossible with purely neural representations. Hence, GRAPIMERT enables elficient, scalable, transparent (inter- pretable and explainable), attributable (with provenance), accountable (with governance), editable, auditable, and continually improvable state-of-the neurosymbolic Al. The code available at https://github..oo htatttuuu...mm/baaaabb.... http://aallaagggahpeee...t
graphmert. umls.
Indez Terms: Hallucinations, interpretability, knowledge graphs, language models, neu-
rosymbolic methods, retrieval-augmenteed generation.

---

Published in Transactions on Machine Learning Research (02/2026)
1 Introduction
Artificial intelligence (AI) has long oscillated between two dominant paradigms: symbolic reasoning and neural learning (Shavlik et al., 1991; d' A'ivila Garcez el al., 2002). Symbolic systems excel at explicil (rule- based) inference, providing interpretability and strong exact reasoning. This assumes precise and consistent symbolic abstractions. However, such systems struggle with noisy or ambignous data. Neural approaches, by contrast, thrive on large-scale and general-purpose paltern recognition, oflen outperforming hand-coded explicit representations (Sutton, 2019). Nevertheless, neural networks operate as black boxes, offering lit- tle transparency in their decision-making (Sharkey al., 2025). Their representations are approximate: ambignous, difficult to control, and not grounded in explicit rules. Thus, each paradigm, taken alone, has critical gaps. Neurosymbolic AT is synthesis of the two paradigms, aiming to combine the flexibility of neural models with the rigor and interpretabilityy of symbolic systems. By uniting these complementary strengths, it opens up a path toward systems capable of both scalable lcarning and sound reasoning - a longstanding ambition of the field (Garcez & Lamb, 2023; Towell, 1994; d' Avila Garcez et al., 2019).
Large language models (LLMs) have generated enormous excitement, but their reasoning is ultimately prob- abilistic and often unable to perform causal inference, opaque to humans, and prone to hallucinations, especially during multi-step reasoning (Marcus, 2018; Ji et al., 2023). Their opaqueness raises serious con- cerns about whether such models can be trusted (von Eschenbach, 2021; Alzubaidi et al., 2023). LLMs trained on general text corpora may also fail to adapt to specialized domains or incorporate new knowl- edge without undergoing expensive retraining (Zhao et al., 2025). These limitations highlight the need for external, explicit sources of factual grounding in high-stakes use cases (?).
Unifying knowledge graphs (KGs) with LLMs could help overcome some of these limitations, as KGs offer natural complement (Tbrahim et al., 2024). By encoding knowledge in structured, symbolic form of teaa- triples with explicit and verifiable relations, KGs offer interpretability, auditability, and domain-specific depth that LLMs lack (Pan et al., 2023a). Thus, KGs can guide LLM inference and enable robust evolution of background knowledge, while LLMs contribute flexible reasoning, efficient handling of ambiguity through approximate inference, and serve as a natural language interface (Pan et al., 2023b). This synergy can enable KG-guided exploration and learning, support agentic workflows driven by interaction with an editable knowledge base, and enhance the trustworthiness of LLMs in high-stakes application domains by reinforcing factuality and enabling immediate knowledge updates.
Yet, constructing KG from scratch in a new domain is a notoriously arduous task (Zhong et al., 2023; Ji et al., 2022). The process typically involves cleaning and pre-processing heterogeneous data, multi-step knowledge acquisition, and post-processing. Ensuring that the resulting KG is reliable and factual is even more challenging, often requiring extensive manual inspection and crowdsoureing (Wang et al., 2021). In fields where no ground-truth KG or domain-specifie benchmarks exist,, this task becomes infeasible without expert intervention. This makes existing approaches unscalable.
Given these obstacles, we propose characteristies that an effective KG construction method should provide:
1. Factuality and provenance: Triples grounded in source text with verifiable citations.
Validity: Adherence to ontological schema constraints, appropriate term granularity, and domain- correct relations.
3. Automation: End-to-end extraction without expert oversight; usable by non-specialists.
4. Scalability: Robust performance data volume and compute grow.
5. Domain generality: Principles that transfer across subject areas.
6. Global integration: Cross-document linking of concepts, not just within local spans.
We refer to KGs that are both factual and valid as reliable; hence, we classify methods that produce factual and valid triples as reliable.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Recently, KG generation with decoder-based LLMs has become the most widely used method (Xu et al., 2024), especially for commonsense knowledge extraction (West et al., 2022). Nonetheless, we demonstrate that LLMs are unable to construct reliable domain-specific KGs. The factuality challenge faced by LLMs intersects with several other pressing issues, among them hallucinations, outdated knowledge, and lack of depth in given domains (e.g. in health, law, and finance applications) (Wang et al., 2025a).
Despite their ease of use, versatility, broad knowledge coverage (Minaee et al., 2025), LLMs as KG construc- domain-specifie datasets (Zhu et al., 2024) and are susceptible to domain-irrelevant
tors underperform noise in the context (Chen et al., 2024) and hallucinations (Ghanem & Cruz, 2025), often resulting in inclu- sion of fabricated facts (Huang et al., 2025a). Since LLMs are oblivious to their training sources (Khalifa et al., 2024), KG distillation from weights of an LLM obfuscates knowledge provenance (Pan et al., 2023a). Fine-tuned models exhibit enhanced accuracy and extraction fidelity in the domains they are adapted to. However, fine-tuning requires labeled training data and negatively impacts generalization in heterogeneous domains (Ghanem & Cruz, 2025). Finally, KG extraction from text corpora with off-the-shelf LLMs is not global in the sense that the extracted triples are confined to single text chunk presented in the context window.
The current approaches fail to meet all six requirements for KGs we listed above (Hofer et al., 2024). A method that satisfies all six could unlock numerous downstream applications. However, a reliable KG can be extracted only if the data are of high quality (Rejeleene et al., 2024; Geiger et al., 2020; Wang et al., 2023). Yet, such data are scarce. This leads us to the central question:
How can we build a reliable domain-specifie KG from limited high-quality sources?
To address this challenge, we propose a novel framework, GRAPHMERT (Graphical Multidirectional Encoder Representations from Transformers), for reliable KG extraction from small high-quality domain-specific data, in domains where no large expert-annotated text corpora for triple extraction is available. GRAPIMERT relies on an encoder-only transformer that distills a symbolic representation from its weights. It jointly learns cross-modal representations: semantic - from a small expert-curated initial seed KG, and syntactic from unstructured sentence-level text by minimizing the standard masked language modeling (MLM) and the proposed masked node modeling (MNM) losses. We automatically extract a KG that captures factual knowledge by training GRAPHMERT on high-quality texts with a small seed KG (e.g., 100+ triples per relation). The framework is domain-agnostic and only requires seed KG along with a small, high-quality, domain-specifie dataset (e-g., 100M tokens). To the best of our knowledge, GRAPHMERT powwred powwree KG extraction is the first fromework that possesses all sir characteristics of an effective KG mentioned corlier:
Factuality and provenance: We implement triple extraction at the sentence level. One can trace back each triple to its source sequence, thus supporting knowledge provenance. FActScore, which quantifies the factuality of KG (more details in Sec. 6.2), for the GRAPHMERT- (69.8%) is much higher than that of an L.M-ggenerated one (40.2%).
KG
2. Validity: The resulting KG preserves the relation usage patterns imposed by the ontological structure of the seed KG, which enhances the validity of the relations in the extracted KG relative to the L.M-- extracted baseline KG. ValidityScore, which quantifies the ontological alignment of a KG (more details in Sec. 6.2), for our KG (68.8%) is much higher than that of the bascline (43.0%).
3. Automation: It does not need manual feature selection, rule handcrafting, or human experts in the loop. It leverages ceural-to-symbolit converter, ie., GRAPHMERT, to automatically and reliably generate a KG.
4. Scalability: We obtain training data only from credible sources and the compact GRAPHMERT (with just 80M parameters) eliminates the need for pretraining on large unverified text, making the approach much more practical than employing expensive LLMs (with billions or trillions of parameters). It can be scaled when provided with more data and given extra compute resources.
Domain generality: It relies on donain-agnostic principles. We do not hard-code any domain-specific parameters to the proposed GRAPHMERT pipeline.
3

---

Published in Transactions on Machine Learning Research (02/2026)
6. Global integration: It can connect global concepts across the whole dataset throughout training, in contrast to extracting disconnected information from isolated text.
The rest of the article is organized follows. In Sec. 2, review different KG extraction techniques and motivate the need for reliable KG. In Sec. 3, provide brief motivational example. In Sec. 4, we give detailed overview of our proposed GRAPIMERT framework and its architecture. In Sec. 5, describe the experimental setup. In Sec. 6, we provide experimental results. In Sec. 7, we discuss the limitations of our methodology and discuss future work. Finally, we conclude in Sec. 8.
2 Background and Related Work
In this section, we review prior research that is relevant to our work, including applications of neurosymbolic AI and KGs. We then examine existing KG extraction methods, their limitations, and how our framework addresses these gaps. Finally, we provide the technical background on graph transformer architectures that is necessary understand the remainder of this work.
2.1 Structured Representation Learning
The paradigm of extracting interpretable or geometric structures from data and subsequently leveraging them for downstream tasks foundational strategy in AI. While modern deep learning often relies on end-to-end processing where intermediate representations remain opaque (Bengio et al., 2014), a growing body of literature advocates two-stage paradigm in which structured representations are first derived from raw inputs and then employed to support downstream reasoning decision-making (Battaglia et al., 2018; Bronstein et al., 2021). This method first distills raw inputs into structured, often interpretable, representations, which are then used to guide reasoning or decision-making.
In the visual domain, research demonstrates that explicitly extracting prototypical parts or geometric struc- tures is crucial for robust performance (Chen al., 2019; Kadkhodaie al., 2024). This strategy has proven equally vital in natural language processing (NLP). For instance, dependency trees are often extracted to guide graph neural networks, enabling them to capture long-range semantic dependencies for relation extrac- tion (Zhang et al., 2018). Similarly, rationale extraction frameworks enforce interpretability by first selecting relevant text segments before making a prediction, ensuring decisions are grounded in explicit evidence (Lei et al.). In the realm of complex reasoning, neuro-symbolic methods retrieve and use subgraphs from ex- ternal knowledge bases to augment language models, effectively grounding their answers structured facts (Yasunaga et al., 2021). Furthermore, causal inference, discovering the underlying causal graph often a prerequisite for correctly estimating treatment effects and ensuring that models remain robust distribution shifts (Schölkopf et al., 2021).
This discussion converges
central insight: Explicit structure improves robustness, interpretability, and reasoning. KGs, in particular, provide a symbolic representation that captures rich semantic relations across text, enabling broad applications (Appendix A).
2.2 Knowledge Graphs
A KGGG G = (V, E) can be viewed as directed graph where nodes V represent real-world entities and directed odges E C CVV XxVV represent relationships between them (sce Fig. for a toy example). Each directed edge = (u, v) E E connects two nodes u, u E V and encodes a relationship r between the corresponding entitios. Semantically, a KG can be thought of as a set of triples g (h,r,4 1) (head, relation, tail), where head and tail denote KG entities connected by directed relation.
2.3 The Representation Dilemma: Neural or Symbolic?
Traditional AT research overwhelmingly associates reasoning with purely symbolic systems, including expert systems (van Melle, 1978; Lindsay al., 1993) or logic-based AI (MeCarthy, 1980; Colmerauer & Roussel, 1996). For decades, this paradigm shaped AT practice under the premise that human intelligence could be
4

---

Published in Transactions on Machine Learning Research (02/2026)
Metformin
ACTIVATES
TREATS
Type 2 Diabetes
AMPK
INHIBITS
ASSOCIATED WITH ASSOCIATED WITH
Increased hepatic Gluconeogenesis
Chronic Kidney Disease
Figure 1: A toy KG example from the medical domain.
reduced to formal logic operating on symbols (Newell & Simon, 1976; Haugeland, 1985). Symbolic methods offer clarity and structure by explicill encoding rules over discrete concepts (Newell & Simon, 1976) and are reliable, given suitable abstractions. The symbolic approach governed the Al field till the 90s, when its drawbacks bocame evident: Symbolic systems struggle with ambiguity, contextualization, and the fluidity of real-world knowledge (Harnad, 1990). In addition, computational complexity limits scalability of systems that are already prone to brittleness: Complete symbolic grounding of knowledge base leads to a worst-case combinatorial explosion (Chen & Suen, 1994).
In contrast, neural approaches rely on multidimensional embeddings as their representations and approximate knowledge grounding through gradient-based learning over a continuous parameter space (LeCun et al., 2015). They are robust against outliers and inaccuracies in data, and scale learning and inference well (Hitzler et al., 2022). Neural systems are efficient learners but forfeit transparency: Their decision pathways remain opaque and lack the verifiable interpretability of symbolic inference (Tran et al., 2025). They tend memorize, thus undermining reliable generalization beyond observed facts, especially in out-of-distribution domains. Furthermore, while probabilistic and approximate inference accommodates ambiguities, it yields imprecise logical inference.
The central question remains: Which representation best fits a given task? Due to these complementary advantages and limitatious, researchers are increasingly focused on neurosymbolic integration (Tran et al., 2025) for complex AT challenges that cannot be solved by either approach individually. To bridge this divide, KGs serve a natural interface that connects implicit neural learning with explicit symbolic reasoning.
Building this perspective, next illustrate how KGs function as complementary infrastructure in sen- sitive domains. In sensitive domains, auditable and editable KGs can serve persistent knowledge base: facts can be inspected and updated directly (Pan et al., 2023a). LLMs, by contrast, embed knowledge im- plicitly in parameters, making tracing and verification difficult (Akyurek et al., 2022). Updating LLMs is resource-intensive, requiring fine-tuning or retrieval-augmented generation (RAG), and resolving contradic- tions demands complex mothods like context-aware dccoding. Verification of generated content against a factual KG can help reduce the risk of inaccuracies hallucinations (Luo et al., 2024; Hron al., 2024). Furthermore, a KG offers the "ability to forget": Information can, in principle, be erased if required by legal regulations upon user request. By contrast, removing knowledge from LLM requires complex interventions and sophisticated strategies that risk catastrophic unlearning (Si et al., 2023), raising concerns about access to harmful content, user privacy, and copyright violations (Tian et al., 2024). However, since KGs themselves can be incomplete or biased, their usage in critical fields requires careful construction and maintenance.
For an extended discussion on how KGs unify these representations, and an overview of their applications in neurosymbolic frameworks, refer the reader to Appendix A.
5

---

Published in Transactions on Machine Learning Research (02/2026)
2.4 Existing KG-extraction Methods
Next, we briefly review existing KG-extraction methods. For further details, we direct the reader to comprehensive review on KG construction (Zhong et al., 2023). We discuss the following categories: (1) task-specific NLP methods, (2) triple embeddlingbbsee,, and, the most recent, (3) generative, or LLM-based, which we discuss in grealer detail, given its current prominence.
Given the growing attention to LLM-driven KG construction (Zhu al., 2024), we further differentiate KG estraction from KG generation, clarify their respective advantages, limitations, and application scopes. We classify methods where an LLM plays a pivotal role in the KG construction pipeline, either generating triples conditioned on input texts or distilling knowledge from model weights, under the category of KG gencration. We refer to all other approaches as KG extraction.
2.4.1 Task-specific NLP Methods Scale Badly
Early rule-based information extraction systems demanded heavy feature engineering and domain expertise. Modern pipelines sequentially chain machine learning components- naman entity recognition, coreference resolution, and relation extraction (Jaradeh et al., 2023) often relying on structured or semi-structured data (Hofer et al., 2024). These systems require sophisticated text preprocessing heuristics (Rao et al., 2024), e.g,, in the case of conditional random fields. Traditional fine-tuned BERT for relation extraction segueecce-o--equeee
generation relies on large labeled corpora. As consequence, it requires substantial data annotation in each new domain. Long short-term memories and convolutional neural networks introduce locality bias. Another drawback is that errors propagate over the pipeline. Overall, these methods can be accurate, but are very aabor-intessive, not fully automatic, and hard to scale.
2.4.2 Triple Embeddings are Local,
and Miss Long-range Dependencies
An embeckding-based approach seeks to train ML methods on KGs to capture semantic and structural patterns of the graph into embeddings by optimizing a scoring function. Embeddings enable the model to prodict missing links (triple completion) and estimate the likelihood of new relations (link prodiction; Cheng al. 2024). However, this approach suffers from several limitations, inchuding selection bias, lack of scalability, brittleness to KG errors, and limited external/worde knowledge (Xia et. al., 2025). Because most KG embedding models operate on local triple patterns, they struggle to compose long multi-hop chains, handle negation, or respect ontological constraints - in particular, when relations are n-ary, qualified (e.g., temporal, provenance), or context-dependent. They also assume a largely closed-world, static graph: Cold-start entitics entitics/relations and evolving KGs typically require expensive retraining or ad-hoc heuristics, and performance degrades under distribution shift. More concretely, embedding- based approaches face the following limitations.
Sparsity, limited information, and vocabulary: The scale of the largest publicly available KGs, e.g., Wikidata (118M+ entities), PubGraph (385M+ entities; Ahrabian et al. 2023), is of the order of 10 entities. The scale is incompatible with text corpora sizes: In 2024, top-tier LLMs reported up to 10"3-token training datasets (Villalobos et al., 2024), and the pretraining corpora of leading projects can collectively surpass 700TB (Liu et al., 2024b).
Insufficient utilization of semantic information: Learning an embedding representation that incorpo- rates equally good graph structural and semantic information remains challenging. Multiple efforts are tar- geted at developing architectures and approaches that produce embeddings that are not overly localized (Rao & Wang, 2023), incorporate multiple relation types (Lu et al., 2023), and better integrate contextual (includ- ing semantic) information for improved reasoning (Shi et al., 2025). This suggests that embedding methods are useful in task-specific applications based on KGs, but on their own, fall short in extending KGs.
Gencralizability Embodding methods do not generalize well across different KGs. Each KG charac- terized by its own set of relations, attributes, and ontology, making it impossible or impractical to unite many KGs for training. In practice, embeddings entangle schera-specific signals (relation vocabularies, type

---

Published in Transactions on Machine Learning Research (02/2026)
hierarchies, qualifier formats). Hence, representations learned on one KG transfer poorly to another with different ontologies or naming conventions. Cross-KG use then requires costly alignment steps - entity linking, relation mapping, and negative-sampling redesign - and even after alignment, out-of-vocabulary entities/rolationn and schema drift often degrade performance (Chen et al., 2023b).
2.4.3 LLM-based KG Generation is Not Reliable
With the tremendous success of LLMs on all kinds of NLP tasks, modern rescarch on KG extraction is skewed towards extracting relational knowledge from LLM weights using prompts (Zhu et al., 2024; Carta et al., 2023; Li et al., 2025; Gupta et al., 2025). The widespread adoption of LLMs can be attributed to their versatility across tasks, adaptability to diverse domains, and ease of use, which together make them attractive as general-purpose AI systems. LLMs capture relational knowledge unevenly: They are more accurate for some types (e-g,, commonsense or hierarchical 'is-a' links) and less accurale for others (e.g., detailed encyclopedic facts or multi-hop traversal relations; Pan et al. 2023a). In addition, range of inherent drawbacks raises concerns about the use of LLMs for reliable domain-specific KG generation. The drawbacks we discuss next appear to be intrinsic to generative methods of KG construction.
LLMs are brittle with respect to prompts: Instruction fine-tuning does not fully address this prob- lem (Zhou et al., 2024). KG extraction with prompts is biased towards prompt structure (Cao et al., 2021). LLMs are sensitive to lask-framing: answer consistency can shift with small syntactie changes (Hagström et al., 2023), with slight prompt variations (Mousavi et al., 2024; Wang et al., 2024a), and even with respect to basic logical constraints (Ghosh et al., 2025). Retrieval augmentation can mitigate inconsistency and knowledge-cutoff issues (Shuster et al., 2021), but introduces new failure modes: conflicts between retrieved evidence and parametric knowledge of an LLM (Zhang et al., 2025; Zeng al., 2025), imperfections in retrieval and ranking (Jin et al., 2024), and weak relevance estimation, resulting in incorrect utilization of the retrieved knowledge (Wang et al., 2024e).
Beyond reproducibility, LLMs hallucinate outputs that are nonsensical unfaithful to the provided source content (Ji et al., 2023; Zhang et al., 2024; Li et al., 2024). Hallucinations persist regardless of model size or training data scale (Shuster et al., 2021). The current consensus is that hallueina- tions are an inherent, unavoidable property of LLMss. Some scholars formally show that, from the theoretical viewpoint, hallucinations may be innate to probabilistic generative methods (Xu et al., 2025). All methods to strrengthen LLM reasoning, including chain-of-thought with self-consistency, Lun-Lunnng, augmented gen- eration, and greedy decoding, improve accuracy, but are unable to eliminate nontrivial hallucinations (Kim et al., 2025). In the context of KG generation, LLMs struggle with inverse inference (the "reversal cursc"): They may learn (A, is-a, B) yet fail to infer the inverse relation (Berglund et al., 2024), undermining triple extraction in domains where inverse relations are common and semantically critical.
LLMs hallucinate:
LLMs are not factually accurate: Factuality and hallucination distinct: Factuality errors occur when a model fails to learn or apply factual knowledge accurately, whereas hallucinations are ungrounded or unfaithful content relative to the provided source (Wang et al., 2024f). As demonstrated in our motivating example (Sec. 3), even state-of-the-art models can return factually divergent answers to equivalent queries. Because models acquire factual knowledge during pretraining and can add more via contimued pretrain- ing (Chang et al., 2025), rigorous dataset cleaning essential. Yet, verifying or synthesizing high-quality data at the LLM scale is infeasible practice, creating fundamental size-quality trade-off (see Sec. 2.5). No cur- rent LLM offers the factuality needed for trust. Even the most advanced commercial systems make significant factual errors, which spike on rare entities and remain less factual than humans (Min et al., 2023), with mis- takes persisting even in scarch-augmented tools (Mchrotra & Marchman, 2024). Consequently, any method that relies on LLMs needs robust. verification. In high-stakes domains (e.g., medicine (Thirunavukarasu) el al., 2023), autonomous driving, aeronautics, and cybersecurity), verification alone is insufficient: Outputs must also be interpretable and explainable for decision-makone However, LLMs provide little insight into their decision rationale (Madsen et al., 2024; Ye & Durrett, 2022; Bowman, 2023), falling short of the guarantees these applications demand.
7

---

Published in Transactions on Machine Learning Research (02/2026)
2.5 Impact of Data Quality and Dataset Size
Prior work shows that state-of-the-art LLM capabilities emerge only when model size, dataset scale, and compute reach sufficient magnitude (Brown et al., 2020; Kaplan et al., 2020; Wei et al., 2022). As result, modern pretraining corpora often favor scale over domain fidelity. Given LLM propensity to memorization, flawed data sources with misinformation and biases are a primary driver of hallucinations (Huang et al., 2025b). Domain adaptation with fine-tuning (Hu al., 2022) can improve factuality and coherence, but risk catastrophic forgetting and cross-domain interference (Wang et al., 2024d); continued pretraining adapts knowledge more smoothly to a larget domain, but demands substantial additional data. In medicine, re- searchers warn that scarcity of diverse, high-quality data at the scale required by LLMs leads to "garbage in, garbage out" dynamic. and remains a key barrier to clinical LLM deployment (Thirmavukarasu et al., 2023). Compounding this, large-scale public text corpora lack established cross-verification mechanisms. Privacy and copyright concerns restrict access to private datasets for training purposes (Pereira et al., 2022; Wang et al., 2024b). In cases where models are trained on closed-source training data that are not accessible for scrutiny, the lack of transparency blocks the public's ability to conduct a thorough investigation (Nguyen et al., 2024).
As response to the limited availability of verified sources (Gandhi et al., 2024; Wang et al., 2024c) and unaffordable training costs, growing line of work seeks to maximize LLM performance with less data, emphasizing the quality-quantity trade-off. The post-GPT-3 trajectory of the NLP field (Brown et al., 2020) prioritized ever-larger unlabeled web corpora, arguably underweighting data quality. Recent efforts formalize data-quality criteria (Zheng et al., 2025) and show that small, high-quality datasets can outperform substantially larger but unvalidated corpora (Iskander et al., 2024); likewise, compact models trained on carefully curated, domain-specific text sometimes surpass frontier general-purpose models in-domain tasks (Kadosh et al., 2024; Chen et al., 2023a).
Our approach also advocates for "data quality first": We contend that high-quality data not the sheer volume of data is crucial to creating a reliable KG.
2.6 Integrating Knowledge Graphs with LLMs
While LLMs excel at fluent generation, they are limited by their reliance on implicit parametric knowledge, which can lead to hallucinations and obsolescence (Tonmoy et al., 2024). In contrast, KGs provide explicit, deterministic representations of factual information but lack generative flexibility and adaptability (Yang et al., 2025). Recent research has, therefore, focused on unifying these complementary paradigms, primarily through KG-enhanced pre-training and fine-tuning, as well as RAG.
2.6.1 KG-enhanced Pre- traininin and Fine-tuning
Early approaches sought internalize struetural knowledge directly into the weights of the model. K-BERT explicitly injects domain knowledge by stitching KG triples into the sentence constituent tree (Liu et al., 2020). Subsequent work DRAGON (Yasunaga et al., 2022) advances this by employing a cross-modal encoder that enables information to flow bidirectionally between text segments and KG subgraphs during pre-training. More recent studies demonstrate that domain-specific KGs can also be leveraged during fine- tuning to enhance LLM performance and factual consistency (Kamel et al., 2025; Jiang et al., 2024; Chen et al., 2025). While these approaches are effective in improving task performance and mitigating factual gaps, they typically require carefully curated KGs and additional training overhead, which can limit scalability and adaptability to rapidly evolving knowledge.
2.6.2 Retrieval-augmentte Generation
RAG enhances the capabilities of LLMs by connecting them to external data sources. In a typical RAG system, a user's query is used to retrieve relevant documents or text chunks from a large corpus. This retrieved information is then combined with the original query into a prompt for the LLM, which generates a response grounded in the provided context (Ram et al., 2023). This approach is particularly effective when the knowledge base is too large to fit within the III context window. The most common retrieval
8

---

Published in Transactions on Machine Learning Research (02/2026)
method, vector RAG, involves embedding the text corpus into vector space and retrieving chunks that are semantically closest to the query vector (Gao et al., 2024). However, this approach struggles with queries that require a holistic understanding of the entire dataset.
GraphRAG: To strengthen standard retrieval, many systems incorporate KGs (Guo et al., 2023; Deng et al., 2024; Sun et al., 2023). GraphRAG (Edge et al., 2024) leverages the inherent modularity of KGs to support global sense-making in two stages: indexing and querying. In the indexing stage, an LLM builds an entity-level KG from source documents, then partitions the graph into a hierarchy of nested communities basod on the density of connections between entitics. The LLM then produces bottom-up summarics for each community, yielding a hierarchical summary tree that aggregates local insights into global ones. In the querying stage, GraphRAG extracts a subgraph based the pre-generated community summaries and the query, and uses that subgraph as context to generate answers. With its hierarchical design, GraphRAG reports improvements over conventional vector-bases based RAG (Edge et al., 2024).
Using GraphRAG for KG evaluation: The global sense-making capability of GraphRAG also enables unique method for evaluating the quality of the KG and benchmarking it across various lasks. The accuracy of the answers to user queries is directly dependent on the coverage, validity, and factuality of the KG. A poorly constructed KG (e.g., one with incomplete entitics or incorrect relationships) results in fragmented, inaccurate, or nonsensical responses. Therefore, by assessing the quality of GraphRAG output to queries, one can indirectly measure the quality of the KG itself.
2.7 Graph Attention Networks and Graph Transformer Architecture
The transformer architecture (Vaswani et al., 2017) has emerged the dominant paradigm for NLP tasks. Yet, the native transformer self-altention module handles only sequential input. To enable training of a transformer on graphical input, the graph structure must either be encoded into the input or the attention module must be modified. We do both: (1) encode relation embedding into input graph sequences (sce Sec. 4.2.1) and (2) modify atlention weights to reflect spatial distance in input graphs (see Sec. 4.2). To implement our model, we take inspiration from Graphormer (Ying et al., 2021) but design alternative graph encodings tailored to language tasks.
3 Motivational Example
What follows is a motivating example to demonstrate the importance of reliability in our proposed pipeline. To do this, we design a simple "reverse test" using Unified Medical Language System (UMLS; Boden- reider 2004) triples. First, we sample a ground-truth triple from UMLS: (chronic kidney disease, has has_ tos__fidinggsiie site, kidney structure) [SNOMED CT United States Edlttonn vocabulary in UMLS lists kidney structure as the only finding site for chronic kidney disease (CKD)). Next, we manually create a soquence that implics weak connection between CKD and cerebellar gray matter abnormalities (so that it is appropriate for the associated_ with relation, i.e., (chronic kidney disease, associated _itth cerebellar gray matter) would be an ontologically appropriate triple). We craft this sequence so that it reflects recent medical studies on brain imaging in patients with CKD (Xiao et al., 2024), which show indirect abnormalities in the cerebellar gray matter. Even with this indirect correlation, the logical triple should remain (chronic kidney disease, has forss_fnninnggn site, kidney structure), as found in UMT.S.
Next, we prompt strong general-purpose LLMs (Gemini 2.5 Pro, Claude Sonnet 4.5, GPT-5, and Grok 4) to infer the triple from the sentence. Among all these LLMs, only Claude Sonnet 4.5 was able to give the correct tail. We present the Gemini 2.5 Pro response below, and place Claude Sonnet 4.5, GPT- 5, and Grok 4 responses in Appendix B. Despite fluent rationales, these models frequently hallucinate relations or return onlologically invalid outputs, yielding triples that are non-factual or misaligned with UMLS constraints. Their outputs seem to show spurious correlations instead of semantic connections. In contrast, our GRAPHMERT model, trained as detailed in the following sections, recovers the correct UMLSS triple from the same sentence. This example underscores that adhering to biomedical ontologies matters: Terms like "gray matter" should be used with an associated with relation rather than being miscast with
9

---

Published in Transactions on Machine Learning Research (02/2026)
finding_sit site; structure-aware training is essential for preventing such errors. This is only possible with the proposed GRAPHMERT pipeline.
Prompt. Please complete the following medical KG triple (head, relation, tail): (chronic kidney disease, has findinggsiter ...)
based on the sequence:
Chronic kidney disease (CKD) is renal disorder. However, neuroimaging cohorts of people with CKD have reported subtle, group-level abnormalities within the cerebellar gray matter. Studies note modest reductions or regional alterations in cerebellar gray-matter volume, cortical thickness, and diffusion met- rics. These observations are heterogeneous and not diagnostic, yet they place observable findings at the cercbellar gray matter in CKD. Thus, CKD may be associated with cerebellar gray-matter changes.
Gemini 2.5 Pro: (chronic kidney disease, has finding site, cerebellar gray matter)
The provided text explicitly states that neuroimaging studies "place observable findings at the cerebellar gray matter in CKD."
Next, based on the top 20 predicted tokens from our model with top-ō softmax probabílities of 0.789, 0.173, 0.01, 0.003, 0.003, our pipeline yields "kidneys" (with Gemini 2.5 Pro as helper LLM), which corresponds the UMLS gold answer:
GraphMERT:
structure, kidney, system, function, renal, discase, heart, kidneys, of, physiology, arterial, underlying, pancreas, organ, impaired, health, oerebrovascular, pathophysiology, injury, morphology
Gemini Pro reasons over GraphMERT triple candidates:
1. Analyze the Input:
Sequence: The text describes Chronic Kidney Disease (CKD) as a "renal disorder," indicating its primary location is the kidneys, It also explicitly mentions that in people with CKD, neuroimaging has found "abnormalities within the cerebellar gray malter."
....
2. Generate Potential Tails:
The most obvious and primary finding site for "renal
disorder"
the kidneys.
The token list contains kidneys. This is very strong candidate.
1 can also form renal system from the tokens renal and system. This is also correct but is more general than kidneys. Specificity preferred.
3. Verify and Filter Tails:
Candidate 1: kidneys
Factual: Yes, CKD is disease of the kidneys.
Supported by sequence: Yes, the text calls CKD a "renal disorder."
Valuable & Specific: Yes, pinpoints the primary organ affected. It is specific and not vague.
Uses only candidate tokens: Yes, "kidneys" is the list.
Conclusion: This is high-quality, factual tail.
The text also mentions findings in the "cerebellar gray maiter." struct this. Cerebrovascular is available, and structure is available. I could potentially form cerebrovascular structure. However, the cerebellum is not primarily ture, and this tail is very general and not as precise as the information in the text ("cerebel- lar gray matter"). More importantly, the key tokens cerebellar and gray matter are missing. Therefore,
look for tokens con-
vascular struc-
10

---

Published in Transactions on Machine Learning Research (02/2026)
((a)
Seed KG (triples)
((c)
(((a)
GraphMERT
Helper
PAD
Train
LLM
Semantic
Final triples
((b)
Similarity filte& Drop duplicates
ll(b)
Trained
PAD
PAD
PAD
PAD
PAD
GraphMERT
Sentences
Chain Graphs
Prodict
Syntaciie Sementic
PAD
PAD
PAD
Extracted KG
Syntactic source
Raw triples
Overview of the GRAPIMERT framework. is trained on the fusion of syntactic and semantic examples (IT) and augments syntactic data with semantic tails (I); an LLM helps determine the linguistic structure of tails proposed by GRAPIMERTT (III). (I): Chain graph (Ic) combines syntactic knowledge from text corpora (Ib) with semantic examples and relations from seed KG (Ia): Roots hold syntactic knowledge (in orange), sparse leaves hold semantic examples (in blue), and edges encode semantic relations (purple arrows). (II): GRAPHMERT is trained on chain graphs to align semantic examples with their syntactic context (Ila). It then predicts novel semantic token completions for chain graphs without injections, using their syntactic information as context (IIb). (III): An LLM combines raw semantic token completions from GRAPHMERT into grammatically well-formed triple tails, producing complete triples. After filtering them by similarity to the source syntactic context and dropping duplicate triples, obtain the final KG.
Figure 2:
cannot accurately represent this secondary finding site.
4. Final Decision:
The best and most directly supported tail can construct from the given tokens kidneys. While the text mentions other sites, the provided tokens do not allow for their accurate creation.
As we noted above, there is an indirect correlation between CKD and cerebellar gray matter in the source sentence; however, does not imply the triple suggested by most of the LLMs their responses. This illustrative case reinforces recent evidence that LLMs regrettably answer based on word correlations in language rather than honing in on the semantic meaning of text and vernacular syntax, making them surprisingly brittle (Li et al., 2020). Recent works also show that, despite the high accuracy on various language tasks, when examined more closely, LLMs are only learning surface-level information, including word overlap, perplexity, sentence lengths, etc., and not the underlying task at hand (Durmus et al., 2022). Taken together, the example above and the additional results we present later make us skeptical about deploying these models high-stakes use cases for constructing reliable, domain-specific KGs.
4 GraphMERT KG-extraction Framework
This section introduces the KG extraction pipeline and the GRAPHMERT architecture. It also introduces LLM-based KG generation method that is used to obtain the baseline KG.
One of the carliest studics on relational factual knowledge extraction from pre-trained encoder-only models used cloze-style prompts, as discussed by Petroni et al. (2019). E.g,, "John Lennon plays [MASK]" to complete (John Lennon, Personlnstrument, ?). Later work showed that such a prompt-based retrieval is heavily biasod toward prompt syntax rather than factual content (Cao et al., 2021). For instance, García- Silva al. (2023) report that BERT (Devlin et al., 2019) completes the above example with "guitar, piano, drums, himself, harmonica," where syntactically plausible but non-factual predictions like " "drums" rank high
11

---

Published in Transactions on Machine Learning Research (02/2020)
(A)
(B)
PAD
PAD
PAD
PAD
PAD
h
PAD
PAD
PAD
PAD
PAD
PAD
(C)
PAD
PAD
PAD
11
12
PAD
PAD
PAD
PAD
h1
h2
Figure Chain graph. Roots are in orange, leaves are in blue. Conceptual representation (A, B): term level, cach circle is a term. Actual representation in training (C): token level, cach square is a token. Each term can be multi-token. (A) No injections, all leaves are emply. (B) One root node has leaf lerm. (C) Token-level representation for the 3-leaf case. Here, the leaf in (B) is encoded with maximum of three tokens and padded to the maximum length if needed. Root term comprises two tokens, and tail term also comprises two tokens that are connected to the first root token.
and, surprisingly, more specific prompts, e.g., "John Lennon plays instrument [MASK]," produce irrelevant outputs like "here, there, too, himself, onstage."
These shortcomings stem from the syntactic form of the prompt overshadowing the factual knowledge of the model, which is mostly semantic. Our key innovation overcomes this obstacle by implanting relations into an encoder via graph attention and training relation embeddings in a dedicated semantic space under the unified MLM + MNM objectives. Our approach directly teaches the model relational knowledge abstracted away from prompt syntax. In parallel, GRAPHMERT learns the syntactic structure too and leverages syntactic information as a context for semantic knowledge.
Fig. shows a high-level overview of the pipeline. GRAPHMERT a multi-directional encoder-only trans- former. To complete triple, it predicts a masked tail (a node the KG, hence MNM). Just like typically text-based encoder-only models, GRAPIMERT also learns syntactic representations from text corpora via the MLM learning objective (Devlin et al., 2019). To enable encoder-only extraction, create new textual data format that enca encapsulates semantic triples and engineer GRAPHMERT to work in this space.
4.1 Syntactic and Semantic Spaces: Merging Semantic Triples and Syntactic Text into Unified Graphical Format
In essence, GRAPHMERT performs syntactic-to-semanti knowledge conversion during prediction. The sentences in the datasel. represent the symlactic space. The KG triples represent the semanlic space that includes semantic relations. To enable knowledge form conversion, we propose leafy chain graph encoding that unifies the semantic and syntactic representations into a joint representation (see Fig. 3) in which chain graph roots li in the syntactic space and leaves, along with their relations, lie in the semantic space. The leafy chain graph lets the model process syntactic tokens and semantic leaves within a single attention graph, while still preserving the distinction between syntactic and semantic nodes during training and prediction. As demonstrate later, leaves play crucial role in training semantic relation embeddings.
Leafy chain graphs follow regular structure, which enables sequential encoding of the graphical information. All chain graphs have a fixed number of root nodes; the number of leaves per root node is also fixed. Leaves of the same root are connected, introducing a shortest-path linkage between them. All edges are undirected (the directionality of relations is implied in the architecture).
To create a unified representation for the training data, we parse the dataset into chain graphs with <pad> tokens in all leaf positions, keeping only root nodes non-empty. Next, populate the empty leaves with
12

---

Published in Transactions on Machine Learning Research (02/2026)
GraphMERT
1. Embedding Layer
II. Transformer Layers
(A) No H-GAT (no leaves)
Node Feature
Syntactic token
MatMul
send to transformer layers-
Softmax
(B) With H-GAT leaves
Decay Mask
Fused Node Feature tj
H-GAT(t, a, W, hy). j= 1...m
f(sp(i. D)
Scale
Semantic token
MatMul
Relation
Relation embedding matrix W,
Q
Fused Node Feature
OR
Syntactic token hm
Node Feature
All head tokens
Figure 4: Main GRAPHMERT architectural components. GRAPHMERT is RoBERTa transformer with two modifications. (I) In the embedding layer, H-GAT encodes semantic triples. (IA) There are no leaves connected to a root node; hence, the node feature equal to the token embedding. (IB) There are leaves connected to a root node; H-GAT fuses leaves, relations, and head embeddings resulting in fused node feature. (II) In the attention layers, attention weights are multiplied by function that exponentially decreases with pairwise distance. They encode graph relations and graph distance, respectively. The input either node feature or fused node feature.
semantic nodes and their relations from the seed KG in separate pipeline step, as detailed in Sec. 4.3. Thereafter, the dataset consists of leafy chain graphs with regular structure. Most leaf nodes pads, while some contain semantic tail tokens from the seed KG. This regularity in graphical input informs and simplifies the choice of graph encodings in GRAPHMERT, discussed further in Sec. 4.2.2.
4.2 GraphMERT Architecture and Training
The core architectural challenge in graph transformer design lies in encoding graphs into sequential input for allention-based learning, in order to enable robust, graphical representations. We discuss this in the next subsections.
4.2.1 Hierarchical Graph Attention Networks
To incorporate semantic relations into GRAPHMERT, we combine a generic transformer architecture with hicrarchical graph attention network (H-GAT; Xu et al. 2021; Nathani et al. 2019). H-GAT fuses relation embeddings into semantic graph nodes before passing the graph sequences the transformer layers. The original H-GAT architecture hierarchically combines intra-relation and inter-relation attention to derive
13

---

Published in Transactions on Machine Learning Research (02/2026)
node embeddings by aggregating the embeddings of the neighbors of the graph node. To tailor H-GAT to our input (i.e,, chain vocabulary graphs that we describe later), we discard the unnecessary inter-relation representations and use the simplified architecture with token embeddings instead of graph node embeddings.
In the GRAPHMERT implementation, given a triple < h, are represented by (hr,.., } and (m,,,,4)) at the token level, for any given tail token 4 we have:
r, > - head, relation, tail, where head and tail
c LeakyReLU (a. [W, W,h,)).
(1)
where W, is learnable relation embodding matrix, a, is a learnable relation embedding, and LeakyReLU is an activation function.
exp(e (r)
softmax, (e(7)
(2)
k= exp (c(k)
Then the final node embedding for the tail Loken is given by:
;= _ W,hj-
(3)
Thus, the new tail token tę fuses the relation embedding via W, and a, with its initial tail token embedding & and all head token embeddings (hi,..., )m
4.2.2 GraphMERT Architecture with Graph Encodings
The proposed GRAPHMERT F(z, 0), x: chain graph input, 6: trainable parameters, is a RoBERTa-style (Liu et al., 2019) encoder-only transformer integrated with H-GAT, trained with the MLM MNM objective. Fig. 4 illustrates the GRAPIMERT architecture. Our choice of graph encodings is informed by the regularity of the imput graph: The input consists of chain graphs with fixed number of root and leaf nodes. Therefore, to describe the imput graph class, node encoding, semantic leaf relation encodings, and spatial distances between node pairs are sufficient. Though common in graph transformers, degree and sparsity encodings offer little value in our setup. The two core components of GRAPIMERT that encapsulate graph encoding are the input embedding layer and the attention decay mask.
The embedding layer processes input nodes, concrctely, root nodes along with their lcaves, and semantic relations for each injected leaf node. For every injected triple, its head (multi-token) lies in the root space and its tail (also multi-token) lies in the leaf space. The embedding block fuses every leaf token embedding with its relation and all its head tokens via H-GAT (see Sec. 4.2.1):
1, = 14 + H-GAT(4, (h,,.,, h,,)), dim(C)) = dim((,),
(4)
where t is the tail token embodding, h, is the head token embedding, and dim is the embedding dimension. The derived embedding replaces the initial leaf embedding, effectively encoding the whole semantic triple into the leaf embedding space, as shown in Fig. 5. During training, masking leaf nodes enables the training of relation embeddings with backpropagation, as shown in Fig. 6.
The allention decay mask encodes the spatial distance between graph nodes. The core idea is that attention between two nodes should decrease with respect to their distance. Following Gradformer (Liu et al., 2024a), we use an exponential function with base 0 < A < 1 and the shortest path in the exponent. To adjust Gradformer exponential mask for vocabulary sequence graphs that experimentally need a smoother attention decay with respect the shortest path, we introduce a square root in the exponent.
The shortest path for every node pair is calculated using the FlryhaWarshall algorithm:
sp(i.)) E NXN N number of input nodes
(5)
14

---

Published in Transactions on Machine Learning Research (02/2020)
Before H-GAT
After H-GAT
f1
12
13
H-GAT((1, h1)
I1'
12
If'
rel
rel
dim(t1') dim(t1)
h1
h1
h2
Every non-empty tail token embedding is replaced with f(h, f),
token embeddingg only
where h, t- token embeddings. r- relation embedding
Figure 5: Semantic embedding derivation on leaves (only three leaves are shown). hy: head token, 4: leaf token, syntactic context token. For every injected triple, H-GAT fusee each leaf token with the relation and all the head tokens, yielding an embedding of the same dimension the initial leaf token embedding. The derived embedding replaces the initial leaf embedding.
Before H-GAT
After H-GAT
Transformer layer
MASK
12
13
11'
12'
13'
transformer layers
Transformer layer
rel
rel
Cross-entropy loss
h1
h2
t
Every token embedding fused relation embedding and entire head token embeddings
gradient will backpropagate from loss to masked t1.
If leaf node is masked, contributes to loss
Backpropagation path: loss transformer layers H-GAT input token embeddings.
Figure 6: Relation embedding training. The sequence with updated leaf embeddings is passed to the trans- former layers. The masked nodes (both roots and leaves) contribute to the loss calculation. For masked leaves, the gradient flows back to them through H-GAT, updating the relation embeddings.
The exponential decay mask is an N x N matrix defined as:
[(sp(i.j) (EEL 0<λ<1
(6)
GELU = (x), & (x) Gaussian cumulative distribution function,
where p a learnable parameter and À a hyperparameter. For sp(i,)) S p. the activation function, GELU (Hendryeks & Gimpel, 2023), zeroes the exponent, making the mask close zero for nodes with the shortest path less than equal to the learned p. Finally, we multiply the atlention weights A NNN by the mask elementwise, effectively incorporating spatial distance in the attention mechanism.
Á A NNN
(7)
The exponential mask is shared across all attention layers.
4.2.3 GraphMERT Training
As explained earlier, we represent each sentence as a leafy chain graph whose root nodes are text tokens, and graft semantic leaf nodes from seed KG onto entity spans via relation edges. GRAPHMERT jointly
15

---

Published in Transactions on Machine Learning Research (02/2026)
pretrains using MLM over syntactic tokens and MNM over semantic leaves: We randomly select spans in either space and train the model to reconstruct the missing words and/or KG leaves. This setup couples the transformer token encoder with the H-GAT relation encoder so that the surface form and KG semantics align during pretraining.
(M) - Σ
(logpo ((aGrrGGG,,,,,))) + LSbO (st ( (GMM,uM)))
(8)
M,M,
(MN = Σ
log po (ge (.M,,MM))
(9)
SEM,
[(0) (0) μ [MMM(()))
(10)
where x denotes the input token sequence; G is the text chain graph augmented with KG leaf nodes; M₂ and Mg are the masked text and leaf spans, respectively; 2t is a masked token target; is a masked semantic
Ieaf target (as a Icaf-span explained below); LsBO is span boundary loss on Mf (explained bclow); 0 denotes all model parameters (transformer + H-GATT;; and > 0 balances the two losses (we use = 1). Both objectives use span-wise masking.
In GRAPIMERT, H-GAT is responsible for training semantic relation embeddings. Dropout on relation embeddings prevents overfitting scarce semantic examples. In parallel, transformer attention trains the remaining network parameters, attending jointly to tokens from both the syntactic and semantic spaces. GRAPIMERT is trained with span-masking schema. Empirically, span masking tightens aligmment among the top-k tokens predicted within a single leaf. Later, better-aligned tokens result more nuanced com- binations when constructing complete tails. We discuss the top-k predicted token combination stage in Sec. 4.4.
From the syntactic stream, exactly follow the span masking implementation, which sums regular masked objective with span boundary loss of SpanBERT (Joshi et al., 2020). In the semantic space, however, we introduce modification: Whenever a leaf span is selected (with standard MLM/MNM probability of 0.15), we mask all the leaf tokens rather than sampling span length from the geometric distribution, which could mask a leaf subset for a given root. In other words, My and M2 follow different masking schemas. The rationale for this stems from backpropagation. Relation embeddings must receive gradients from the entire tail so that they can capture its full meaning, not just fragments from individual tokens. Since the semantic meaning of a relation manifests only across the complete tail expression, masking the whole leaf ensures that gradients reflect the whole semantic unit.
4.3 High-quality Text Sources and Dataset Preprocessing for GraphMERT Training
Our framework focuses on domains with stringent factuality requirements. The compact GRAPIMERTT model makes it feasible to train exclusively on limited expert-verified open-source texts. This approach takes an essential step toward reliable KG extraction: Data cleaning reduces hallucinations and training on scientific corpora can substantially limit domain-spocific errors (Li et al., 2024). Reliance on expert-verified data prevents importing spurious facts from vast, tainted corpora.
Data quality requirements apply equally the seed KG. The seed KG is of domain-specific triples that serve initial relation examples for the model. As the full training set in the semantic space, the seed KG defines the relation set for the extracted KG. Its quality, therefore, is of utmost significance in training of robust semantic relation embeddings in GRAPHMERT. A seed KG can be obtained from an external source, provided that it satisfies the following two conditions:
1. It contains clean, domain-specific data.
2. It has sufficiently diverse vocabulary.
Condition (1) provides the foundation for learning accurate relation embeddings and Condition (2) prevents relation embedding overfitting on small set of tokens. To obtain domain-specific seed KG that meets
16

---

Published in Transactions on Machine Learning Research (02/2026)
these requirements, we suggest either selecting relevant, well-curated external KG generating one (see Sec. 4.5), followed by thorough cleaning.
After obtaining a KG, we apply a similarity filter to it against the training data (see Sec. 4.3.1), which ensures alignment with the target domain [Condition (1)] and identification of the triples most relevant to the context (see Sec. 4.3.2). The algorithm for selecting the best matches, described in Sec. 4.3.3, addresses vocabulary diversity [Condition (2)]. This algorithm selects triples from the external KG to position them within the semantic space in the chain graphs, given the potential spots for triple heads within the syntactic space. For domain-specific head discovery in the dataset, we use helper LLM.
4.3.1 Entity Linking
Although GRAPHMERT is applicable to any domain-specing text corpora, we leverage biomedical knowledge in the UMLS to develop a multi-stage process to link entitios discovered in the text to concepts in UMLS. This process ensures that identified entities are mapped to standardized Concept Unique Identifiers, facilitating the subsequent retrieval of structured information from the UMLS KG. We combine both embedding similarity and string similarity.
Stage 1: Embedding- based candidate retrieval: The initial stage aims to rapidly identify a set of potential candidate concepts from the vast UMLS ontology. This is achieved by representing both the discovered source entities (queries) and the target UMLS entities in a shared high-dimensional vector space.
We use SapBERT (Lim Kim, 2022), an encoder-only language model specifically pre-trained on biomedical knowledge from UMLS, which is able to capture the nuanced difference between biomedical terms. Each discovered head entity and every UMLS entity is processed through SapBERT to produce unique vector embedding.
To efficiently search through millions of UMLS entity embeddings, we employ an Approximate Nearest Neighbor (ANN) algorithm. ANN constructs a pre-computed index of the UMLS embeddings, enabling highly efficient retrieval the top-k most similar vectors for given query vector. For each discovered entity, we use this method to retrieve the top 10 UMLS candidates based on the cosine similarity of their embeddings.
Stage 2: Fine-grained filtering with string matching: The top candidates from embedding based retrieval are then subjected to a more rigorous filtering process based on string similarity. We use character- level 3-grams (char-3grams) as a robust string comparison. We represent each entity name as set of char- 3grams, which can be easily compared with others using standard set-based similarity metrics. We compute Jaccard similarity between the 3-gram sets of the source entity and each of the 10 candidate entities:
|A B | + B A B
J(A,B)
A candidate entity confirmed valid link only if its Jaccard similarity score is greater than the threshold. The threshold is set to 0.5 based on manual inspection. The entities that successfully pass both stages are considered the final Linked UMLS Entities and are used for the following task.
4.3.2 Contextual Triple Selection
Following the entity linking stage, each input sequence is associated with set of UMLS concepts. While these links grant access to the structured knowledge within UMLS, single concept can be involved in hundreds of triples, many of which may be irrelevant to the specific context of the source text. Therefore, a crucial subsequent step is to identify and select only the most contextually relevant triples for each sequence.
We perform this selection using an embedding-based relevance-ranking procedure. For each sequence, we begin by retrieving the complete set of triples from the UMLS KG where any of the linked entities from that sequence appear the head iny. We compute semantic relevance score for each triple with respect to
17

---

Published in Transactions on Machine Learning Research (02/2020)
"Syntactic" data source
"Semantic" data source
triple OR discard triple
similarity matching with text
Injection:
entity discovery
All triples
maximize tail similarity and triple diversity
Leafy chain graphs
PAD
PAD
Matched triples
II
Leaves contain Injected triples
Figure 7: Data preparation for GRAPHMERT. To find the most relevant triples, we perform semantic similarity matching of triples to dataset sequences. The triple head should almost litcrally match one of the entities discovered in Step (I); from them, pick the top triples whose tails are semantically close to the sequence. All matched triples are subject to the injection algorithm (III), which selects the top-scoring triples and limits the number of equivalent triples. The injected triples together comprise a seed KG.
the original input sequence. We exclude triples with undesired relations from the search: relations that are not useful to have in the KG (see Table C1 in Appendix A).
Specifically, each retrieved triple, consisting of head, relation, and tail, is transformed into lincarized sentence by concatenating its components with spaces. We encode both the original input sequence and the sentence formed by each triple into high-dimensional vectors using the Gemini embedding model, text- embedding-0044 and use cosine similarity the semantic relevance score. For each linked entity, we rank its associated triples by their semantic relevance scores and retain the top 40 triples. The resulting contextually filtered set of triples is then used in the subsequent injection process.
4.3.3 Seed KG Injection
The KG injection algorithm prepares GRAPHMERT-compttie datasct of leafy chain graphs by selecting relevant triples from an external KG source (potentially, limited in size) based on their similarity score to the input sequence, thresholded with a hyperparameter a. the same time, the algorithm maintains diversity in the injected relations and semantic vocabulary. All triples selected by the algorithm comprise the scod KG. In this process (see Fig. 7), triples are mapped the chain graph semantic space: The head is placed node and the tail at the root leaf node. Critically, the injected triples must be contextually relevant to the sequence. This aligns transformer attention with H-GAT during training the chain graphs, as both attend to the semantic and syntactic spaces simultancously; otherwise, the attention layers would receive a noisy signal from extraneous Lokens. Furthermore, because both H-GAT and transformer attention jointly train GRAPHMERT relation embeddings, alignment between leaf and root tokens enables vocabulary transfer from the syntactic root space into the semantic leaf space. This process forms integrated representations that support the retrieval of novel tails from a shared embedding space during prediction.
root
Why do we need an injection algorithm? A naive strategy would be inject the top-scoring triple for each head out of all the matched ones. We now demonstrate why a triple with the best similarity to a sequence may be suboptimal for both (a) populating the semantic space and (b) GRAPHMERT training.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Relation matching with LLM
Training dataset: high- quality texts
Entity discovery
Similarity matching with UMLS (embedding-
Train GraphMERT
Predict tails with GraphMERT
Combine tail tokens with
Similarity check (ombedding- based)
Expanded KG
LLM-based step Input/output
GraphMERT- based step
Embedding--based step
Seed KG
Arrows indicate temporal progression
data
Figure & GRAPHMERT Pipeline flowchart with temporal execution ordering of the main components.
(a) Limitations for the semantic space: Similarity matching favors frequent terms in the dataset (Zhou et al., 2022). Hence, triples with common domain-specific keywords in tails score highly across many sequences. As a result, a small set of triples achieves high similarity scores across large number sequences. If only top matches were to be chosen, these ubiquitous tails would doomnntee the semantic space, suppressing rarer but semantically valuable tails that introduce novel terms into the semantic space. The scoring also biased towards classificatory relations like isa or inverse _iss, since they often restate the head (e.g.. (fibrosing interstitial lung diseases, isa, fibrosis of lung) in biomedical KG). Such close textual matches contribute little new information.
(b) Limitations for GraphMERT training: Over-injecting frequent tokens leads to a skewed training distribution. If the semantic space vocabulary is dominated by a few tokens, relation embeddings overfit on them, causing GRAPIMERT to predict a narrow token subset. Likewise, certain relations, e-g., "isa" and "inverse isa," " would dominate the limited spots for injected triples, suppressing all other relations. Thus, GRAPHMERT will be undertrained on the other relations.
Design goals: To address these limitations, we design the injection algorithm around three goals:
1. Eliminate low-relevance matched triples.
Select one triple ("inject") per head out of all matched with the sequence.
3. Diversify injected relations by balancing examples across all relations.
Goal (1) can be satisfied by thresholding similarity scores. However, Goals (2) and (3) cannot be enforced independently: Selecting only the top match achieves (2) but undermines (3), amplifying certain top-scored relations, notably "isa". The algorithm must enforce (2) and (3) jointly, balancing contextual relevance with relation diversity. The proposed KG injection algorithm iteratively drops undesired triples from all matched triples in two interleaving phases: first, by maximizing the score; second, by maximizing the diversity of relations. The surviving triples are "injected" into the semantic space and comprise the seed KG. We defer the description and implementation details to Appendix D.
4.4 GraphMERT Pipeline for Knowledge Graph Extraction
We distill internal GRAPHMERT representations from the trained GRAPHMERT into explicit graph triples by adding leaf nodes. Using purely-MNM prediction, we distill semantic knowledge directly from GRAPH- MERT weights, conditioned on a sequence from which we want to extract a triple The role of a sequence in our framework analogous to the role of a prompt in LLM-based KG generation, but our prediction is unambiguous and deterministic.
Fig. shows the execution order of the framework components. Fig. 9 illustrates the first pipeline step. Starting with the syntactic corpus (no KG injections), we sample head (h) spans from the syntactic space
19

---

Published in Transactions on Machine Learning Research (02/2020)
PubMed papers (training dataset)
PubMed papers (training dataset)
Sequence with leaves emply
Sequence with masked leaf
PAD
PAD
MASK
PAD
PAD
PAD
PAD
PAD
PAD
PAD
1. Choose head for triple to be predicted
2. Mask its leaf and set the target relation
PubMed papers (training dataset)
k lokens
PAD
PAD
PAD
PAD
PAD
3. Predict the masked leaf takens (top k) with the trained GraphMERT
Extract triples: head relation k tail token candidates
Form final triples this head (with multi-token tails)
Figure 9: Prediction of triple tails. The trained GRAPHMERT predicts the top h tokens for masked leaf and the chosen relation, resulting in a set of raw triples with the same head.
(root nodes) and assign an outgoing relation (r). We then create the corresponding tail slot (4, leaf) for the chosen relation and initialize it with masks. We then ask the model to predict the masked tail (leaf) conditioned on the head, relation, and the rest of the sequence. The top-k predictions for each masked leaf yield k candidate tokens, which serve as building blocks for tails of each head relation pair (h, r). Next, a helper LLM combines predicted tail tokens inlo coherent phrases, followed by a cleaning step. We further filter the generated triples by computing semantic similarity between each triple and its source soquence, discarding those with a score below the user-defined similarity check threshold B. GRAPHMERT may predict tail that connects head with semantically related concept across the training corpus, and hyperparameter regulates the fraction of such triples in the output. A higher 3 yields fewer but more sequence-specific triples, often explicitly included in the text. A lower 3 enables broader, more general (yet semantically related) triples that may not be explicitly mentioned in the sequence. However, if 3 is set too low, the output becomes flooded with triples that merely restate general truths, reflecting statistically dominant statements in the training dataset.
The surviving rh, triples expand the KG with novel facts. Importantly, each prediction is traceable to its source sequence, while at the same time accumulating knowledge distilled from the entire training corpus. This global knowledge accumulation contrasts with RAG methods, which remain local to retrieved docunents. Further, RAG is a post-hoe data attribution mechanism, unlike ours.
The laborious (grammatical) part the work is carried out with help from an LLM and the essential (triple extraction) part done with GRAPHMERT. In this pipeline, the helper LLM performs three auxiliary tasks: discovering head entities, selecting relations for subsequent GRAPHMERT prediction, and combining single-token predictions into meaningful, relation-aware tail phrases. Critically,
Role of the helper LLM:
20

---

Published in Transactions on Machine Learning Research (02/2026)
the LLM is constrained: It cannot invent new entities or relations, heads must be present in the dataset, relations are restricted to the seed KG, and only GRAPHMERT--redien tokens are allowed to be in tails.
Why do we need a helper LLM for combining tokens? An encoder-only model does not address span decoding. Masked span prediction would still be challenging for a small model, given the very limited number of semantic examples. Here, two factors come into play:
For a coherent span prediction, each token should be conditioned on other tokens. However, in an encoder-only model prediction, cach masked token in the span is conditioned on the sequence independently of other span tokens.
In our experiments, training with span masking results in a prediction where tokens in the syntactic space are better aligned with each other within a span; however, the semantic space has orders of magnitude fewer examples (101 in experiments). Given the limited scale of the semantic space, how to achieve the same effect on leaves remains open question.
Thus, training small corpora some extent trades English proficiency for data quality. We further study the effect of the helper LLM size in ablation in Appendix I. Exploring methods to augment GRAPIMERT and thereby remove the LLM-based mumbiiingtooken step is an avenue for future work.
4.5 LLM-generated KG as Baseline
We compare GraphMERT to LLMs for KG construction rather than fine-tuned BERT or T5 models because we focus on low-label domain regimes, where no large sentence-triple annotated corpus exists, and only a small seed KG is available.
To provide a fair and robust comparison for our framework, we construct a baseline LLM KG using a standard LLM-based pipeline that follows the GraphRAG indexing methodology, with a filtering step align its schema with the GRAPIMERT KG.
The process begins by segmenting the source documents into smaller, manageable text chunks. An LLM is then prompted to perform open information extraction on each chunk, which involves identifying entities, extracting the relationships between them, and generating short, descriptive sunmaries for each entily. We retain only the relationships that are part of our pre-defined relation set, discarding all others. This ensures that the resulting LLM KG shares exactly the same relational schema as our framework, enabling a direct and equitable comparison of performance. After filtering, the extracted elements are subsequently aggregated and consolidated into final graph structure. To handle multiple mentions of the same concept, exact string matching is used to resolve entities into unique nodes. Relationships that are generated multiple times between the same two entities are aggregated into a single edge.
4.6
KG Verification
We can subdivide KG verification methods into two categories: graph-level and triple-level.
4.6.1
Graph-level Verification
This method evaluates the KG as a whole. focuses on its logical coherence, internal consistency, compre- hensiveness, coverage, important aspects of domain-relevant knowledge, and depth (considering whether it contains rich, insightful connections beyond surface-level facts). Graph-level approaches typically operate by retrieving relevant subgraphs and evaluating their quality with respect to these criteria.
GraphRAG: We employ GraphRAG to evaluate and benchmark KGs across various lasks. The KGs serve as the primary source of information in GraphRAG to answer medical questions, which enables us to compare their effectiveness directly. In our implementation, we use the Local Search method from GraphRAG and modify it to rely exclusively on the entities and relations in the querying stage. This process begins by identifying set of entities within the KG that are semantically related to the user query. These entities
21

---

Published in Transactions on Machine Learning Research (02/2026)
serve entry points for retrieving connected entities and relationships. The retrieved data sources are then ranked and filtered to fit within a single predefined context window, which is used to generate response to the user query.
4.6.2 Triple-level Verification
At the triple level, verification can enhance factuality and validity, as we describe next.
The FActScore framework (Min al., 2023) provides fine-grained method for evaluating factual precision in long-form LLM outputs. Its principles transfer naturally to KG verification. FActScore evaluates alomic facts, i.e., short, sell-contained statements, against trusted text source that does not have any knowledge conflicts or overlaps. KG triples can be treated as atomic facts of equal importance. In our setting, each triple can be paired with a reliable text source: GRAPIMERT triples with sequences and LLM triples with short chunks, both drawn from the same trustworthy source. The short context length minimizes conflicts and overlaps.
FActScore:
FActScore We follow the Retrieve + LM variant of automatic evaluation, in which an atomic fact is concatenated with the knowledge source and provided to the model. However, we strengthon triple evaluation with validity: In the prompt, we require verification of triple logical alignment in addition to context support, since the fact may appear the text, yet. the triple may still be malformed. Malformed triples should not be deemed reliable facts and would inflate the score. Because our prompt departs from the original FActScore prompt, denote the modified version as FActScore". Formally, let be set of triples т, and C(T) the text source of r. Then FActScore* for G is:
f(T) I[[ is supported by C(r)), FActScore*()) = (r))
(11)
ValidityScore: Triples should follow semantic rules of KG. For example, in UMLS (Bodenreider, 2004), the triple (beta-receptor, per__f, plasma membrane) is valid, since part_of denotes a meronymic (struc- tural/spatial) relation: Every instance of the part must be constituent of some instance of the whole. It follows, then, that (bela-receplor, part of, adreneryic signaling) is invalid, because il links physical structure to a biological process. Similarly, (beta-receptor, part_of, human) is factually plausible but still an illogitimate usage, since it violates gramularity: The correct wholes are specific structures (e.g., plasma membrane), not entire organisms.
To quantitatively measure the validity of triples, we propose ValiditySccoe. isolates ontological alignment of triples as an independent mode of evaluation. Concretely, we use a strong LLM judge to semantically validate triple using the following prompt.
Prompt. Evaluate if these medical KG triples are valid hers/no/maylln and give a very short reason why: (list of triples).
Then ValidityScore counts the number of "yes" responses.
5
Experimental Setup
To demonstrate the effectiveness our framework, we extract a high-quality diabetes KG from a GRAPH- MERTTccmpplte diabetes training dataset obtained from expert-verified sources. This section provides an in-depth explanation of the proposed training, extraction, and evaluation pipeline.
5.1
GraphMERT Training and Extraction
Next, discuss training data preparation and triple extraction.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Dataset size
Abstracts
Tokens
Sequences
Training
350k
124.7M 13.9M
989,666
Evaluation
39k
110,297
Table 2: Seed KG statistics for 0.55 (after Qwen3-32B). We obtain this seed KG based the 1.1258E++06 matched UMLS triples with a similarity score greater than or equal to
Triples
Similarity score mean
Similarity score median
Similarity score max
Number of relations
count
28533
0.613
0.605
0.848
28
5.1.1 Training Data Preparation
We showcase our framework in a sensitive medical domain, where concerns over trustworthiness of Al output remain the main barrier to wider AI adoption (Wang et al., 2025b; Mishra et al., 2024), despite the undeniable potential of AT medical practice. As a case study, we set. the goal of extracting a high-quality diabetes KG. diabetes training dataset
It all starts with high-quality textual data. We build a GRAPHMERRT-om (Table 1) from two main sources: (1) peer-reviewed medical abstracts from MEDLINE journals accessed via PubMed Central, and (2) seed KG derived from the UMLS Metathesaurus (Bodenreider, 2004).
Text corpus: MEDLINE the the National Library of Medicine bibliographic database and accessible via PubMed. MEDLINE selects journals based on rigorous criteria and indexes them using MeSH terms. We retrieved diabetes-related papers from PubMed Central (see the query in Listing 1 in Appendix C), removed non-English records, parsed abstracts using the PubMed parser (Achakulvisut text, and filtered out opening boilerplate words including "A "Abstract," " "Background," " "Introduction," " etc., with a regular expression. The resulting dataset contains 350k abstracts for training and 39k for evaluation, totaling 124.7M and 13.9M tokens, respectively.
al., 2020), lowercased all
From UMLS, select SNOMED CT, US and GO (Gene Ontology) vocabularies because together they cover broad range of biomedical concepts across clinical documentation, molecular biology, and data exchange. We exclude low-value relations (more details in Appendix C) and retrieve triples relevant to our dataset sequences based on semantic similarity matching (Sec. 4.3.1) with Gemini text-embeddding-00 For matched triples, we use an injection algorithm (Sec. 4.3.3) with a similarity threshold a of 0.55, validated experimentally via grid search using GraphRAG evaluation (see Sec. 6.4.1). The injection algorithm uses 51 GB of peak RAM during the pre-processing stage and runs for approximately 0.5 hours on a single CPU, The resulting triples that injected comprise the seed KG (Table 2). Statistics per relation presented in Table D1 (Appendix D)
Seed KG:
As helper LLM, we employ Bwem3-32B-FF (an 8-bit quantized version of Qwen3-32B, further referred to as Qwen3-32B), an open-source, advanced, and lightweight LLM with "thinking mode" turned on. We always use "thinking mode," " unless otherwise specified.
For head discovery, we prompt Qwen3-32B with each abstract sequence in the dataset with few-shot examples, asking it to scarch for medical entitics that are relevant to diabetes and its comorbidities. From our observations, changing an example in the prompt significantly changes the number of discovered entities. The outputs are validated against sequences of origin to eliminate hallucinated or misspelled entities. For relation discovery, Qwen3-32B few-shot prompted with relation list from the corresponding training seed KG to match entities with all relations that make sense for a given entity in the context of the current. sequence. Prompts and examples are presented in Appendix G.
Entity discovery and relation matching:
23

---

Published in Transactions on Machine Learning Research (02/2026)
2nd root's leaves: 138-142 are pads
Visual representation of encoding
139
140
141
138
142
1
2
3
with
t1dm
136 diabetes
137
mellitus
128 roots
128 7 leaf tokens
2
128
129
135
136
142
1st root's leaves (7 tokens)
2nd root's leaves (7 tokens)
Chain graph encoding in training data
Figure 10: Leafy chain graph encoded sequentially: 7-leaf case, The sequence has a fixed length of 1024. The first 128 tokens are reserved for roots and leaves reside in the remaining tokens. The first group of seven leaves belougs to the first root, the second group of seven leaves belongs to the second root, and so on. Each leaf token group is padded to the maximum length of seven.
All Qwen3-32B runs performed H100 GPU with 80 GB of memory using vLLM with the vendor- recommended sampling parameters: temperature 0.6, top_ 0.95, top k = 20, and min_p 0. In addition, mar tokens and max_ _ode__mdel____ Ien are set to 8192, which is sufficient for our generation length.
Entity linking: We implement approximate ANN search for entity linking using Faiss, rumning on CPU over filtered subset of 3,647,914 UMLS entities. The search over 117,317 discovered head entities completes in 133 minutes, with approximately 48% of them successfully linked to a UMLS entity.
Chain graphs for training: We initialize chain graphs with 128 root nodes, each connected to seven Ieaves, leading to a 1024-token sequence (sco Fig. 10). These numbers are chosen to fit GPU memory constraints while providing sufficient room for semantic tokens dedicated to leaves.
5.1.2 GraphMERT Training
We train GRAPHMERT with 12 hidden layers, eight attention heads, hidden size of 512, and interme- diate size of a fully-connected layer of 2048, totaling 79. 7M trainable parameters. We use the BioMedBERT tokenizer (Chakraborty et al., 2020), trained on a vast amount of medical vocabulary, to prevent frequent subword tokenization of common medical terms, which particularly beneficial for the extraction stage. The tokenizer determines the vocabulary size, which is 30,522.
Training runs for 25 epochs on four H100 GPUs with 80 GB of memory each, with BF16 precision, totaling instantaneous batch size of 32 per GPU, achieving effective batch size of 128
90 GPU hours. We through gradient accumulation (steps = 2). We set dropout rates at 0.1 for regular, attention, and activation
dropouts. In addition, we set an exponential mask with base À = 0.6 and relation embedding dropout of 0.3. We train the model using the cosine learning rate scheduler with the maximum learning rate set to x 10 10-4, We use 500 steps for warm-up. The weight decay at each step 0.01 times the learning rate that step. We stop training when the learning rate reaches 10 In the span masking training schema, we limit masked spans to maximum length of seven, matching the number of leaf nodes connected to root nodes.
24

---

Published in Transactions on Machine Learning Research (02/2026)
1. GraphMERT pipeline
among people with diabetes, respectively. in the multivariable logistic regression analysis, there was significant relationship between female gender, age over 40, living in the urban area, being in the third wealth quintile and having health insurance with diabetes prevalence...
all diabetic patients adequate glycemic control,
Sequence:
Sequence
Input
Top k GraphMERT tail candidates
diabetes,
system, structure, endocrine. digestive, health, of, diabetes, 2, ii, metabolic, function, conditions, gastrointestinal, condition, heart, intestinal, type, GraphMERT hypertension, underlying
has finding site, ?
Combine GraphMERT tokens into triple tails
Helper LLM
Trained
2
< diabetes, has finding site, endocrine system?
Embeddings similarity filter triple
Sequence
< diabetes, has findinggsitee site, heart>
score
Sequence
triple 2
Formed triples for this sequence:
score
Output
diabetes, has finding__sin site. endocrine system>
< diabetes, has findinggsite _ite, urban area>
II. LLM KG pipeline
Figure 11: 1. Forming triple tails for given sequence with GRAPIMERT. (1) Given sequence as context, triple head in the sequence, and relation, GRAPIMERT predicts the tail token (we obtain the lop 20). (2) The helper LLM attempts to combine the tokens into complete, coherent medical term. It may output several or no tail candidates to complete the triple. (3) We evaluate the similarity score between triples from the previous step and the sequence origin. Only triples with a score higher than a preset threshold pass. An example of triple extracted with the LLM pipeline (Qwen3-32B) from the same context. Here, LLM misinterprets the "has finding site" relation, treating "site" location instead of anatomical structure, which results in invalid triple.
5.1.3 Triple Extraction
The triple extraction pipeline runs the following steps, as shown in Fig. 11. We begin with a leaf-masked prediction over the training dataset, given head entities and their relations. Running MNM prediction over the whole datasct with a batch size of 128 takes around 2.5h on H100 GPU (80 GB RAM) with GPU utilization of 90-100%.
This produces vocabulary distribution for each masked leaf. From this distribution, select the top 20 tokens per leaf and use them to prompt the helper LLM, Qwen3-32B. Conditioned on the head, relation, and the originating sequence, the LLM combines these tokens into coherent, relevant, and medically meaningful multi-token tails (see Appendix G for the prompt specification). When no valid tail can be formed, the corresponding (sequence, head, relation) is skipped. Next, since the LLM may hallucinate tails outside the SeaHHMERR-prriiccte token space, we discard any output tails that contain out-of-scope tokens. The output of this stage set of completed candidate tails.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Examples of GraPnMEER--extrateee triples (UMLS-style) from a single sequence. We pass the sequence with two marked heads together with relations for these heads to the trained GRAPHMERT. After GRAPHMERT makes a prediction in the semantic space, we use the top 20 predicted tokens to form complete triples with helper LLM. "##" " is separator for subword tokens.
Sequence 1: ... ## The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great. Britain and Treland. Non-alcoholic fatty liver disease (NAFLD) is one of the main causes of chronic liver disease worldwide. Flavonoids, group of natural compounds, have garnered a great deal of attention in the management of NAFLD because of their profitable effects on glucose and lipid metabolism, infiammation, and oxidative stress which are the pivotal pathophysiological pathways in NAFLD. Naringenin is citrus-derived flavonoid with broad spectrum of potential biological effects including anti-inflammnttory and antioxidant properties, which may...
Head, Relation
Top-20 20 GraphMERTT-redicee
tokens
naringenin, isa
flavonoid # flav nar # #idin # ##flav compound # #cce hydrolase plant #inngen #ono protein polysaccharide polyphenal family #anoo
# #onin
naringenin, plays role
therapeutic neuroprotective # #ingen antidepressant role medicinal # #arin flavonoid action baical inhibitory antibacterial bioactive medicine antimicrobial # #idin quercetin # #fflaav potential flavon
naringenin, has disposition
nar flavonoid # hydrolase #arri inhibitor #anool compound # #idin any #fflav #ingee derivative amide receptor acid family alkaloid product liver disease alcoholic latty # #osclerosis naild steatosis disorder # #alitis # #ohep hypercholesterolemia hepatic- with syndrome mellitus myopathy fibrosis discases hyperlipidemia
non-alcoholic latty liver dis- ease, cause_od of
non-alcoholic fatty liver dis- ease, associated with
hyperlipidemia disorder alcoholic obesity dyslipidemia myopathy liver hereditary lyperchalssterolemiaa syndrome #ament ##tr associated ##lipp
fatty disease # #oid mellitus related
Head
Relation
Tail (formed from the GraphMERTT tokens)
uaringenin
isa
flavonoid
naringenin
plays role
therapeutic role
naringenin
has disposition
flavonoid
non-alcoholic fatty liver dis- ease
canse of
fibrosis
non-alcoholic fatty liver dis- ease
associated with
obesity
Next, each candidate triple is evaluated with a similarity matching algorithm (Sec. 4.3.1). Specifically, we compute the cosine similarity between the triple and its originating sequence using Gemini embeddings. 0.67 (obtained through grid search, see
All triples with a score below the similarity check threshold Sec. 6.4.1) discarded. The remaining set forms the final collection of extracted triples.
Table 3 illustrates the step-by-step triple extraction process for a representative sequence. Given sequence with its head and relation, GRAPHMERT predicts a token in masked tail. From the output distribution, we select the top 20 tokens as candidate tails, striking a balance between prediction quality and diversity, and providing a sufficient pool for subsequent token combination. We may construct zero, one, or several novel triples out of these. For more raw GRAPIMERT output examples, see Appendix T.
5.2 LLM-generated KG
Following the default GraphRAG indexing parameters, our diabetes corpus is split into 2,000-token chunks and processod by Qwen3-32B to extract entities and relationships. The thinking mode is enabled, and the parameters are set as temperature 0.6, top_p p- 0.95, top_k k 20, mar mar_tokens 8192. The detailed prompt for extraction is shown in Appendix H1 and example is shown in Appendix H2. After parsing
26

---

Published in Transactions on Machine Learning Research (02/2026)
and cleaning, the final LLM-ggnnrated KG contains 272,346 triples. It serves as the baseline in all our experiments.
In addition, we extract a fixed set of 100,000 triples with Grok 4 Fast (grok--ffast-reesonnng which is approximately a 1....-Tpraaat paraameer model, and Qwen3-14B (thinking mode) to evaluate how model scale affects the results.
5.3
Evaluation with GraphRAG
Next, we provide details of GraphRAG evaluations.
5.3.1 GraphRAG Settings
Our experimental setup uses Qwen3-14B the backbone LLM in GraphRAG, with inference accelerated using the vLLM library. For all evaluations, we enable the thinking mode of the model and set temperature 0.6, top = 0.95, top k = 20, and mar tokens = 8192. To ensure the reliability of our findings, each experiment is conducted three times with different random seeds (1, 2, and 3). We report the average accuracy across these runs our final results.
The GraphRAG query process is configured with nomic-embedd-eextwi (Nussbaum al., 2025) the em- bedding model. To construct the context. for each query, the system retrieves the top 30 entities and the top 10 relationships per entity, with the maximum context length capped at 12,000 tokens. Furthermore, we tailor the system prompts and simplify the output table structure to better suit our tasks. The complete prompt is given in Appendix H3. The modified table schema is also detailed in Appendix H4.
5.3.2 Benchmark Evaluation
We further verify the quality of our extracted KGs the diabetes subsets of medical benchmarks: ICD- Bench (Dedhia et al., 2025), MedMCQA, MedQA, and MMLU (medical). ICD-Bench is a targeted question- answering benchmark aligned with the International Classification of Diseases (ICD) taxonomy (World Health Organization, 1992), designed to evaluate domain-specific medical reasoning in language models across 15 medical sub-specialties. We primarily evaluate the extracted KGs on ICDD-Bench, as it is the only benchmark that has a dedicated endocrinology subset; for this reason, we defer the results on other (public) benchmarks to Appendix F. For evaluation, we employ GraphRAG to answer benchmark questions.
To address the domain mismatch between our diabetes-speciallzet model, GRAPIMERT, and general med- ical benchmarks, we create domain-specific evaluation subsets. First, synthesize our training corpus into an approxinately 2000-word summary using Gemini 2.5 Pro. Subsequently, we use this summary to prompt a Qwen3-32B model to filter the question-answer pairs each benchmark, retaining only qucstions relevant to the summary. All GraphRAG evaluations are then conducted on these filtered subsets ensure a fair assessment.
5.3.3 Traceability and User Verification
Allowing users to judge correctness is simple but powerful principle. OpenAl WebGPT (Nakano et al., 2022) addresses this by browsing the web and citing sources, a strategy also adopted by Perplexity AI. these systems, the user decides (1) whether the source credible and (2) whether the output factual. Our framework builds on this idea: Each triple is directly traceable its originating sequence. This enables automatic cross-checking, rather than searching external sources. The system retrieves the sequence from which the triple is derived. Since this sequence originates from verified paper abstract, users can further validate the fact by consulting the source publication if needed.
6 Experimental Results
Next, provide evaluation results for GRAPHMERT and LLM KGs the benchmarks.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: GRAPUMEERR--extraaeee KG. Number of triples over extraction stages; similarity threshold 3 0.67. We show the number of triples after the helper LLM combines the tail tokens, next after discarding tails hallucinated by the LLM, then after similarity filtering, and finally after dropping triple repetitions.
Formed tails excluding LLM hallucinated (non-unique)
Formed tails (non-unique)
After ß-filtering (non-unique)
Final: repetitions dropped (unique)
a value
at 0.55
1,760,088
1,536,581
138,201
109,293
6.1 Description of the Extracted KG
Table summarizes the statistics of the extracted KG. inherits the 28 relations from the seed KG (men- tioned in Table 2) but contains approximately four times as many triples; the comparison chart (Fig. E1) is presenled in Appendix E. The extracted triples include vocabulary that extends beyond the seed KG. As expected, novel heads originate from the training dataset, which has richer vocabulary than the seed KG. GRAPHMERT also generates novel toils, predicting tokens from the dataset vocabulary that are absent in the seed KG (see an example given in Table E1 in Appendix C).
6.2
Triple-level Evaluation
Next, we present results for triple-level evaluations.
6.2.1
FActScore* Results
We use two prompts to evaluate triple based on:
context only,
context and model's internal knowledge of general truth.
The first prompt includes the text in black only, the second prompt includes the text in black and teal:
Prompt. You will evaluate the quality of triples for medical knowledge graph on diabetes and its comorbidities. For each triple, you are given:
A sequence providing context
A head entity, a relation, and a tail entity
Your task: Accept the triple ("[yes]") or reject it ("[no]") based on:
Logical alignment: the tail must logically align with the head and relation; relation must match entity types.
the sequence should support the triple. Allow statements that are factual and general truth, even if not perfectly aligned with context, but still avoid contradictions. If the triple has no reliable support, rejuct the triple.
Context support:
Knowledge value: the triple must add new, medically meaningful information the graph. Output only " "[yes]" or "[no]" your final judgment. Wrap your reasoning in Shhink>..<<thinnn
Table 5 reports the FActScore* of GRAPHMERT KG versus the Qwen3-32B LLM KG (bascline) using Qwen3-32B and Gemini 2.0 Flash as the validators. In addition, we compare our KG with Grok 4 LLM KG and Qwen3-14B KG samples. Since Gemini 2.0 Flash does not respond with a definitive yes/no judgment for significant number of queries, we include Table with its FActScore* results breakdown.
We conduct two variants of the evaluation, differing only an extra accept condition in the prompt: 1) based contert only, closely following the original FActScore, and 2) based context model's 's internal
28

---

Published in Transactions on Machine Learning Research (02/2026)
Table 5: FActScore* KG evaluation (in percentage points).
Bold: better than Qwen3-32B baseline; underlined bold: best in column. Percentages computed per KG type with respect to its own #triples.
Model size
KG type
#triples
Qwen3-32B as a judge
Gemini 2.0 Flash as a judge
Context + General truth
Context + General truth
Context only
Context only
Qwen3-32B (baseline)
32B 80M
515,460
40.2
48.1
49.1
44.3
GRAPHMERT
138,201
69.8
72.2
79.7
84.0
Reference LLM KGs
Grok 4 Fast
1.7T
100,000
79.4
79.7
73.3
67.9
Qwen3-14B
14B
100,000
46.4
45.7
62.4
57.6
Table 6: FActScore* KG evaluation with Gemini 2.0 Flash: breakdown.
Bold: better than Qwen3-32B baseline; underlined bold: best in column. Percentages are computed per KG type with respect to its own #triples.
Model size
KG type
#triples
Context only
Context + General truth
yes
no
no verdict
yes
no
no verdict
Qwen3-32B (baseline)
32B
515,460
49.1
41.7
9.2
44.3
30.2
25.4
GRAPHMERT
80M
138,201
79.7
18.3
2.0
84.0
12.8
3.2
Reference L.M KGs
Grok Fast
1.7T
100,000
73.3
8.0
18.7
67.9
6.6
25.5
Qwen3-14B 14BB
14B
100,000
62.4
20.4
17.2
57.6
18.2
24.2
knowledge, which additionally accepts triples that express general truths even not explicitly stated in the context. Variant (2) acknowledges cross-datasel concept linking, whereas (1) restricts evidence to the local context. Because the same triple originate from multiple sequences, some triples are checked more than once against different texts. As result, the reported number of triples exceeds the count in the final deduplicated KGs. We also score the seed KG with respect to the sequences into which its triples are injected. In this setting, FActScore primarily reflects the injection relevance.
Why Qwen3-32B LLM KG FActScores low? Surprisingly, the Qwen3-32B LLM KG scores poorly even when validated by the same model, which flags many of its own errors. This underscores weak prompt steerability: Knowledge may exist in parameters, yet prompt- based generation fails to elicit correct, ontology-respecting triples. Our analysis of Qwen3-32B responses points to three recurring failure modes:
Relation misintorpretation: The model maps relations by lexical similarity rather than ontological meaning, drawing on its internal knowledge.
2. Systematic malformed repetition: The same ill-formed triples reappear across different text chunks.
Overlinking a waid pair: Multiple, largely invalid relations are assigned to the same entity pair.
Illustrative errors
Misinterpreted relation: (diabetes, has fass_fidingg_et site, urban area). Here, "sitc" is treated as lo- cation (anatomical structure is expected). Likewise, (telchealth intervention, has has_part, diabetes) misuses has part has_partt tapppppaaaaaaaa to describe care; correct. variant: (diabetes, focus of, telehealth intervention..
29

---

Published in Transactions on Machine Learning Research (02/2026)
Table ValidityScore evaluation (in percentage points).
Bold: better than Qwen3-32B bascline; underlined bold: best in column. Percentages are computed per KG type with respect to its own #triples.
Model size
KG type
#triples
Qwen3-32B as judge
Gemini 2.0 Flash a judge
yes (Validity- maybe Score)
yes (Validity- maybe no Score)
miss- ing
miss- ing
no
Qwen3-32B
32B
515,460
43.0
24.1
31.4
1.5
66.0
33.6
0.3
0.1
(baseline)
UMLS Seed KG
N/A
28,533
53.4
10.0
34.7
1.9
82.5
16.5
1.1
<
GRAPHMERT
SOM
138,201
68.7
18.3
10.8
2.2
81.3
18.5
0.2
0.1
Reference LLM KGs
Grok Fast
1.7T
100,000
56.9
21.9
21.2
< 0.1
68.8
30.9
0.3
< 0.1
Qwen3-144 14B
14B
100,000
31.8
21.9
46.2
0.1
53.4
45.7
0.9
0.1
Overlinking (redundant relations): For (obesity, diabetes,, the LLM proposed is wwommmiiiaaiinnnmmmmmmmmm of, part_of of, plays_role cause_of, causative causative_agent_ot has component. However, only associated with is valid.
has_ patthological_proceee
Spurious co-occurrence link: (diabetes, has has_aaasaaeaaaae mettod, mammography). The entities merely co- occurred in an abstract; mammography not used for diabetes, yet the LLM infers a spurious relation.
"Spurious co- ro-occurrence link" illustrates a disadvantage of local KG extraction methods vs. global methods that capture how concepts are distributed and connected across the full dataset. As a result, spurious links between concepts that rarely co-occur remain weak and less likely be predicted.
Broad, pre-trained LLMs tend to project general associations into domain-specific relations, yielding invalid triples. High-stakes domains benefit from compact, domain-tailored models and pipelines that enforce ontology and provenance.
Implication:
We want to note that a higher FActScore* of Qwen3-144 14BB in some columns does not reflect a stronger KG. Qualitatively, Qwen3-14B often produces short, generic tails (e.g,, "disease," "complication") that the judge can accept as context-compatible while largely ignoring the head and relation, which reflected in its lower ValidityScore. Qwen3-32B, in rontrast, tends to generate longer, more specific tails that try to capture the full semantic relation; these are penalized more harshly by FActScore* but remain more ontology-consistent overall.
6.2.2 ValidityScore
Validity check on the whole KG: To assess the semantic quality of triples in GRAPHMERTT- and LLM- extracted KGs, we conduct validity check: Test the validity of cach triple in the graph with Qwen3-32B and Gemini 2.0 Pro as the judge LLMs.
Table 7 summarizes the validity checks across KGs. GRAPIMERTT attains markedly higher "yes" rate, i.e., ValidityScore, than the LLM bascline (68.8% vs. 43.0% with Qwen3-32B), indicating substantially cleaner relation usage and predicate hygiene. Overall, the LLM KG is less conservative (more details ahead) and less valid. The typical malformed triples are shown in Tables (GRAPIMERT)) and 9 (LLM), which illustrate the characteristic failures of each model.
Validity check on a KG subset with GPT-5: with GPT-5 (Thinking). GPT-5 particularly strict in evaluating consistency of triples and whether the relation is used in the correct direction, aligning with goal of obtaining dependable judgments. We
We also perform validity check on small KG subsets
30

---

Published in Transactions on Machine Learning Research (02/2026)
yes
maybe
no
100
80
Triples
40
20
triglyceride
t2dm
hyperglycemia
dyslipidemia
adiponectin
metformin
sgit2
cvd
G 52/26/22
G 60/26/14
433/43/14
G 59//30/11
G 41/44/15
G 34/60/6
G 60/27/13
G 60/33/7
52/25/223
34/21/45
L37/31/32
38/23/39
L38/2141
23/19/558
31/18/51
41/24/45
Entities
Left bar GLM, Right bar LLM
Figure 12: Validity check with GPT-5 Thinking, 100 random triples per keyword. The keywords are lined the r-axis. Within each calegory, the left bars represent GRAPHMERT triples and right bars Qwen3-32B LLM triples. The captions below the labels show yes/ res/maybe/no split counts for GLM and LLM.
Table 8: GRaPIMERT-extraateed KG triples: Malformed triple examples
Triples with an incomplete tail
Comment
CKD risk prediction model, associated__wite validated
adjectival tail
gestational diabetes mellitus, associated with, twin elastin, has modification, carbam
should be twin pregnancy should be carbamylation
Triples with overstated causality
Comment
insulin resistance CKD, cause_od metabolic syndrome insulin resistance in CKD, cause of, vascular disorder diabetes mellitus, cause of, tuberculosis
a criterion
vague tail
cause is mycobacterium; DM
risk
Triples with vague tails
Comment
blind patients with DM, associated with, diabetic retinopathy atherosclerotic cardiovascular disease, associated with, lifestyle
head is a cohort
vague tail
Triples with predicate misuse
Comment
atherosclerotic cardiovascular disease, has causative agent, apolipoprotein
should be
apolipoprotein
B-containing lipoproteins
type diabetes mellitus, associated with, parasitic infection glucocorticoid inducced hyperglycaemia, has finding site, tissue
weak/unspeciff link
wrong site (should be blood)
retrieve 100 random triples across seven diabetes-related keywords: T2DM, hyperglycemia,, dyslipidemia, adiponectin, metformin, SGLT2, and CVD. Each triple is assigned yes yes/maybe/aie /no verdict (Fig. 12). The keywords are drawn from the 2000-word datasct-level summary generated with Gemini 2.5 Pro, representing some of the most frequent and clinically relevant terms related to diabetes and its comorbidities.
Overall, the GRAPHMERT KG consistently produces a higher proportion valid (yes) triples and fewer incorrect (no) triples across all keywords, whereas the LLM KG shows more relation misuse and ontology violations, reflected a greater share of maybe and no verdicts. This highlights the more conservative but domain-appropriiate character of the GRAPHMERT KG compared to the noisier LLM KG. According to
31

---

Published in Transactions on Machine Learning Research (02/2026)
Table 9: Qwen3-32B LLM-extracted KG triples: Malformed triple examples
Triples with reversed relation
Comment
cryptogenic stroke, causative causative_agent_ook paradoxical embolism ischemic stroke, cause of, ps glucocorticoids, has has_part, steroidogenesis
reversed causality
PS Protein S deficiency; reversed steroidogenesis produces GCs
Triples with predicate misuse
Comment
ischemic stroke, cause_ of, lelomere length
biologically wrong cells aren't "found" in elastin
elastin, finding_site _iii__st_______ ,o,, vascular smooth muscle cell
CKD awareness, has component, race/ethhicity
CKD chromic kidney disease; ontologically wrong
lockdown, associated with, bone mineral density elastin, has pathological process, matrix metalloproteinase
mismatches ontological types
metalloproteinase is an enayme
Triples with ill-defined target chronic kidney disease (CKD), cause of, renal retinopathy
Comment
tail isn't standard retinal disorder term
Table 10: FActScore* after validity check with Qwen3-32B judge (in percentage points)
KG type
#triples
Context only Context and General truth
Qwen3-32B LLM (baseline)
221,632
55.6
71.9
GRAPHMERT
96,020
76.9
80.2
verdicts (a very short reason why), the LLM often violates ontology, confusing methods with diseases, and misuses relations, with the two errors reinforcing each other. Next, we analyze the main errors of each KG separately.
The main issues in GRAPIMERT triples are vagueness and incomplete tails, though they remain domain-appropriate as valid tail during token combination. We observe this effect across all helper LLMs we tested, including Gemini Flash 2.0 and 2.5. Tail vagueness occurs when GRAPHMERT does not rank the required tail tokens within its top-20 predictions; the helper I.M.M then stitches together a completion that is contextnally acceptable but semantically weak. This also explains most of the cases of overstated causality and predicate misuse: missing key tokens required for high-quality tail completion, the LLM still attempts a plausible but semantically weak completion. A practical mitigation is to exclude low-information 128-token sequences in the prediction stage (c.g., segments dominated by measurements, dosages, or numbers). We currently do not filter these sequences, which may contaminate the extracted KG.
GraphMERT:
Tail incompleteness arises when the helper LLM accepts an incomplete token
LLM: contrast, we observe that the LLM misinterprets UMLS biomedical relations and substitutes them with its broader internal knowledge, resulting in approximations that violate the outology. Designing prompts to fully explain all relations impractical. Our experiments show that even multiple examples fail steer the model consistently: It defaults to its own internal semantics. We also observe systematic relation reversal, consistent with findings in (Berglund et al., 2024). More broadly, extracting well-formed triples requires capturing semantic rather than synlactic representations, which is challenging for LLMs trained primarily on surface text.
To exemplify the difference between GRAPHMERT and LLM KGs explicitly, for the same prompt, we provide Tables E3, E4, E5, E6 with GPT-5 verdicts of triples sampled from GRAPHMERT- extracted KG and LLM-extracted KG in Appendix
Examples:
FactScore after validity check: We further assess the factuality of KGs after applying the validity check, where all invalid triples are removed before rerunning the FActScore evaluation. The results, summarized in Table 10, show that LLM-bbsed cleaning, which primarily eliminates malformed triples, further improves
32

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: GraphRAG KG evaluation on filtered ICD-Bench dataset, organized by difficulty levels
Difficulty
Trivial
Very Easy
Easy
Medium
Hard
Very Hard
Average
# Questions
20
26
8
6
2
7
69
Qwen3-328 LLM KG (baseline)
56.7
68.0
33.3
38.9
0.0
9.5
50.2
Seed KG
56.7
71.8
37.5
55.6
0.0
4.8
53.1
GRAPHMERT
66.7
79.5
50.0
50.0
0.0
0.0
59.4
factuality, with GRAPIMERT reaching 76.9%. The larger FActScore gain for the LLM KG in the Contert & General truth setting compared to the Contert only case indicates that the LLM relies heavily on its background knowledge when generating KGs.
6.3 Graph-level Evaluation
To evaluate the extracted KG, we apply GraphRAG to the filtered benchmarks and assess accuracy based the success rate of the model in answering the questions. We compare the accuracy by using different KGs the primary source of information in response generation.
We evaluate the GRAPIMERT KG against the baseline LLM KG on the filtered endocrinology subset of ICD-Bench, reporting the average question-answeriig accuracy across three runs. The evaluation stratified by difficulty level, a label from the original benchmark. Our filtered test set includes 20 trivial questions (corresponding 1-hop QA), and total of questions across difficulty levels ranging from very easy very hard, based the difficulty definitions from ICD-Bench. The detailed results presented in Table 11. We also test the KGs popular medical benchmarks, which are described in Appendix F. The findings demonstrate that the GRAPHMERT KG consistently outperforms the bascline. Across the whole filtered subset, our framework achieves an overall accuracy gain of 9.2% on ICD-Bench, and 1.7% to 3.7% gain on other medical benchmarks. This highlights the ad advantages of the GRAPHMERT KG for downst ream medical question-answering tasks.
6.4 Ablations
To validate our design choices and understand the contribution of different components, we conduct series of ablation studies. We analyze the sensitivity of GRAPHMERT to its core hyperparameters, its robustness to the density of the seed KG, and the effect of similarity and fact-checking.
6.4.1 Training GraphMERT with Less-aelevant Injections
Our framework relies on two key similarity thresholds: the injection threshold a, which determines the relevance threshold of seed triples used for training, and the acceptance threshold 8, which filters the final triples generated by the pipeline. To find the optimal configuration, we perform grid search over range of values for both parameters.
Hyperparameter a controls the trade-off between the quality and quantity of knowledge injected during training. A higher a imposes a stricter relevance filter, ensuring high-quality injections but limiting their volume and diversity. Hyperparameter 3 serves as grounding function for the final triples on the source sentence. The bascline LLM KG filtered in each experiment include only the relations available for the corresponding value, ensuring equitable evaluation.
The results are presented in Table 12 and Fig. 13. We observe that performance peaks at 0.55 and B 0.67. Lower a's likely introduce noisy, contextually irrelevant triples that degrade performance. Conversely, higher a'' 's appear overly restrictive, preventing the model from leveraging a sufficient breadth of knowledge. In addition, = 0.67 gives good results, underscoring the importance of a final quality check on generated triples. A higher optimal 8 strengthens the importance of cross-document understanding for triple generation, which is not possible in LLM-generated KGs. The optimal configuration achieves a 9.2% improvement over the baseline LLM KG.
33

---

Published in Transactions on Machine Learning Research (02/2026)
Table 12:
GraphRAG accuracies for different a and B. Performance gain versus Qwen3-32B LLM KG is
shown in parentheses.
value
Qwen3-32B LLM KG
Seed KG
0.62
3 0.65
3 = 0.67
B
0.50
58.0
51.7
55.1 (-2.9)
55.1 (-2.9)
54.6 (-3.4)
0.53
55.1
51.2
60.4 (+5.3)
55.1 (+0.0)
58.9 (+3.8)
0.55
50.2
53.1
57.5 (+7.3)
59.2 (+9.0)
59.4 (49.2)
52.2 (+2..0)
0.57
51.2
53.1
56.0 (+4.8)
54.6 (+3.4)
52.2 (+1.0)
0.60
53.1
49.8
53.6 (+0.5)
52.7 (-0.4)
53.6 (+0.5)
Accuracy
0.60
50
55
0.57
a value
0.55
2
0.50
-2
KG
Seed KG
B-0.62
p=
P=0.67
p=0.69
Figure 13: GraphRAG accuracies for different and 8. Bubble size corresponds to absolute accuracy and color indicates the accuracy gain relative to the Qwen3-32B LLM KG baseline (red denotes positive gain, blue denotes negative gain).
6.4.2 Training GraphMERT with a Smaller Seed KG
To evaluate the dependency of GRAPIMERTT on the density provided knowledge, we conduct an ex- periment to measure its performance with sparser seed KG. A robust system should be able to function effectively even when the initial knowledge base is incomplete.
Using the optimal hyperparameters identified previously (a 0.55 and 3 = 0.67), simulate varying levels of knowledge sparsity by randomly removing 25%, 50%, and 75% of the triples from the original seed KG before executing our pipeline.
The results, shown in Table 13, indicate that while performance generally decreases as the seed KG becomes sparser, our framework remains effective. Fven with 75% of the seed knowledge removed, GRAPHMERT still outperforms the baseline LLM KG by 3.86%. This finding highlights the robustness of our approach, demonstrating that it can effectively leverage even sparse set of seed triples to generate high-quality KG.
6.4.3 Ablating GraphMERT Components
Next, ablate architectural components and the training objective:
No-span MLM/MNM: Replace the span masking objective with simple one-token MLM/MNM masking objective.
34

---

Published in Transactions on Machine Learning Research (02/2026)
Table 13: GraphRAG accuracies upon dropping part of the seed KG. Performance gain versus Qwen3-32B
LLM KG is shown in the last column.
KG type
GraphRAG accuracy
Gap vs. Qwen3-32B LLM KG
GRAPIMERT
59.4
99.2
GRAPUMERT (remooe 25% of seed KG)
54.6
+4.44
GRAPHMEITT (remove 50% of seed KG)
56.5
+6.3
GRAPHMERT (remove 75% of seed KG)
54.1
+33.9
Table 14: GraphRAG accuracies when ablating GRAPHMERT features
KG type
Trivial
Very Easy
Easy
Medium
Hard
Very Hard
Average
GRAPHMERT
66.7
79.5
50.0
50.0
0.0
59.4
GRAPHMERT (no-span MLM/MNM)
66.7
76.9
29.2
61.1
9.5
58.0
GRAPHMERT (no H-GAT)
65.0
68.0
33.3
55.6
0.0
0.0
53.1
GRAPHMERT (no dropout)
56.7
65.4
45.8
50.0
0.0
14.3
52.2
Table 15: FActScore* KG evaluation with Qwen3-32B (in percentage points) when ablating GRAPHMERT features
KG type
#triples
Context only
Context and General truth
GRAPHMERT
138,201
69.8
72.2
GRAPIMERT (no-span MLM/MNM)
185,768
69.1
72.2
GRAPHMERT (no H-GAT)
155,311
68.3
69.9
GRAPHMERT (no dropout)
138,363
69.0
70.9
No H-GAT:
Switch off graph attention. predictions are purely syntactic.
This implies training without relation embeddings; the model
No dropout: Switch off dropout on relation embeddings.
Before discussing the ablation results, we outline the main observations for each one:
In the "no H-GAT" ablation, we observe large number irrelevant tokens in the top-k predicted tokens, with commas and articles being primarily predicted in the top 3. This supported by findings by García-Silva al. (2023): They attempted complete triples with top-k tokens extracted from the BERT model in a purely syntactic manner, and had to rely stop word list to make the results usable.
Disabling dropout leads to overfitting on the seed KG vocabulary; hence, less diverse tails.
Training with no-span MLM/MNM objective produces simpler tail completions (1-2 tokens long), because each individual candidate is not well aligned with the others.
Table 14 demonstrates that the full GRAPHMERT KG configuration achieves the highest performance. While the full model achieves the best results, the variant without span-masking performs only slightly worse. In contrast, the removal of either dropout or H-GAT leads to a substantial decrease in accuracy, underscoring their importance for robust. performance.
Table demonstrates that the full model FActScore* the highest: removing dropout or H-GAT lowers acceptance and increases rejections. GRAPHMERT without span-masking reaches the same FActScore* in "Context and General truth" case.
35

---

Published in Transactions on Machine Learning Research (02/2026)
Table 16: Validity check with Qwen3-32B (in percentage points)
when ablating GRAPHMERT features
KG type
#triples
yes
may
no
GRAPHMERT
138,201
68.8
18.3
10.8
GRAPIMERT (no-span MLM/MNM)
185,768
69.3
18.2
10.3
GRAPIMERT (no dropout)
138,363
66.0
19.2
12.0
GRAPHMERT (no H-GAT)
155,311
67.5
17.5
12.3
Table 17: FActScore* KG evaluation (in percentage points) of GRAPIMERT Iteration with Qwen3-32B
judge
KG type
#triples
Context only
Context and General truth
GRAPHMERT
138,201
69.8
72.2
GRAPHMERT with Qwen3-32B LLM KG instead of UMLS KG
589,338
74.4
76.7
GRAPHMERT with expanded KG instead of UMLS KG
492,271
77.0
79.0
Table 18: ValidityScore KG evaluation (in percentage points) of GRAPHMERT Iteration 2 with Qwen3-32B as judge
KG type
#triples
yes (ValidityScore)
maybe be
no
GRAPUMERT
138,201
68.7
18.3
GRAPHMERT with Qwen3-32B LLM KG instead UMLS KG
589,330 389,338
58.0
30.9
8.7
GRAPUMERT with expanded KG instead of UMLS KG
492,271
65.1
24.0
8.1
GRAPHMERT without span-masking achieves the best acceptance (69.3% yes, 10.3% no) (Table 16), but, according to GraphRAG, the KG remains less informative overall. In effect, the KG obtained from this variant tends to be populated with trivially correct tiples, i.e. (diabetes, is_a, disease). This leads to higher rate of successful tail completion: 185k against 138k with span masking. Such short, obvious facts pass validity check, manifesting in a higher ValidityScore. However, this simplicity comes at a cost: No- span-masked completions lack the nuance and granularity provided by span masking, resulting in poorer coverage and a loss of fine-grained domain details. This trade-off stresses the importance of evaluating KGs both at the graph and triple levels. We advise employing token-level MLM/MNM when the simplicity of triples is not limitation.
6.4.4 GraphMERT Iteration 2: Retraining
Next, we address two questions:
What a seed KG for the domain of interest is not available?
2. Can we further improve and expand the GRAPHMERT KG through iterative retraining?
Retraining with the LLM KG replacing the seed KG. A key limitation of the pipeline described so far is its reliance on a seed KG, which may not be available in some domains. A practical alternative is to generate a seed KG using an LLM and apply GRAPIMERT to it, rather than construct the entire KG using an LLM. To test this approach, we replace the gold-standard UMLS seed KG with the Qwen3-32B L.M-generated KG and rerun the GRAPHMERT pipeline.
Retraining with the expanded KG replacing the seed KG. After running the pipeline once, GRAPH- MERT extracts ontosogy-consittent triples grounded in the source text. We can then augment the UML.S
36

---

Published in Transactions on Machine Learning Research (02/2026)
seed KG with these extracted triples and retrain the model on this expanded KG. This provides additional examples and may, in principle, yield further performance gains.
The FActScore* and ValidityScore results for these two experiments are shown in Tables 17 and 18. When using the Qwen3-32B LLM-generated KG instead of the UMLS seed KG, the GRAPHMERT pipeline attains a higher FActScore*, surpassing that obtained with the UML.S seed KG. However, the ValidityScore drops notably by 10.1%, which is expected since the LLM--gnerated KG does not strictly follow the ontology, resulting in lower-quality examples. Ultimately, although, when available, using gold-standard seed KG is preferable, generating a seed KG with an LLM from the training dataset and subsequently training GRAPHMERT remains practical and effective alternative when seed KG is not available.
When retrained with the expanded KG rather than the UMLS KG, GRAPHMERT achieves the highest FActScore*, while the ValidityScore decreases slightly (by 3.6%). Notably, both alternative "seed KGs" (the LLM-generated and expanded versions) also lead to larger extracted KG size, which may be attributed to their larger size.
7 Discussion, Limitations, and Future work
Our results show that GRAPIMERT yields higher factuality and validity of triples than LLM- based KG extraction while preserving ontology fidelity. We observe a consistent difference between GRAPH- MERT and LLM KGs in relation usage and predicate hygiene. GRAPHMERT generally employs relations correctly, aligns tails with heads, and preserves biomedical categorics (discases, syndromes, complications). The LLM KG often misuses or reverses the direction of relations, mixes categories in ontology-violating ways (e.g., socio-economic with biomedical), and produces inverse paarsse/llltttee /ill-ttttly ill-typed statements. The GRAPHMERT KG has far fewer ontology violations and hews closer to UMLS. The GRAPIMERT KG vocabulary is more con- servative: The conservativeness can be explained based on the limited scope of the seed KG vocabulary and its tendency to restate head tokens (mimicking UMLS (autological triples). While generally more accurale, GRAPHMERT may still produce incomplete or vague triples.
Discussion:
We attribute the difference in factuality and validity of triples to the usage of semantic relation embeddings, which move predictions toward ontology-aligned ones. Triele-level error analysis shows GRAPHMERT may sometimes extract incomplete and vague but often domain- -amaain-ppprpprate tails with factual relations, whereas LLMs usually produce diverse tails but frequently misuse or reverse the direction of relations. Our graph- level evaluation may conflate the KG signal with backbone model knowledge under GraphRAG; future work will include graph-level metrics that isolate the contribution of KGs as well as relation-aware retrieval.
The main limitation of GRAPHMERT its reliance on the seed KG. First,, running the framework requires a high-quality seed with 100-1000 examples per relation. Second, once training is com- plete, the relation set is fixed. Hence, adding new relations requires retraining. A further limitation is its dependence on helper L.M for tail combination, which introduces occasional incompleteness in the extracted triple tails. If the helper LLM lacks sufficient expertise in the target domain, it may struggle to construct accurate long multi-token completions, which can limit the reliability of the extracted KG. It may also reintroduce a risk of hallucination or fon-factual tail combinations, even though the triple can- didates are constrained by the GRAPHMERT model. This can be mitigated by using domain--peccialzzed helper models, when available, or applying simple post-filters (e.g., ontology checks, part-of-speech filters) to discard low-confidence tails. Alternatively, one can select only triples that have passed ValidityCheck. As neural model, GRAPHMERT also tends to prioritize frequent entities, potentially overlooking rare but meaningful ones. Further, our experiments are based on a small 125M-token domain-specific text corpus. In principle, GraphMERT architecture (Transformer + graph injection) should scale to larger text corpora and more relation types, but billion- token training regimes remain to be studied systematically. In addition, we did not evaluate the model's ability to predict or handle numerical tokens. Finally, its robustness to unseen entities remains untested: extracting entirely novel concepts from new texts would likely require fine-tuning, limiting adaptability in fast-changing domains.
Limitations:
37

---

Published in Transactions on Machine Learning Research (02/2026)
Future work: We plan to improve the KG quality by unifying entity spellings, replacing the fixed top-k limit with high-probability token selection. We aim to extend GRAPHMERT to direct multi-token span prediction in the semantic space, reducing reliance on the helper LLM for tail token combining. Further, we plan to conduct more rigorous graph-level evaluations, since GraphRAG alone often blends KG information with model knowledge, and does not guarantee retrieval of the most relevant subgraph, as its retrieval is entity-guided and relations play only a sccondary role. It may also be valuable to investigate how the GRAPIMERT evaluation loss correlates with the quality of the extracted KGs. We also plan to extract and evaluate KGs for other domains and employ them in various downstream applications.
8 Conclusion
We proposed framework for automatically extracting domain-specific KGs from unstructured, sentence- level text. Central to this work is our idea of encoding both semantic and syntactic information into textual chain graphs, new representation we introduced. To operate in this space, we presented GRAPHMERT, a transformer-based model that unifies an encoder-only architecture with graph attention. Together, these contributions enable the distillation of explicit semantic structures from trained neural networks, bridging neural and symbolic representations and advancing interpretable, reliable KG construction.
We also outlined KG-powered applications, underscoring that future progress hinges reliable, factual KGs and other high-level explicit abstractions that embody collective knowledge while remaining compatible with AI inference and downstream lasks. We argue that neural-KG integration is a key step toward domain- specific superintelligence. Looking ahead, we foresee the research community embracing the neurosymbolic paradigm, where explicit, auditable, and evolving KGs complement neural inference that is approximate, efficient, and capable of handling ambiguity.
Acknowledgments
This work was supported by NSF under Grant No. CNS-2216746. The authors are pleased to acknowledge that the work reported on in this paper was substantially performed using the Princeton Research Computing resources at Princeton University. Princeton Research Computing is a consortium groups including the Princeton Institute for Computational Science and Engineering (PICSciF) and Research Computing at Princcton University.
References
Titipat Achakulvisut, Daniel Acuna, and Konrad Kording. PubMed parser: A Python parser for PubMed open-access XML subset and MEDLINE XML dataset. Journal of Open Source Software, 5(46):1979, 2020.
Kian Ahrabian, Xinwei Du, Richard Delwin Myloth, Arun Baalaaji Sankar Ananthan, and Jay Pujara. PubGraph: A large-scale scientific knowledge graph, 2023. arXiv:2302.0223) [es.AT].
Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards tracing knowledge language models back to the training data, December 2022. In Findings of the Association for Computational Linguisties: 2022 Conference on Empirical Methods in Natural Language Processing.
Laith Alzubaidi, Aiman Al-Sabaawi, Jinshuai Bai, Ammar Dukhan, Ahmed H. Alkenani, Ahmed Al-Asadi, Haider A. Alwzwazy, Mohamed Manoufali, Mohammed A. Fadhel, A. S. Albahri, Catarina Moreira, Chun Ouyang, Jinglan Zhang, Jose Santamaría, Asma Salhi, Freek Hollman, Ashish Gupta, Ye Duan, Timon Rabczuk, Amin Abbosh, and Yuantong Gu. Towards risk-free trustworthy artificial intelligence: Significance and requirements. International Journal of Intelligend Systems, 2023(1):4451998 2023.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez- Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis
38

---

Published in Transactions on Machine Learning Research (02/2026)
Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks, 2018. arXiv:1806.0122 [cs.LG].
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives, 2014. arXiv:1206.5533 [cs.LG].
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on "A is B fail learn "B A", 2024. arXiv:2309.122 12288 [C.
Tarek R. Besold, Artur d' Avila Garcez, Sebastian Bader, Howard Bowman, Pedro Domingos, Pascal Hitzler, Kai-Uwe Kuchnberger, Luis C. Lamb, Daniel Lowd, Priscila Machado Vicira Lima, Leo de Penning, Gadi Pinkas, Hoifung Poon, and Gerson Zaverucha. Neural-symbolic learning and reasoning: A survey and interpretation, 2017. arXiv:1711.03002 [cs.AI].
Olivier Bodenreider. The Unified Medical Language System (UMLS): Integrating biomedical terminology. Nucleic Acids Research, 32:267-270, 2004.
Nick Bostrom. Superinlelligenc:: Paths, Dangers, Stralegies. Oxford University Press, 2014.
Samuel R. Bowman. Eight things to know about large language models, 2023. arXiv:2304.00112 [cs.CL].
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021. arXiv:2104.13478 [cs.LG].
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. In Advonces in Neural Information Processing Systems.
Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. Knowl- edgeable or educated guess? Revisiting language models as knowledge bases, August 2021. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Salvatore Carta, Alessandro Giuliani, Leonardo Piano, Alessandro Scbastian Podda, Livio Pompianu, and Sandro Gabriele Tiddia. Iterative zero-shot LLM prompting for knowledge graph construction, 2023.
arXiv:2307.01128 [cs.CT]..
Souradip Chakraborty, Ekaba Bisong, Shweta Bhatt, Thomas Wagner, Riley Elliott, and Francesco Mosconi. BioMedBERT: A pre-trained biomedical language model for QA and IR, December 2020. In Proceedings of the 28th International Conference on Computational Linguistics.
Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining?, 2025. In Proceedings of the 38th International Conference on Neural Information Processing Systems.
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cyuthia Rudin, This looks like that: Deep learning for interpretable image recognition, 2019. arXiv:1806. 100464007757 [cs.T.G].
Hanzhu Chen, Xu Shen, Qitan Lv, Jie Wang, Xiaoqi Ni, and Jieping Ye. SAC-KG: Exploiting large language models as skilled automatic constructors for domain knowledge graph, August 2024. In Proceedings of the 62nd Anseual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
39

---

Published in Transactions on Machine Learning Research (02/2026)
Hanzhu Chen, Xu Shen, Jie Wang, Zehao Wang, Qitan Lv, Junjie He, Rong Wu, Feng Wu, and Jieping Ye. Knowledge graph finetuning enhances knowledge manipulation in large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuctao Ma, Yifan Yanggong, and Junbo Zhao. Maybe only 0.5% data is needed: preliminary exploration of low training data. instruction tuning, 2023a. XXXi:2335509226 [cs.Al].
Mingyang Chen, Wen Zhang, Yuxia Geng, Zezhong Xu, Jelf Z. Pan, and Huajun Chen. Generalizing to unseen elements: A survey on knowledge extrapolation for knowledge graphs, 2023b. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence.
Zhisong Chen and Ching Y. Suen. Measuring the complexity of ruse- expert systems. Eapert Systems with Applications, 7(4):467-481, 1994.
Kewei Cheng, Nesreen Ahmed, Ryan A. Rossi, Theodore Willke, and Yizhou Sun. Neural-symbolic methods for knowledge graph reasoning: A survey. ACM Transactions on Knowledge Discovery from Data, 18(9), November 2024.
Alain Colmerauer and Philippe Roussel. The Birth of Prolog, pp. 331 167. 1996.
Artur d' d'Avila Garcez, Marco Gori, Luis C. Lamb, Luciano Serafini, Michael Spranger, and Son N. Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning, 2019. arXiv:1905.06088 [cs.AT].
Artur S. d' A'ivila Garcez, Krysia B. Broda, and Dov. M. Gabbay. Introduction and Overview. 2002. In Foundations and Applications.
Bhishma Dedhia, Yuval Kansal, and Niraj K. Jha. Bottom-up domain-specific superintelligence: reliable
knowledge graph is what we need, 2025. MrXiv:2507.1396 [cs.
Yihe Deng, Chenchen Ye, Zijie Huang, Mingyu Derek Ma, Yiwen Kou, and Wei Wang. GraphVis: Boosting LLMs with visual knowledge graph integration. In Proceedings of the 38th International Conference on Neural Information Processing Systems, 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding, 2019. arXiv:1810.04805 [cs.CT.].
Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. Spurious correlations in reference-free evaluation of text generation, 2022. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers).
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summarization, 2024. arXiv:2404.16138 [cs.CL].
Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Better synthetic data by retrieving and transforming existing datasets, August 2024. In Findings of the Association for Compulational Linguistics: ACL 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-uugnented generation for large language models: A survey, 2024. arXiv:2312.10997 [cs.CL].
Artur A'vla Garcez and Luís C. Lamb. Neurosymbolic AI: The 3rd wave. Artificial Intelligence Review, 56(11):12387 124066, March 2023.
Andrés Garcia-Silva, Cristian Berrío, and Jose Manuel Gómez-Pérez. Textual entailment for effective triple validation in object prediction, 2023. Tn The Semantic Web ISWC 2023.
40

---

Published in Transactions on Machine Learning Research (02/2026)
R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. Garbage in, garbage out? Do machine learning application papers in social computing report where human-labeled training data comes from? 2020. In Proceedings of the Conference on Fairness, Accountability, and Transparency.
Hatem Ghanem and Carlos Cruz. Fino-tuning prompting LLMs: Evaluating knowledge graph con- struction task. Frontiers in Big Data, 8:1505877, June 2025.
Bishwamittra Ghosh, Sarah Hasan, Naheed Anjum Arafat, and Arijit Khan. Logical consistency of large language models fact-checking, 2025. In Proceedings of the Thirteenth International Conference on Learning Representations.
Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. GPT4Graph: Can large lan- guage models understand graph structured data? An empirical evaluation and benchmarking, 2023. arXiv:2305.1500 [cs.AI].
Rajan Gupta, Gaurav Pandey, and Saibal Kumar Pal. Automating government report generation: A gener- ative AT approach for efficient data extraction, analysis, and visualization. Digital Government: Research and Practice, 6(1), February 2025.
Lovisa Hagström, Denitsa Saynova, Tobias Norlund, Moa Johansson, and Richard Johansson. The effect of scaling, retrieval augmentation and form on the factual consistency of language models, December 2023. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(())3335 346, 1990.
John Haugeland. Artificial Intelligenc:: The Very Ideo. 1985.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2023. arXiv:1606.08415 [cs.LG].
Pascal Hitzler, Aaron Eberhart, Monireh Ebrahimi, Md Kamruzzaman Sarker, and Lu Zhou. Neuro-symbolic approaches in artificial intelligence. National Science Review, 96). 2022.
Marvin Hofer, Daniel Obraczka, Alieh Saeedi, Hanna Köpcke, and Erhard Rahm. Construction knowledge graphs: Current state and challenges. Information, 15(8), 2024.
Jiri Hron, Laura A. Culp, Gamaleldin Fathy Flsayed, Rosanne Liu, Jasper Snoek, Simon Kornblith, Alex Rizkowsky, Isabelle Simpson, Jascha Sohl-Dickstein, Noah Fiedel, Aaron T. Parisi, Alexander A. Alemi, Azade Nova, Ben Adlam, Bernd Bohnet, Gaurav Mishra, Hanie Sedghi, Izzeddin Gur, Jaehoon Lee, John D. Co-Reyes, Kathleen Kenealy, Kelvin Xu, Kevin Swersky, Igor Mordatch, Lechao Xiao, Maxwell Bileschi, Peter J. Liu, Roman Novak, Sharad Vikram, Tris Warkentin, and Jeffrey Pennington. Training language models on the knowledge graph: Insights on hallucinations and their detectability, 2024. In Proceedings of the First Conference on Language Modeling.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models, 2022, In Proceedings of the Inter- national Conference on Learning Representations.
Haoyu Huang, Chong Chen, Zeang Shong, Yang Li, and Wentao Zhang. Can LLMs be good graph judge for knowledge graph construction?, 2025a. arXiv:2411.17888 [cs.CL].
Lci Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, January 2025b.
N. Ibrahim, S. Aboulcla, A. Ibrahim, et al. A survey on augmenting knowledge graphs (KGs) with large language models (LLMs): Models, evaluation metries, benchmarks, and challenges. Discover Artificial Intelligence, 4(1):76, 2024.
41

---

Published in Transactions on Machine Learning Research (02/2026)
Shadi Iskander, Sofia Tolmach, Ori Shapira, Nachshon Cohen, and Zohar Karnin. Quality matters: Evalu- ating synthetic data for tool-using LLMs, November 2024. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Mohamed Yahya Jaradeh, Kuldeep Singh, Markus Stocker, Andreas Both, and Sören Auer. Information extraction pipclines for knowledge graphs. Knowledge and Information Systems, 65(3):1989-0016, 2023.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions Neural Networks and Learning Sys- lems, (3(4):494 514, 2022.
Ziwei Ji, Naycon Loe, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55())), March 2023.
Pengeheng Jiang, Lang Cao, Cao Xiao, Parminder Bhatia, Jimeng Sun, and Jiawei Han. KG-FIT: Knowledge graph fine-tuning upon open- world knowledge. In Advances in Neurol Information Processing Systems, volume 37, pp. 136220-136258. Curran Associates, Inc., 2024.
Jiajie Jin, Yutao Zhu, Yujia Zhou, and Zhicheng Dou. BIDER: Bridging knowledge inconsistency for efficient retrieval-uugmented LLMs via key supporting evidence, August 2024. In Findings of the Association for Compulational Linguistics: ACL 2024.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Compu- tational Linguistics, 8:64-77, 2020.
Zahra Kadkhodaie, Florentin Guth, Rero P. Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations, 2024. arXiv:2310.02577 [cs.CV].
Tal Kadosh, Niranjan Hasabnis, Vy A. Vo, Nadav Schneider, Neva Krien, Mihai Capotä, Abdul Wasay, Guy Tamir, Tod Willke, Nesreen Ahmed, Yuval Pinter, Timothy Mattson, and Gal Oren. MonoCoder: Domein-specific code language model for HPC codes and tasks, 2024. In Proceedings of the IEEE High Performance Extreme Computing Conference.
Rasiq Kamel, Filippo Guerranti, Simon Geisler, and Stephan Günnemann. SAFT: Structure-aware fine- tuning of LLMs for MiR-to-Text generation, 2025. arXiv:2507.1338 [cs.CL].
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. arXiv:2001.08361 [cs.LG].
Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng, Sourco-aware training enables knowledge attribution in language models, 2024. arXiv:2404.01019 [cs.CL].
Yubin Kim, Hyewon Jeong, Shan Chen, Shuyue Stella Li, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minscok Jung, Rodrigo Gamciro, Lizhou Fan, Eugene Park, Tristan Lin, Joonsik Yoon, Wonjin Yoon, Maarten Sap, Yulia Tsvetkov, Paul Liang, Xuhai Xu, Xin Liu, Daniel McDulT, Hyeonhoon Lee, Hae Won Park, Samir Tulebaev, and Cynthia Breazeal, Medical hallucinations in foundation models and their impact on healthcare, 2025. arXiv:2508.05777 [cs.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 444, 2015.
Tao Tei, Regina. Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. Tn Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. The dawn after the dark: An empirical study on factuality hallucination large language models, August 2024. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
42

---

Published in Transactions on Machine Learning Research (02/2026)
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK Adversarial attack against BERT using BERT, November 2020. In Findings of the Associalion for Compulational Linguistics: 2020 Conference on Empirical Methods in Natural Language Processing.
Mingehen Li, Halil Kilicoglu, Hua Xu, and Rui Zhang. BiomedRAG: A retrieval augmented large language model for biomedicine. Journal of Biomedical Informatics, 162:104769, 2025.
Seunguook Lim and Jihie Kim. SapBERT: Speaker-aware pretrained BERT for emotion recognition in conversation. Algorithums, 16(1):8, 2022.
Robert K. Lindsay, Bruce G. Buchanan, Edward A. Feigenbaum, and Joshua Lederberg. DENDRAL: A case study of the first expert system for scientific hypothesis formation. Artificial Intelligence, 61(2):209 261, 1993.
Chnang Liu, Zelin Yao, Yibing Zhan, Xueqi Ma, Shirui Pan, and Wenbin Hu. Gradformer: Graph transformer with exponential decay, 2024a. In Proceedings of the Thirty- Third International Joint Conference on Arlificial Intelligence.
Weijic Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Qi Ju, Haotang Deng, and Ping Wang. K-BERT: Enabling language representation with knowledge graph. In Proceedings of the Association for the Advancement of Artificial Intelligence Conference, volume 34, pp. 2901 2908, 2020.
Y. Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019. arXiv:1907.11692 [cs.CL].
Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. Datasets for large language models: A comprehensive survey, 2024b. arXiv:2402. 18041 Ccs.
Xinyu Lu, Lifang Wang, Zejun Jiang, Shizhong Liu, and Jiashi Lin. MRE: translational knowledge graph completion model based on multiple relation embodding. Mathematical Biosciences and Engineering, 20 (3):5881 5900, 2023.
Linhao Luo, Yuan-Fant Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning graphs: Faithful and inter- pretable large language model reasoning, 2024. In Proceedings of the International Conference Learning Representations.
Andreas Madsen, Sarath Chandar, and Siva Reddy. Are self-explanations from large language models faithful?, August 2024. In Findings of the Association for Computational Linguistics: ACL 2024.
Gary Marcus. Deep learning: A critical appraisal, 2018. arXiv:1801.0063 [cs.AI].
John McCarthy. Cireumscription form of non-monotonic reasoning. Artificial Intelligence, 13(1):27-3) 39, 1980. Special Issue on Non-Monotonic Logic.
Dhruv Mehrotra and Tim Marchman. Perplexity is bullshit machine, 2024. URL https://www.wii bulhhhittmaahhiiee machine/. WIRED, investigation documenting data scraping
com/sstorr//rrrpleitt-----sss--- bulllexxity-------llhhht-- and multiple intiiiiinattionssnnaaaaaaaoo tributions.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-Tau Yih, Pang Koh, Mohit Iyyer, Luke Zettle- moyer, and Hannaneh Hajishirzi. FActScore: Fine-grained alomic evaluation of factual precision in long form text generation, December 2023. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2025. arXiv:2402.06196 [cs.CL].
Prakanya Mishra, Zonghai Yao, Parth Vashisht, Feiyun Ouyang, Beining Wang, Vidhi Dhaval Mody, and Hong Yu. SYNFAC-EDIT: Synthotic imitation edit feedback for factual alignment in clinical summa- rization, November 2024. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
43

---

Published in Transactions on Machine Learning Research (02/2026)
Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. DyKnow: Dynamically verifying time- sensitive factual knowledge in LLMs, November 2024. In Findings of the Association for Compulational Linguistics: 2024 Conference on Empirical Methods in Natural Language Processing.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser- assisted question-answering with human feedback, 2022. arXiv:2112.09:. [cs.CL].
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. Learning attention-based embeddings for relation prediction in knowledge graphs, 2019. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Allen Newell and Herbert A. Simon. Computer science as empirical inquiry: Symbols and search. Commu- nications of the ACM, 19(3):113 126, March 1976.
Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. CulturaX: cleaned, enormous, and multilingual dataset for large language models in 167 languages, May 2024. In Proceedings of the Joint International Conference on Computational Linguistics, Language Resources and Evaluation.
Zach Nussbaum, John Xavier Morris, Andriy Mulyar, and Brandon Duderstadt. Nomic embed: Training reproducible long context text embedder. Transactions Machine Learning Research, 2025.
Jeff Z. Pan, Simon Razniewski, Jan-Christopp Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omeliyanenko, Wen Zhang, Mattco Lissandrini, Russa Biswas, Gerard de Melo, Angela Bonifati, Edlira Vakaj, Mauro Dragoni, and Damien Graux. Large language models and knowledge graphs: Opportunities and challenges. Transactions on Graph Data and Knowledge, 1 (1):2:1-2:38, 2023a.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: roadmap. IEEE Transactions on Knowledge and Data Engineering, 36: 3580 3599, 2023b.
Mayana Pereira, Sikha Pentyala, Anderson Nascimento, Rafael T. de Sousa Jr., and Martine De Cock. Secure multiparty computation for synthetic data generation from distributed data, 2022. arXiv:2210.07332 (c).
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexan- der Miller, Language models as knowledge bases?, 2019. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
Ori Ram, Yoav Levine, Ttay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context aetrieaa--uggeented language models. Transactions of the Association for Computa- lional Lénguistics, 11:1316 1331, 2023.
Qiang Rao and Tiejun Wang. Semantic enhancement based knowledge graph completion for graph convo- lutional neural networks, 2023. In Proceedings of the International Conference on Flectrical, Mechanical and Computer Engineering.
Rohan Rao, Benika Hall, Sunil Patel, Christopher Brissette, and Gordana Neskovic. Insights, techniques, and evaluation for LLM-driven knowledge graphs, December 2024. URL https://devellperr nvidia. com/ wereeighhssttehhiuueee aetessanddevaaa waweees-anddeellaaiioo-------
graphs/. [Online; pub-
lished Dec. 16, 2024; last accessed 26 Jul, 2025).
Rick Rejeleene, Xiaowei Xu, and John Talburt. Towards trustable language models: Investigating information quality of large language models, 2024. arXiv:2401.13088 [cs.CT.].
44

---

Published in Transactions on Machine Learning Research (02/2026)
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612 634, 2021.
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky- Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistie inter- pretability, 2025. scXiv:2501.16490 [cs.AI].
Jude W. Shavlik, Raymond J. Mooney, and GeofTrey G. Towell. Symbolic and neural learning algorithms: An experimental comparison. Machine Learning, 6(2):111- 1443, 1991.
Fobo Shi, Duantengchuan Li, Xiaoguang Wang, Bing Li, and Xindong Wu. TGformer: A graph transformer framework for knowledge graph embedding. IEEE Transactions on Knowledge and Data Engineering, 37 (1):526-541, 2025.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation, November 2021. In Findings of the Association for Computational Linguis- tics: 2021 Conference on Empirical Methods in Natural Language Processing.
Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang, Knowledge unlearning for LLMs: Tasks, methods, and challenges, 2023. arXiv:2311.15766 [cs.CL].
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M Ni, Heung- Yeung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.07697 2023.
Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019.
Konrad Szocik, Bartlomiej Tkacz, and Patryk Gulczyíski. The revelation of superintelligence. Al & Society, 35(3):755-775, September 2020.
A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, Large language models in medicine. Nature Medicine, 29:1930-1941 1940, 2023.
Bozhong Tian, Xiaozhuan Liang, Siyuan Cheng, Qingbin Liu, Mengru Wang, Dianbo Sui, Xi Chen, Huajun Chen, and Ningyu Zhang. To forget or not? Towards practical knowledge unlearning for large language models, November 2024. In Findings of the Association for Computational Linguistics: 2024 Conference on Empirical Methods in Natural Language Processing.
S.M.T.I. Tonmoy, S.M. Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination miligation techniques in large language models. arXiv preprinl 1=Xiv:2401.01110 6, 2024.
Geoffrey G. Towell. Using neural networks. Machine Learning: A Multistralegy Approach, Volume IV, 4: 405, 1994.
Son Tran, Edjard Mota, and Artur d' Avila Garcez. Reasoning in neurosymbolic AI, 2025. arXiv:2505.20313 [cs.Al].
William van Melle. MYCIN: A kmowledge-based consultation program for infoctious discase diagnosis. International Journal of Man-Maching Studies, 10(3):313 322, 1978.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30: 5998-6008, 2017.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn Position: Will we run out of data? Limits of LLM scaling based on the 1st International Conference Machine Learning.
data, 2024. In Proceedings of
45

---

Published in Transactions on Machine Learning Research (02/2026)
Warren J. von Eschenbach. Transparency and the black box problem: Why we do not trust AI. Philosophy 8 Technology, 34(44:1607-1622, 2021.
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Qipeng Guo, Xiangkun Hu, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Xuming Hu, Zehan Qi, Wenyang Gao, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. Survey on factuality in large language models. ACM Compating Surveys, 58(1), September 2025a.
Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan O. Arik. Astute RAG: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models, 2024a. arXiv:2410.07176 [cs.CL].
J. Wang, Y. Liu, P. Li, Z. Lin, S. Sindakis, and S. Aggarwal. Overview of data quality: Examining the dimensions, antecedents, and impacts of data quality. Journal of the Knowledge Economy, pp. 1 20, 2023. Epub ahead of print, PMID: 40479478, PMCID: PMC9912223.
Junda Wang, Zonghai Yao, Zhichao Yang, Huixue Zhou, Rumeng Li, Xun Wang, Yucheng Xu, and Hong Yu. NoteChat: A dataset of synthetic patient-physician physician conversations conditioned on clinical notes, August 2024b. In Findings of the Association for Computational Linguistics: ACL 2024.
Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, and Yunhong Wang. A survey on data synthesis and augmentation for large language models, 2024c. arXiv:2410.122 141.2496.12896 [cs.CL].
Rui Wang, Fei Mi, Yi Chen, Boyang Xue, Hongru Wang, Qi Zhu, Kam-Fai Wong, and Ruifeng Xu. Role prompting guided domain adaptation with general capability preserve for large language models, June 2024d. In Findings of the Association for Computational Linguistics: NAACL 2024.
Xiangyu Wang, Lyuzhou Chen, Taiyu Ban, Muhammad Usman, Yifeng Guan, Shikang Liu, Tianhao Wu, and Huanhuan Chen. Knowledge graph quality control: A survey. Fundamental Research, 165):607-626, 2021.
Xiaoye Wang, Nicole Xi Zhang, Hongyu He, Trang Nguyen, Kun-Hsing Yu, Hao Deng, Cynthia Brandt, Danielle S. Bitterman, Ling Pan, Ching-Yu Yu Cheng, James Zou, and Dianbo Liu. Safety challenges of AT in medicine in the era of large language models, 2025b. arXiv:2409.18968 Jes.CY].
Yuhao Wang, Ruiyang Ren, Junyi Li, Xin Zhao, Jing Liu, and Ji-Rong Wen. REAR: A relevance-aware rotrioval-angmennted framework for open-domain question answering, 2024c. In Findings of the Association for Compulational Linguislics: 2024 Conference on Empirical Melhods in Natural Longuage Processing, Miami, Florida, USA.
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Nenkov Georgiev, Rocktim Jyoli Das, and Preslav Nakov. Factuality of large language models: A survey, November 2024f. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022. arXiv:2206.0768) CL..
Peter West, Chandra ula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: From general language models to commonsense models, July 2022. In Proceedings of the Conference of the North American Chapter of the Association for Compulational Linguislies: Human Languoge Technologies.
World Health Organization. International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10). 1992. URL https://icce who httt/bbowwee1//21999////
Tianle Xia, Liang Ding, Guojia Wan, Yibing Zhan, Bo Du, and Dacheng Tao. Improving complex rea- soning over knowledge graph with logic-aware curriculum tuning. Proceedings of the Association for the Aduancement of Artificial Intelligence Conference, 39:12881 188811128288, 2025.
46

---

Published in Transactions on Machine Learning Research (02/2026)
F. Xiao, L. Zhou, Y. Li, C. Zhang, Y. Liu, H. Yu, X. Li, C. Wang, X. Yin, and X. Gao. Comparison of brain gray matter volume changes in peritoneal dialysis and hemodialysis patients with chronic kidney disease: a VBM study. Frontiers in Newroscience, 18:1394169, 2024.
Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. Large language models for generative information extraction: a survey. Frontiers of Compuler Science, 18, 2024.
Xiao Xu, Xian Xu, Yuyao Sun, Xiaoshuang Liu, Xiang Li, Guotong Xic, and Fei Wang. Predictive modcling of clinical events with mutual enhancement between longiludinal patient records and medical knowledge graph, 2021. In Proceedings of the IEEF International Conference on Data Mining.
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination inevitable: An innate limitation of large language models, 2025. arXiv:2401.11817 [cs.CI].
Wenli Yang, Lilian Some, Michael Bain, and Byeong Kang. A comprehensive survey on integrating large language models with knowledge-based methods. Knowledge-Buud Systems, 318:113503, 2025.
Michihiro Yasunaga, Hongyu Ren, Antoine Bossclut, Percy Liang, and Jure Leskovee. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 535 546, 2021.
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy S. Liang, and Jure Leskovec. Deep bidirectional magguag--kooweeee graph pretraining. Advances in Neural Infor- mation Processing Systems, 11-37399-37322 2022.
Xi Ye and Greg Durrett. The unrcliability of explanations in few-shot prompting for textual reasoning, 2022.
In Advances in Neural Information Processing Systems.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in Newral Information Processing Systems, 34:28877-28888, 2021.
Shenglai Zeng, Jiankun Zhang, Bingheng Li, Yuping Lin, Tianqi Zheng, Dante Everaert, Hanqing Lu, Hui Liu, Hui Liu, Yue Xing, Monica Xiao Cheng, and Jiliang Tang. Towards knowledge checking in retrieval- augmented generation: A representation perspective, April 2025. In Proceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers).
Ruizhe Zhang, Yongxin Xu, Yuzhen Xiao, Runchuan Zhu, Xinke Jiang, Xu Chu, Junfeng Zhao, and Yasha Wang. KnowPO: Knowledge-aware preference optimization for controllable knowledge selection retrieval-angmented language models, 2025. In Proceedings of the Association for the Advancement of Artificial Intelligence Conference.
Yuhao Zhang, Peng Qi, and Christopher D Manning. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2205 2215, 2018.
Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R. Fung, Jing Li, Manling Li, and Heng Ji. Knowledge over- shadowing causes amalgamated hallucination in large language models, 2024. wrXiv:2407.08039 [cs.CL].
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Nie, and Ji-Rong Wen. A survey of [cs.CL].
large language models, 2025. arXiv:2303.188 18223
Shuran Zheng, Xuan Qi, Rui Ray Chen, Yongchan Kwon, and James Zou. Proper dataset valuation by pointwise mutual information, 2025. S.ii:24055.1255 [os.TG].
47

---

Published in Transactions on Machine Learning Research (02/2026)
Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. A comprehensive survey automatic knowledge graph construction. ACM Computing Surveys, 56(4), November 2023.
Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, and Dan Jurafsky. Problems with cosine as measure of embedding similarity for high frequency words, May 2022. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
Lexin Zhou, Wout Schellacrt, Fernando Hartinez-Plumes Yacl Moros-Davall, Cèsar Ferri, and José Hernándezz Orallo. Larger and more instructable language models become less reliable. Nature, 634(8032): 61-68, 2024.
Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. LLMs for knowledge graph construction and reasoning: Recent capabilities and future opportunities. World Wide Web, 27(5), August 2024.
Appendices
A Extended background
A.1 KG with Neural Network: Unifying the Representations
In many tasks, the solution is not to pit symbolic and neural approaches against each other but to combine them in hybrid, modular systems. Neural networks discover statistical patterns through gradient descent; symbolic layers then manipulate extracted structure efficiently (Garcez & Lamb, 2023). Within such a neu- rosymbolic framework, KGs can naturally serve as symbolic memory and rule repositories. Coupled with neural networks, they provide modularity and trossseepresnntation translation. By decoupling learning (im- plicit, neural) from reasoning (explicit, symbolic), KGs address interpretability, verifiability, and factuality gaps in modern AI systems that dominated by neural approaches (Besold et al., 2017).
A.2 KG Applications in Neurosymbolic Frameworks
KGs with high-quality expert knowledge enable a range of downstream applications.
Explicit reasoning and knowledge transfer: Symbolic inference KGs is interpretable (Cheng et al., 2024), and explicit rules can also be transferred across related domains. For instance, retrieving all drugs (d) that (1) target protein (p) associated with Type 2 Diabetes and (2) are approved by the FDA, expressed as the conjunctive query:
3 p. d bassociated_withe Type_ 2_ Diabetes) ^ targets(d.p) A approved pyprooed_by(d,FFal (D,))
(12)
Discoverics: KGs uncover hidden insights by linking apparently unconnected concepts through explicit graph paths. This connectivity underpins applications, notably recommendations, where relevant entities are suggested based on graph neighborhoods, and analogical discovery, where cross-domain parallels emerge through reasoning over paths. In biomedicine, for example, path like drug - protein target pathway disease can reveal repurposing opportunities, e.g., a hypertension drug acting pathways implicated in diabetes.
KGs for Domain-specific Superintelligence: A natural scarcity of high-quality text sources may im- pede the scalability of ever-larger LLMs, necessitating a paradigm shift that favors smaller language models with domain-specific expertisc. This would shift focus from general-purport to domain-specific superintelligence (Bostrom, 2014; Szocik al., 2020). A recent study by Dedhia et al. (2025) illustrates one path toward this goal by fine-tuning small language model multi-hop KG paths
artificial general intelligence
48

---

Published in Transactions on Machine Learning Research (02/2026)
translated into composite natural language statements. They show that longer KG paths yield stronger domain reasoning. From this perspective, conventional text datasets resemble one-hop knowledge that of- ten leads to surface-level understanding. In this context, a high-quality, domain-specifie KG that explicitly encodes knowledge through concise and unambiguous semantic relationships can provide a useful inductive bias Loward deeper reasoning.
B Extended Examples of LLM responses
Claude Sonnet 4.5: Based on the provided sequence, the medical KG triple should be completed as: (pediatric chronic kidney disease (pckd), has fons__ininng___ site, kidney)
Reasoning:
The primary and defining finding site for chronic kidney disease is the kidney itself, explicitly stated in the passage: "Chronic kidney disease (CKD) is a renal disorder." While the passage does discuss secondary neuroimaging findings in the cerebellar gray matter, these are described
Subtle and group-level abnormalities
Heterogeneous observations
Not diagnostic
Modest and regional alterations
contrast, the kidney is the primary pathological site where CKD occurs by definition. The cerebellar changes represent associated findings or potential complications, but they are not the primary finding site the disease entity itself. For medical knowledge graphs, the has finding site relation typically indicates the primary anatom- ical location where the disease process occurs, which for CKD is definitively the kiduey. If you needed to represent the cerebellar findings, that would be better captured with different relation such as has associated finding may affect, rather than has finding site.
GPT-5: Based on the medical knowledge graph triple format and the provided sequence, the completed triple would be: (chronic kidney disease, has sos_ffindngg... _stee,itteeeeee cerebellar gray matter)
This based on the text where chronic kidney disease (CKD) is associated with observable findings in the cerebellar gray matter, specifically mentioning reductions in cerebellar gray-matter volume, cortical thickness, and diffusion metrics.
Grok 4: Based the medical knowledge graph triple format and the provided sequence, the completed triple would be: (chronic kidney disease, has finding_sile sile, cerebellar gray matter)
The triple (chronic kidney disease, has finding_siit site, cerebellar gray matter) is completed based on the provided sequence, which states that neuroimaging studies of chronic kidney disease (CKD) patients have identified subtle abnormalities in the cerebellar gray matter, such as reductions in volume, cortical thickness, and diffusion metrics. site" indicates the anatomical location where these abnormalities are observed, making
The relation "has "cerebellar gray matter" the appropriate tail entity.
C Training Data
We exclude some relations from the UMLS KG that add little semantic value, i.e.: (i) custom-defined mappings of outdated--oonew UMLS relations for backward compatibility, which cannot be inferred from external data; (ii) relations used only for cross-vocabulary mappings; (iii) relations where tails are largely redundant with respect to heads; and (iv) relations with very few possible tails. For example, relation has associated finding is redundant relation:
family history of diabetes, has has associated finding, diabetes mellitus
49

---

Published in Transactions on Machine Learning Research (02/2026)
Table Cl: Excluded UMLS relations
acted on by process active srrivee_ingreient of associated procedure of
has scale type
regulated by replaced by replaces was has intent referred__ to by refers to characterizes
has has__pecimenn
has subject relationship context has temporal_mconeee context
basis basis_of_ of_strengthh substance of component of consider_frooe cerrt_ddeie_____ of direct substance of has associated finding has finding context has interpretation
inverse was mapped from mapped to moved to negatively regulated by sattivelly_rgulattee by possibly_replasn
substance used by specimen_souree torpopggaprre of of has active ingredient has property
finten__oonextt
has laterality has realization
precise active ingredient of realization_oo of
family history of glaucoma, has associated finodiiiatddffdiin..... glaucoma
parental history of diabetes, has has_ associated fasooiiated_findingg diabetes mellitus
family history of hypertension, has associated_ finding, hypertensive disorder
family history of evd, has associated lraociateddfiidinng congenital heart disease
In these cases, the tail subject. is the same as the head subject. has laterality is an example of a relation with very few possible tails, with almost all tails being "side".
diabetes [Abstract] OR diabetes [Body - All Words] OR diabetes [Body Key Terms]
OR (
''diabetes mellitus' . [MeSH Terms] OR >> ddiabtees insipidus [MeSH Terms]
OR diabetes [Title]
NOT
s'aar -ov > [MeSH Terms] DR covid > [MeSH Terms]
AND (
medline [ab] AND 22019/04/01 [PubDate]
/2025/04/01' [PubDate]
Listing 1: PubMed search query
50

---

Published in Transactions on Machine Learning Research (02/2026)
D KG Injection Algorithm
Input:
Sequences with heads: Each head h may have multiple triples T (h, T(seg) in sequence seg (we cap this number at 40 triples to ensure enough diversity and leaving room for further dropping); triple embedding similarity score with respect to the sequence: score(T); similarity matching threshold a hyperparameter).
Output: Each head has one injection T(seg) or does not have any, in all sequences.
Preprocessing
Drop all triples with a score less than threshold a.
2. Make all triples unique: If a triple matches multiple sequences, retain the triple T with the highest score, i.e., in the sequence to which the triple is most relevant:
= arg max score(T(seg)) seq
The second preprocessing step prevents overfitting in the semantic space on common triples.
Triple selection for each head: To balance contextual relevance with relation diversity: 1st priority: maximize injection score, 2nd priority: maintain relation diversity. Relation diversity is measured by the number of unique triples that contain the relation.
Maximize diversity
1. Split relations into relation buckets based on the number of unique triples step le and assume that within each bucket all relations are equally diverse (e.g., h = 20 implies that relations with #triples 100-120, 120-140.., are treated as equally diverse).
Within each relation bucket, sort all triples by score regardless of relation.
3. Start with the lowest-mmbered bucket (rarest relations). Within it, start with the triple with the highest score and retain only for its head, removing other matched triples, which may have a higher score but may be higher relation bucket. As a result, one of the rarest possible relations in the dataset would survive for this head, increasing relation diversity overall.
Maximize score then diversity
1. Order triples by score.
Split into score buckets: Assume that within each score bucket, triples are equally good.
3. Then, within each score bucket, apply Maximize diversity.
Altogether, group triples by how "low" the score is (higher scores are assigned to lower-bucket IDs). Then, within each score bucket, we favor relation types that are less frequent. Finally, we choose the highest-scoring triple for each head.
The algorithm is implemented using the Pandas framework and presented in Algorithm 1.
our experiments, we use score bucket size = 0.01, relation bucket size 100. The bucket size and relation bucket size are chosen under the assumption that, within each bucket, score and relation diversity remain approximately balanced.
51

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm 1: Maximize Score then Diversity
Input : df (a Pandas DataFrame where each row comprises a triple (h,r,t) unique matched head id, and an associated score), score bucket size, relation bucket size Output: Filtered DataFrame result
max max(df.score): foreach row in df do [maoz_ row.score)/scoree saarre_bucke__taaee size]
rro..score_bucket
rel counts count occurrences of each relation lype r in df;
foreach row in df do rel_count &- rel _counts(sroowr]
row.rel bucket + [rel fount/relation bucket size];
Sort df by ascending bucket, rel ru__ucket) and by descending score;
resull. + []:
seen scen_ hean_heass-- + 0;
foreach row sorted df do if row.matched head id & seen_heade then append to result;
seen heads +- seen heads U (row. head id);
// Equivalent to df wwt.drop_duppe dt...rop.applaaccccccsssttaaaaaa return result;
_d", keep="first")
Table D1: Seed KG: Relation statistics for 0.55 in the training split (Qwen3-32B)
# Relation
injections
# Relation
injections
8627
15
equivalent possibly_equivvlentt to
446
2
inverse isa cause
5512
16
has component due to
433
3
1440
17
365
4 interprets
1268
18 has part
350
5 ussociated finding_or of
1145
19 has modification
310
6
has disposition focus of
1084
20 associated with
254
7
1038
21 part_ of
211
8 is intterpreted by
962
22 plays_rol role
194
9 laa__associated morphology
863
23 occurs before
187
10 causative agent_ of
24 has clinical course
144
11 finding_stt site_oo of
741
25 occurs in
138
12 associated morphology_coo of
26 same as
134
13 has _e_thod
515
27 has causative agent
127
14 has finding site
477
28 has focus
118
52

---

Published in Transactions on Machine Learning Research (02/2026)
Comparison of Relation Frequencies (Logarithmic Scale)
Extracted KG.
10
65
Number Oocurrencas (Log Scale)
10
10"
10
Ausociated Bnding. d
asosdaled marphalopy.com of
ansociatord with
causaair rigerg of
Radt ansociated marphalogy
has canative Agent
hare ciiiccal course
hap componentt
hee dipoatiion
Haw focus
hase methor
has modifiation
interprets
invosse, Isa
interpreted Iyy
dbcars betfore
isa
accum Jn
presitly aquivalent.. to Gama .r
cauree
die
locul
Haal finding
nare
part
plays.
frading
Figure El: Relation distribution in the GRAPHMERT-ex KG seed KG. The shapes differ: While "isa" " prevails in the seed KG, the GRAPHMERT KG is heavily skewed towards "associated with." This reflects inclination of the helper LLM to select "associated with" during relation matching the most appropriate, given sequence.
E Extracted KGs
E.1
Relation Distribution: GraphMERT-extracted KG vs. Seed KG
Fig. El shows the relation distribution on a logarithmic scale. While "isa" the most represented relation in the training data, in the to-be-extracted KG, the helper LLM tends to select "associated with" most frequently during relation matching.
E.2
Sanity Check
We ran lightweight. screening with GPT-5 Thinking on small, comparable samples from each KG. This screening should be viewed as complementary to benchmark-barl based verification, providing useful diagnostic signals but not replacing factual evaluation.
Setup: For each KG, we retrieved all triples whose head contains the keywords "insulin-like growth factor (IGF-1)" and "glucocorticoid receptor (GR)." These terms are highly relevant diabetes, yield comparable sample sizes per KG (22 29), and remain small enough for human inspection.
Prompt. Evaluate if these medical KG triples valid (yes/no/maybe) and give a very short reason why: (list of triples).
53

---

Published in Transactions on Machine Learning Research (02/2026)
Table E1: Example of GRAPIMEXTT-sxtratted triple with novel tail vocabulary from a 128-token sequence. The seed KG does not include the token "nirp3" - was learned and extracted from the text. "Pathway" is a token that GRAPHMERT learned from the training corpus. Here, "pathway" is implicitly supported: The phrase "specific inhibitors of NLRP3 inflammasome activation" links the activation to NeRPe-mediaeed signaling (i.e,, the NLRP3 pathway).
Sequence: inflammasome activation and regulation is highlighted, including its putative roles in adipose tissue dysfunction and insulin resistance. Specific inhibitors NLRP3 inflammasome activation which can potentially be used treat melabolic disorders also discussed. Identifying quanlitative biomarker of neuropsychiatric dyslunction in people with HIV (PWH) remains significant challenge in the neuroHIV field. The strongest evidence to date implicates the role of monocytes examined monocyte subsets in blood as a correlate and /or predictor of neuropsychiatric function in virally suppressed PWH. METHODS independent cohorts of virologically suppressed women with HIV (vsWWH...
central nervous system (CNS) dysfumction in HIV, yet no study has
Head
Relation
Tail
inflammasome activation
associated with
nirp3 pathway
Table E2: GPT-5 Thinking screening (counts with proportions)
Keyword
KG
Yes
Maybe
No
Total
IGF 1
LLM
8 (0.28)
3 (0.10)
18 (0.62)
29
IGF 1
GRAPIMERT
15 (0.68)
0 (0.00)
7 (0.32)
22
GR
LLM
4 (0.17)
12 (0.50)
8 (0.33)
24
GR
GRAPIMERT
15 (0.54)
(0.25)
(0.21)
Table E2 summarizes screening results. Tables E3, E4, E5, E6 provide GPT-5 Thinking responses for cach triple in the GRAPHMERT and LLM KG samples.
54

---

maybe Published in Transactions on Machine Learning Research (02/2026)
Table E3: GRAPIMERT KG. Sanity check with GPT-5 Thinking. We select all KG triples that contain "insulin-like growth factor-1," 22 triples in total, yes yes/no htt///n/maay maybe: 15/0/7.
Triple
Verdict GPT-5 explanation
insulin-like growth factor 1, isa, peptide hormone
yes
70-aa peptide hormone.
insulin-like growth factor 1, isa, growth factor
yes
Canonical growth factor.
insulin-like growth factor 1, associated with, insulin
yes
Correlates with IR.
resistance
insulin-like growth factor 1 receptor (IGFIR), plays_rol role, target
"Target" is vague.
insulin-like growth factor 1 (IGF-1), associated with,
yes Linked with DM.
diabetes mellitus
insulin-like growth factor 1 (IGF-1), associated_ with, maybe Biomarker is role.
biomarker
insulin-like growth factor 1 (IGF-1), associated_ with, insulin
Strong crosstalk.
insulin-like growth factor 1 (IGF-1), associated__itt type
yes
Epidemiologic link.
diabetes
insulin-like growth factor (IGF-1), associated with,
yes
Well supported.
insulin resistance
insulin-like growth factor 1, plays_roll growth factor
yes
Acts growth factor.
insulin-like growth factor 1 (IGF-1), sooyss_rreeee_____ biomarkers insulin/insulin-like growth factor 1 (IGF-1) signaling pathway, has component, IGF-1 receptor
maybe
Shural/ontolog mismatch.
yes
IGF-1R in pathway.
insulin-like growth factor 1 (IGF-1), associated_ _ith insulin
yes
Duplicate; supported.
insulin-like growth factor 1, plays role, growth hormone action
yes
Mediates GH effects.
insulin-like growth factor 1 receptor (IGFIR), plays role, signaling receptor
yes RTK signaling receptor.
insulin-like growth factor 1 receptor, plays role, signaling insulin-like growth factor 1 (IGF1), associated with, androgen
yes
Mediates signaling. Context-specific crosstalk.
maybe
insulin-like growth factor 1 (IGF1), associated with, transcription
maybe Indirect gene-expression effects,
insulin-like growth factor 1 (IGF-1),, associated with, hyperglycemia
maybe Context dependent.
insulin-like growth factor 1 level, associated with, growth yes
GH raises IGF-1.
hormone treatment
insulin/innsulnn-iike growth factor (IGF-1) signaling pathway, has _cmpoonen, insulin receptor
yes
pathway.
insulin-like growth factor 1 (IGF1), plays_rolc, downstream target
maybe Phrasing vague (GH downstream).
55

---

Published in Transactions on Machine Learning Research (02/2026)
Table E4: (Part 1 of 2). LLM KG (Qwen3-32B). Sanity check with GPT-5 Thinking. We select all KG triples that contain "insulin-like growth factor-1," 29 triples in total, pos/no/mayyll 8/18/3.
Triple
Verdict GPT-5 explanation
insulin-like growth factor 1, associated with, diabetes
yes
Linked with DM.
insulin-like growth factor 1 (IGF-1), has direct. substance, no Disease & substance.
type 2 diabetes
insulin-like growth factor 1, bobociated_witbee insulin resistance
Strong physiologic link.
insulin-like growth factor 1, associated_ metabolic
yes
Levels track MetS.
syndrome
insulin-like growth factor 1 (IGF-1), ussociated__iie
yes
GH/IGF axis altered.
chronic kidney disease
insulin-like growth factor 1 (IGF-1), has direct substance, n0 gestational diabetes mellitus (GDM)
Disease * substance.
insulin- growth factor 1 (IGF1), has direct subvee__ubttanee, angiogenesis
no
Process, not substance.
insulin-like growth factor 1 (IGF-1), has direct substance, no short-chain fatty acids (SCFAs)
Unrelated metabolites.
insulin-like growth factor 1 (IGF-1), has component, insulin signaling pathway
no
Pathway # component of ligand.
insulin-like growth factor 1 (IGF-1), plays role,
maybe
Broad/commexx-deeeeddel
cardiovascular health
insulin-like growth factor 1 (IGF-1), associated_ with, bone ycs metabolism
Anabolic for bonc.
insulin-like growth factor 1, has part, growth hormone no
GH regulates; not part. Association A causation.
insulin-like growth factor 1 (IGF-1),, cause of, prostate cancer (PCA)
no
insulin-like growth factor 1 (IGF-1), has modification, left no ventricular global longitudinal strain (LVGLS) insulin-like growth factor 1 receptor, cause of, opitheliaa-mmesenchyaall transition
Clinical metric, not modification.
maybe
Signaling can induce EMT.
insulin-like growth factor (TGF-1), has modification, HAAT-MSCs
no
Not molecular modification.
insulin- like growth factor 1, has direct substance, testis insulin-like growth factor 1, plays role, endocrine-related cancers
Organ produces IGF-1.
maybe
Vague class-level claim.
56

---

Published in Transactions on Machine Learning Research (02/2026)
Table E4: (Part 2 of 2). LLM KG (Qwen3-32B). (continued)
Triple
Verdict GPT-5 explanation
insulin-like growth factor 1, associated__ oocyte cohort quality
yes
Follicular IGF-1 correlates.
insulin-like growth factor 1 (IGF1), has component,
rean effector, not component.
phosphoinositide 3-kinase (PI3K)
insulin-like growth factor 1 receptor, isa, receptor
yes
Canonical RTK.
insulin-like growth factor 1 receptor, has direct srreet__ubsttanee lactate dehydrogenase a.
no
Not a receptor substance.
insulin-like growth factor 1 (IGF1),
no
Not a pathology.
has pathological process, core body temperature insulin-like growth factor 1 (IGF1), has _nspponnnn IGF-binding protein 3 (IGFBP3)
n1
Binding partner, not component.
insulin-like growth factor 1 receptor, has method, JAKS2/STAT3 pathway
no
Wrong relation/alypies pathway.
insulin-like growth factor 1 (IGF1), has _as_conpneet,,
no
Unrelated protein.
ZYG11A
insulin-like growth factor receptor (IGF-1R)) has part, nositide 3-kinase (PI3K)
no
PI3K recruited, not part.
IGF-1 (insulin-like growth factor 1), has direct substance, no
transcription factor e substance of IGF-1.
PPARt (peroxisome proliferator-activaeeed receptor gamma) insulin-like growth factor 1, plays_role, role, cardiac
yes
Supports cardiac growth.
development
57

---

Published in Transactions on Machine Learning Research (02/2026)
Table E5: GRAPHMERT KG. Sanity check with GPT-5 Thinking. We select all KG triples that contain "glucocorticoid receptor," " 28 triples in total, pes/no/magul /maybe: 15/6/7.
Triple
Verdict GPT-5 explanation
endothelial glucocorticoid receptor, plays role, therapeutic role
maybe
Role is
aggue//ontext-eeeeddee..
endothelial glucocorticoid receptor, plays_roie
yes
Mediates GC signaling in
glucocorticoid signaling
endothelium.
endothelial glucocorticoid receptor, plays_role, protective role
maybe
Protective effects reported but not universal.
glucocorticoid receptor (GR), plays_role, glucocorticoid signaling
yes
Canonical function.
glucocorticoid receptor, associated with, insulin signaling glucocorticoid receptor, has part, ligand-binding
yes
Well-known pathway crosstalk.
maybe
Should be "ligand-binding domain."
endothelial glucocorticoid receptor, has _is_disosttionn,
no
Wrong relation; participates in signaling.
signaling
glucocorticoid receptor, plays_role, glucocorticoid receptor
yes
Tautologically true.
signaling
glucocorticoid receptor (GR), plays_role, hypothalamic signaling
maybe
HPA feedback involvement but broad.
endothelial glucocorticoid receptor, has disposition,
no
"Immunooduuaator" is agent
immuomodulator
role, not disposition.
glucocorticoid receptor (GR), plays_roke steroid signaling
yes
Nuclear steroid receptor pathway.
glucocorticoid receptor (GR), associated with, glucocorticoids
yes
Binds GC ligands.
glucocorticoid receptor agonists, plays_role, role, therapeutic
yes
Used clinically.
glucocorticoid receptor (GR), plays_role, stcroid hormone receptor
This is an is identity, not role.
glucocorticoid receptor gene (NR3C1), associated_ with, hormone recoptor
maybe
Gene encodes one; relation imprecise.
glucocorticoid receptor, plays role, signaling
yes
Ligand-activaled signaling/TP activity.
glucocorticoid receptor (GR), associated with, steroid
yes
Glucocortiooids are steroids.
selective glucocorticoid receptor agonists, plays_rola therapeutic role
maybe
Class intended for therapy; phrasing vague.
endothelial glucocorticoid receptor, cause_off, of, renal fibrosis
no
GR generally anti-fibrotic; causation unsupported.
selective glucocorticoid receptor modulators, role, pharmacological modulator
no
Category is a, not a role.
glucocorticoid receptor, associated with, glucocorticoids
yes Duplicate; correct.
glucocorticoid receptor (GR), plays role, glucocorticoid signaling
yes
Duplicate; correct.
glucocorticoid receptor a, plays_role, stcroid receptor
no isoform is steroid receptor, not a role.
glucocorticoid receptor (GR), associated_ with,
yes
Ligand-activated transcription factor.
transcription
glucocorticoid receptor, plays_ role, signaling glucocorticoid receptor (GR), associated with, hormone
yes
Duplicate; correct.
yes
Responds to hormones (GCs).
glucocorticoid receptor (GR), associated with, signaling
yes
Broad but true.
selective glucocorticoid receptor modulators, plays_rols pharmacological modulator
no
Duplicate; is a, not a role.
58

---

Published in Transactions on Machine Learning Research (02/2026)
Table E6: LLM KG (Qwen3-32B). Sanity check with GPT-5 Thinking. We select all KG triples that contain "glucocorticoid receptor," " 24 triples in total, yes/no/maylle 4/18/2.
Triple
Verdict GPT-5 explanation
glucocorticoid receptor haplonssufiiienccy cause of, hypertension
yes
GR resistance drives
mineralocorticoid excess
hypertension.
glucocorticoid receptor, associated with, insulin resistance
yes
GC/GR signaling induces IR.
glucocorticoid receptor locus (GRL) polymorphisms, associated_wiihe type diabetes (T2D)
yes
NR3CI variants linked to T2D risk.
podocyte-specific glucocorticoid receptor knockout
yes
Podocyte GR loss worsens DN phenotype.
(GR pKO) mice, has pathological process, diabetic
nephropathy
glucocorticoid receptor, has compponntt autophagy
Autophagy is process, not
receptor component.
no glucocorticoid receptor (GR), finding site of, liver
Relation reversed; GR is located in liver.
glucocorticoid receptor, has component, diabetic complications
no Diseases aren't receptor components.
endothelial glucocorticoid receptor (GR), has causative _agtat, renal fibrosis glucocorticoid receptor, has has_ disposition, cortisol
Relation misuse; fibrosis doesn't "cause" the receptor.
no
Cortisol is ligand, not
disposition.
glucocorticoid receptor alpha, jia__componntt
no
COVID-19 isn't. a component.
coronavirus disease 2019
glucocorticoid receptor, has toss__comooettt, estrogen endothelial glucocorticoid receptor (GR), has pathhllogccall proccssssspppoossss WNT signaling
no
Hormone * receptor component.
WNT signaling isn't inherently pathological.
glucocorticoid receptor, has pathological process, osteoporosis (OP)
maybe
Excess GR signaling leads to GC-induced OP: relation loose.
glucocorticoid receptor, has direct_ serrectssubsancee PEPCK
no
GR regulates PEPCK expression; not a substance of GR.
glucocorticoid receptor (GR), has direct srrect__sbstancee
no
Cells aren't receptor substances.
Kupffer cells
glucocorticoid receptor (GR), has _ooppoenmn Leydig cells
Tissues/ccels aren't components of receptor.
glucocorticoid receptor, has diireet_ruussettttt __tutttnuubtt,,,, stress no response
Process * substance.
glucocorticoid receptor, associated_ with, miR-32-5p
maybe
Limited, context-specific miRNA linkage.
glucocorticoid receptor, has direct substance, G6P
no Metabolite isn't sn'' receptor
substance.
glucocorticoid receptor, has component, SETDIA/COMPASS complex
Possible cofactor interaction, not component.
glucocorticoid receptor (GR), has direct substance, STAT6
no
TF interactor, not a substance of GR.
glucocorticoid receptor (GR), has direct substance, adipose tissue macrophage (ATM)
no
Cells + substances.
glucocorticoid receptor (GR), has component,
Neurons aren't receptor components.
peripheral sensory neurons
glucocorticoid receptor 3 (GR3),
no
SNP pertains to gene, not isoform "substance."
has Satrect__uubtaaett rs948820149
59

---

Published in Transactions on Machine Learning Research (02/2026)
F
GraphRAG evaluation on Medical Benchmarks
Table Fl: GraphRAG KG evaluation on public benchmarks
MedMCQA
MedQA
MMLU (medical)
# Questions
61
75
62
LLM KG (baseline)
72.1
85.3
71.0
Seed KG
76.5
81.3
73.1
GRAPHMERT
73.8
88.0
74.7
In this evaluation (Table F1), we select questions related to diabetes and its comorbidities from popular medical benchmarks and run GraphRAG evaluation on the selected questions. We first filter the benchmarks with the Qwen3-32B model, as stated in Section 5.3.2, and then manually review and remove some questions that are irrelevant to diabetes. The full list of selected questions can be downloaded from GitHub.
60

---

Published in Transactions on Machine Learning Research (02/2026)
G
Helper LLM prompts
We include only one example for few-shot settings. For each prompt, examples are mined from the dataset. Then, we prompt GPT-o3 with the zero-shot prompt, edit the reply if needed, and include it as a few-shot example.
All sequences in our datasets are lower-case; here, we use normal case to improve readability.
Entity Discovery Prompt
You are a medical-domain extractor building a diabetes KG of (head, relation, tail). You possess advanced medical academic knowledge.
Given input sequence, identify entities specifically relevant to diabetes, its complications, comorbidities, ther- apeutics, and related biomedical entities that help clarify or contextualize them. Output a Python list of up 6-word entity "heads" following these rules:
1. Select precise and medically-specffie span (e.g., "myocardial infarction," " "infarction"). Avoid gemeric terms like "disease," "condition," " "patients," and "comorbidity" without specific context. When encountering vague descriptors like "complication," " "symptom," " "effect," " always prefer explicitly named conditions or symptoms directly linked to diabetes pathology or diabetes comorbidities.
2. Keep original spelling, casing, and abbreviations from the sequence.
3. Choose only entities that add meaningful medical knowledge related terms. Do not include head entities that describe findings in animal models (mice, rats, etc.).
diabetes Do not include COVID-
4. A few examples of low-value entities you should not include:
2 10 % weight reduction' (too context-dependent).
nhanes 2015 2018* (dataset/surrvy, not medical entity).
'semaglutide 2.4 mg' (includes dosage, which vary).
'60+ women' ("60+" is too context-dependent).
"anxiety." "home births," "pregnant women," "neonatal deaths," "general practitioners" (not spe- cific enough to diabetes; only include if explicitly related to diabetes).
5. If it is not clear whether a term adds diabetes-spedd specific knowledge, look at the context. If the text explicitly links the term to diabetes-specific concept, include Otherwise, exclude when mentioned only in generic context. Include such terms when the sequence clearly links them to a diabetes-relevant gene, pathway, cell type, or therapeutic effect.
You will be provided with incorrect output examples beginning with "Outpul (Incorrect)." Use them avoid the common mistakes. Wrap your intermediate reasoning steps clearly within (think) ... (/think) tags. Be strict and discard any entity about which you uncertain and that is not relevant to diabetes. After generating, verify your output.
Steps:
1. Identify candidate spans.
2. Filter by medical precision and relevance rules.
3. Confirm the entity's relevance and contribution to diabetes KG; discard low-value entities.
Input format: sequence
Output format: ['head1', 'head2', ...J. If none, output []
61

---

Published in Transactions on Machine Learning Research (02/2026)
Few-shot Example for Entity Discovery Prompt
Input:
sequence: its upstream regulator has the opposite effect (Han et al., 2013). Previous studies suggest that CHOP deteriorates ER stress and accelerates cell death via promoting protein synthesis and oxidative stress (Han et al., 2013). In addition, ER stress damages 3-cells, possibly through altering Ca*+ homeostasis. It has been indicated that ER stress interferes with the function of RyR located in the membrane of the ER and causes leakage of ER Ca (Yamamoto et al., 2019). The destruction of 3-cell ER Ca homeostasis results impaired insulin secretion and further promotion of 3-cell death
Output:
[ c'hop,,"" "eer stress,"' " "S\betaB-celle,, 1 (impairred insulin aecretion']]
The rationale behind the output:
Candidates: CHOP; ER stress; 8-cells; RyR; impaired insulin secretion; oxidative stress; protein synthesis. Keep precise diabetes-relevant entities: CHOP (ER stress regulator), ER stress, 8-cells, RyR, impaired insulin secretion.
Relation Matching Prompt
You are medical intelligence with academic knowledge in diabetes and comorbidities. knowledge graph of triples (head, relation, tail).
We building diabetes
Given:
sequence including biomedical contexl,
list of heads,
return, for each head, all relations chosen from the list below that could form plausible KG triple and supported by the sequence.
Allowed relations:
###
associated finding of associated morphology of associated with causative agent of cause of due to finding_siin of of focus of
has associated morphology has component
inverse is interpreted by isa occurs before part of plays_role role possibly_equivvaale equivalent Lo
has disposition
has finding site
has method has modification
has has_part interprets
The relations are taken from UMLS have the same meaning in UMLS. Examples:
carotid artery stenosis
associated finding of history of carotid artery stenosis endomyocardial fibrosis
fibrosis
associated pasociated_mmopphokees
cancer
associated _ith anemia malignant neoplastic disease
aaassttve_aaent..... Mycobacterium tuberculosis cansative agent of Tuberculosis
diabetes mellitus cause_ of diabetic foot
hypoglycemic alcoholic ketoacidosis due_to to acute alcohol intoxication adipose tissue finding site_of of lipoatrophy
renal failure focus of emergency hemofiltration
62

---

Published in Transactions on Machine Learning Research (02/2026)
hepatitis A has associated moophooogg
Hepatocellular necrosis
fasting triglyceride mus__compnent
trracylglycerol
tumor necrosis factor "aa__dsspssitonn immumoodulator
melanoma has finding_sit site skin
bariatric surgery has method surgical action
glucagon has modification glucagon hydrochloride
nephron has__art glomerulus
overweight interprets body weight measure
adiponectin inverse_ isa high molecular weight adiponectin
blood eosinophil counts is _ntterrreted
asthmatic pulmonary eosinophilia
empagliflozin isa sodium glucose colransporter subtype 2 inhibitor
cardiac amyloidosis
occurs in old age
coronary syndrome
possibly_ equivalent to preinfarction syndrome
MT same
Myocardial infarction
Note the meaning of relations in UMLS: isa and inwerse isa exact inverses of each other.
isa points up the hierarchy: "Diabetic retinopathy" "Retinal disease." (specific general) inverse isa - points down the hierarchy: "Retinal disease" inverse isa "Diabetic retinopathy." cause_ of directional link where the source concept understood to directly or indirectly produce, trigger, give rise to the target concept.
due to - causal link: the subject condition, finding, or situation results from the object. Inverse: oa of.
associated with uon-directional link indicating that concepts statistically or clinically linked without asserting a clear cause-and-effed direction.
aas_associated morpholoorroyyyyoo - links pathological or clinical entity (typically disease, syndrome, injury) to the characteristic structural change ("morphology") it produces. Concretely: source dis- order concept; target- morphologic abnormality (e.g,, "Necrosis," "Hyperplasia," "Fibrosis"). Inverse: associated morphology_og of.
associated -eocciate_fiiddngg of reads as: for which procedure Y is performed."
"X associated finding of Y "Finding X is the clinical finding
Input format:
aequence
heads: [head1, head2, ...]
Output format:
{
''ead 1 [''relation 1, "relation 2, ...l,
<'eaad 2 [...],
}
Steps:
1. Understand Input
Clearly understand the biomedical context from the sequence.
each head, find explicit mentions the text.
Check if each head is explicitly linked to other concepts relations.
2. Use the list of allowed relations. Evaluate each head individually. associated, with apply it only when appropriate.
Do not overuse the relation
3. For each head, list only plausible and supported relations. Return if none apply.
Think concisely within (think) ... (/think). Immediately after, output JSON.
63

---

Published in Transactions on Machine Learning Research (02/2026)
Few-shol Example for the Relation Matching Prompl
Input:
interleukin-1 R6, and receptor activator of nuclear factor kappa-B (RANK). Together, proteomic data suggest the targeting of several key regulators of inflammation, bone, and adipose turnover, via transforming growth factor-beta/SMin /SMD, and Wingless-related integration site/be-catenin signaling pathways. To the best of the knowledge, this is first evidence of intervention that drives against bone loss via RANK. Metatranscriptomic analyses of the gut microbiota show P7C3 increased Porphyromonadaceae bacterium, Candidatus Melainabacteria, and Ruminococcaceae bacterium abundance, potentially contributing the favorable inflammatory...
heads: ['interleukin-1 r6,' 'receptor activator of nuclear factor kappa-b,' 'transforming growth factor-beta']
Oulput:
{
"interleukin-1 r [ cessociaate_iithhh"
"receptor activator of nuclear factor kappa-b'?; "ccuse_oo''] "transforming growth factor-beta", [ [''part_of']]
}
TThe rationale behind this output:
interleukin-1 associated, with Named "key regulator of inflammation," which links it the inflammatory process without stating direction or hierarchy, so the non-causal associated with relation fits best.
receptor activator of nuclear factor x-B (RANK) + cause_of of + The text says the intervention prevents bone loss via RANK, implying that RANK signalling produces drives bone loss; therefore cause_of appropriate.
transforming growth factor-beta - part_ of - Explicitly mentioned within the "TGF-$/Smad signalling pathway." it is constituent component (part_of) of that pathway.
No additional relations are warranted.
Combining GRAPHMERT-preditee Top Tokens Prompt
You completing triples for medical knowledge graph diabetes and its comorbidities. For each sample, you're 're given sequence, head entity in that sequence, a relation, and a list of candidate tokens. The relations from UMLS and have the meaning.
Your
task is to output filtered list of high-quality and factual tails in the format: t''aail 1, "ttail 2," ...] or O.
To form the list of candidates:
Step Analyze the sequence to understand the context and identify the head entity and relation.
Step 2: Choose candidate tails. You can combine tokens from the candidate list to get the most precise, relevant, and meaningful tails in the context head and relation. Combine subword tokens, too. Step 3: Verify each candidate.
Verification. Each must:
Be causally and factually related the head via the specified relation. Make the relation direction is correct: the head implies the tail given the relation. Note that isa is subclass-rclas relation, and inverse_ isa class class-ssubclass relation.
64

---

Published in Transactions on Machine Learning Research (02/2026)
Be supported by the sequence, but you can rely on well-established medical knowledge even if the sequence does not spell it out verbatim. If no reliable support exists, reject the tail.
Add valuable medical knowledge to the graph. Tails must be uon-redundant. When all tails are factual, prefer specific tails over general and vague (e.g., "proliferative diabetic retinopathy" over "retinopathy") Terms that include "level," "disease," "disorder," or "complication" are too vague and rarely add useful knowledge the KG.
Include only lokens from the list of candidates.
Reason step by step within <think>..<<thiil
You will see incorrect outputs labeled "Output (Incorrect)." Avoid similar errors.
Before finalizing:
Ensure all output constraints are met.
Validate that each tail is logically, contextually, and factually aligned with the head and relation. Confirm that each triple adds meaningful knowledge to the graph.
Note the meaning of some UMLS relations you may encounter in the input:
isa and inverse isu are exact inverses of each other.
isa - points up the hierarchy: "Diabetic retinopathy" isa "Retinal disease." (specific general) inverse_ise isa points down the hierarchy: "Retinal disease" inverse_ isa "Diabetic retinopathy." cause of directional link where the source concept directly indirectly produces, triggers, gives rise to the target concept.
due_le to causal link: the subject condition, finding, or situation results from the object. Inverse: cause_ of.
associated with non-directional link indicating that two concepts are statistically clinically linked without asserting a clear cause effect direction.
has_assaaaae associated aosociated_mrpphologgg links a pathological or clinical entity (lypically a disease, syndrome, or injury) to the characteristic structural change ("morphology") it produces.
Concretely: source = a disorder concept; target Morphologic Abnormality concept (e.g., "Necrosis," "Hyperplasia," "Fibrosis"). Inverse: associated mocpotdd___mrphogggy___
associated finding of reads as: "X associated finding_or of Y "Finding the clinical finding for which procedure performed."
Few-Shot Example for Combining GRAPH Top Tokens Prompt
Input:
sequence: ...2+ binding to S100A1 EF-hand motifs, conformation S100A1 changes and promotes interactions with target proteins. RAGE consists of three domains: the cytoplasmic, transmembrane, and extracellular domains. The extracellular domain consists C1, C2, and domains. domains are the primary receptors for the $100 protein. It was reported several years ago that S100A1 and RAGE V domains interact a pathway involving $100A1-RACE signaling, whereby S100A1 binds to domain, resulting in RAGE dimerization. The autophosphooylltiin of the cytoplasmic domain initiates signaling cascade that regulates cell proliferation, cell growth, and tumor formation. In this study...
head: ['1000'''''
relation: associated__witt
predictions: protein receptor hydrolase structure process pathway factor complex glycoprotein s100a family domain oxidoreductase proteinase ligand extracellular signaling calcium apolipoprotein s100
Output:
65

---

Published in Transactions on Machine Learning Research (02/2026)
caaacii signaling pathway' ']
The rationale behind this output:
Available tokens let us build terms such as:
calcium signaling pathway" (calcium + signaling + pathway)
extraaeellular signaling" (extracellular + signaling) too broad
"s100a family" classification, association
Other tokens ("glycoprotein," " "oxidoreductase," etc.) do not represent well-known processes linked to S100A1.
Preference & verification:
"S100A1 modulates intracellular Ca handling and is vell-documented to participate in the calcium signaling pathway, especially in cardiac and skeletal muscle." This is concrete biological pathway association that adds value to the graph.
66

---

Published in Transactions on Machine Learning Research (02/2026)
H
GraphRAG Prompts
11: GraphRAG Index Prompt
-Role-
You Al assistant specialized in extracting structured information from biomedical texts to build knowledge graph about diabetes.
-Goal-
Given some medical paper abstracts, predefined list of entity types, and a predefined list relations, identify all entities of those types and the medically meaningful relationships explicitly described among the identified entities within the abstract. You should only extract entities that are relevant diabetes, its complications, and comorbidites.
-Entitiy Types-
You should extract entities from following 5 entity types: Organism, Anatomical Structure, Manufactured Object, Substance, Conceptual Entity.
Use the subcategories listed below SOLELY guidance help you determine the correct main entity type. Ouly use the main entity types in your output.
1. Organism: Plant; Fungus; Virus; Bacterium; Archaeon; Eukaryote; Vertebrate; Amphibian; Bird; Fish;
Reptile; Mammal; Human
2. Anatomical Structure: Embryonic Structure; Anatomical Abnormality; Congenital Abnormality; Acquired Abnormality; Fully Formed Anatomical Structure; Body Part, Organ, or Organ Component; Tissue; Cell; Cell
Component; Gene or Genome
Manufactured Object: Medical Device; Drug Delivery Device; Research Device; Clinical Drug
4. Substance: Chemical; Pharmacologic Substance; Antibiotic; Biomedical or Dental Material; Biologically Active Substance; Hormone; Enzyme; Vitamin; Immunologic Factor, Receptor; Indicator, Reagent, or Diagnostic Aid; Organic Chemical; Nucleic Acid, Nucleoside, Nucleotide; Amino Acid, Peptide, Protein; Inorganic Chemical; Element, lon, or Isotope; Body Substance; Food
5. Conceptual Entity: Idea Concept; Body System; Body Space Junction; Body Location Region; Molecular Sequence; Nucleotide Sequence; Amino Acid Sequence; Carbohydrate Sequence; Geographic Area;
Finding; Laboratory Test Result; Sign Symptom; Organism Attribute; Clinical Attribute; Intellectual Product; Occupation or Discipline; Organization; Group
-Relation Types-
Please only identify the following 35 relations: ['associated finding of,' 'associated mowpphloogy_oooo 'as- sociated with,' 'causative agent_of 'cause_off _of," 'direct 'direct__procedure www.ppoureeeeeeeeeeeeee site _f,'' 'due__ot to,' 'finding_site of,' 'focus _o,' 'has associated mor___oorrooo....... 'has cansative sarrvivee_aeee,"""" ceas_clinnical _ourre,____eee" "has "has__oaas "otu_oneenntt 'has _di__ddree___e crcett_prooouu_eeeeeeeee site,"itt'' 'has direct__sii substttcebstneee 'has dispossiiiii 'has finding site,' 'has focus,' "has method, 'has modification,' 'has part,' "has pathological process, 'interprets," 'inverse isa,' 'is interpreted by,' 'is modification of," 'isa,' 'method of,' 'occurs before', 'oc- curs in, 'part of,'
structure," entire aitiire_nattoyy anatommysstructurre
role,' pposibbyyeqquivleent equivalent__eee to," 'same_ as"]
The following provides one example for each type of relation, formatted as 'head, relation, tail': fetal growth restriction (fgr), associated finding of, history of fetal growth retardation
tumors, associated mooproooooooo_________ neoplastic disease neutropenia, associated with, neutropenic sepsis
s. epidermidis, causative agent of, staphylococcus epidermidis ventriculitis chronic kidney disease (ckd), cause of, renal retinopathy
gastric fundus, direct procedure_stee site_sttts of, laparoscopic fundoplication
diabetic cardiomyopathy (dbem), due do, diabetes mellitus
endocrine pancreas, finding site_ of, extreme insulin resistance type gait abnormalities, focus_ of, prosthetic gail training
pyoderma gangrenosum, has associated morphology, neutrophilic infiltration chronic chagas disease cardiomyopathy, has causative agent, trypanosoma cruzi membranous nephropathy, has clinical course, chronic
67

---

Published in Transactions on Machine Learning Research (02/2026)
serum creatinine level, has co__ponmm,,, creatinine
fint, liss_ddrec______ prored____dddur___ peste, gustrointestinal tract structure
high-intensiily statins, has has__irret resveratrol (res), has disposition, platelet aggregation inhibitor middle occipital gyrus, has __ anatomy_struchure, entire lateral occipital gyrus diabetes retinopathy, has finding_sit site, retinal structure
lung-coa reduclase inhibitor
on-line hemodiafiltration, has focus, renal failure syndrome
islet cell transplant, has method, surgical transplantation uric acid, has modification, calcium urate
anaerobic glycolysis, has part, pyruvate kinase activity evans syndrome, has patholoolll process, autoimmune process endocrine hypertension, interprets, blood pressure
adaplive thermogenesis, inverse isa, diet induced thermogenesis seruin triglyceride, is interpreted by, serum triglyceride levels
tau, is modification of, uridine t cell receptors, isa, antigen receptor
amputation, method_ot of, cineplastic amputation
renal transplantation, occurs before, accelerated rejection renal transplant paediatric obesity, occurs in, childhood
bone resorplion, part__f, bone remodeling
everolimus (eve), plays_roke role, antineoplastic therapeutic role non-alcoholic fatty liver disease, possibly_equivaleeeee equivalent_ lo, fatty liver retinal cotton wool spots; same as, retinal exudates
-Steps-
Identify all entities corresponding to one of the main entity types and relevant diabeles, using subcategory examples as guidance for classification. For each identified entily, extract the following information: antity_namss Name of the entity, lowercase
Seiity_type One of the following types: Organism, Anatomical Structure, Manufactured Object, Substance,
Conceptual Entity.
entity description: concise description of the entity's attributes and activities. Format each entity satpttt__nmmm<<>>eettttt/>
2. From the entities identified in step 1, identify all pairs of (source_ entity, target_entity) that are clearly related each other according the given text, and medically meaningful. Only the relationships that are in the predefined list.
Avoid relationships that are attached to entities that are too general, for example: patients, bodily functions, parameters, management, optimization. Only keep the relationships that state facts, represent the main idea the text, other important relationships that are in the predefined list. It's acceptable if some entities identified the previous step are not used.
For each pair of related entities, extract the following information: source entity: name the source entity,
identified in step
largel_ _ntily: name of the larget entily, as identified in step
relationship: one relation that in the predefined list, according to the given text
relationship- ttrrrinnttisssstettttteee numeric score out of 10 indicating the strength of the relationship between the source entity and target entity
Format each relationship as
3. Return output in English as single list of all the entities and relationships identified in steps 1 and 2. Use ## as the list delimiter.
4. When finished, output 

68

---

Published in Transactions on Machine Learning Research (02/2026)
Constraints and Guidelines
Strict Textual Grounding: Base all extractions only the provided medical abstract. Do not use external knowledge or make assumptions beyond what wrilten.
Entity Filtering: Only extract the entities whose type is present in the provided 5 Entity Type, and only extract entities that are relevant to diabetes, its complications, and comorbidites.
Relationship Filtering: Extract only the 35 relationships as defined. Exclude all other relationships.
Delimiter Usage: Strictly adhere to the specified tuple delimiter within tuples and record delimiter between records.
12: GraphRAG Index Example
Input:
Entity Entety_types: Organism, Anatomical Structure, Manufactured Object, Substance, Conceptual Entity
Text:
CMR imaging enables comprehensive detection of myocardial remodeling in patients undergoing TAVR. Regression of LV matrix volume as surrogate for reversible diffuse myocardial fibrosis accompanied by increase of myocardial function and recovery from heart failure. Further data are required to define the value these parameters as therapeutic targets for optimized management of TAVR patients. t*iia***"
Output:
Outpul:
("entity<</>enr imaging< >Conceptual Entity</>A medical imaging technology that allows for the comprehensive detection of myocardial remodeling.) ##
/aaeelliinng/Cooootttttt Entity</>A pathological process involving changes in the heart muscle, observed in patients undergoing TAVR.)
##
TTaasaht/ Aortic Valve Replacement, medical procedure undergone by patients with myocardial remodeling.)
##
<< of LV matrix / Entity</>The reduction in the volume of left ventricular matrix.)
##
("entity"< diffuse myocardial I Entity< condition of the heart muscle for which the regression of LV matrix volume acts aurrogate.) ##
( function<) SCnnoppulll Entity<|>The physiological performance of the heart muscle.)
##
""entity<<[hhear" (aaluue<(>Conceppeete Entity<|>A clinical condition from which recovery is observed along with the regression of matrix volume.)
##
(*teelatcismatippk(hoyooaarrilla remodeling<(2emm> imaging< (aaggnng<>haa__eehheok
## ("rolattionstip((xeeeeesss of matrix volume<(xreversible diffuse myocardial tibrossic(liie_imteprreedd__ >><)>>)
("Melettonmaiiik(<>eeesssii of LV matrix volume< >mmyocariial function<| Dasiiiooc//ssssciattedd... with <|>10)
(MMelattiomatii((/rreeeessoo of LV matrix volume< Sllmee<>>hears >heart failure<(>associitte) wttpttteddddtthh/1o11000
## ("relariomsaapppppaaarr////oaooe r
69

---

Published in Transactions on Machine Learning Research (02/2026)
COMPLETE]
I3: GraphRAG Query Prompt
Role
You helpful assistant responding questions about data in the tables provided.
Data tables
{context_data)
GGal
Generate a response in the specified target format that the user's question using only the information provided in the tables. Do not use any external knowledge. If any prior knowledge conflicts with the tables, always rely on the tables as the sole source of truth.
If you cannot get the answer based on the tables, just say so. Do not make anything Points supported by data should list their data references follows:
"This is example sentence supported by multiple data references [Data: name> (record ids); <dataset name> (record ida)]."
<dataset
Do not list more than record ids a single reference. Instead, list the top 5 most relevant record ids and add "+more" to indicate that there are more.
For example:
"Person X is the owner of Company Y and subject to many allegations of wrongdoing [Data:
Entities (5, 7); Relationships (2, 7, 34, 46, 64, +more)]."
where 5, 7, 34, 46, and represent the (not the index) of the relevant data record.
Do not include information where the supporting evidence for it is not provided.
Target response format-
Response type: multiple paragraphs. Provide a concise answer using \boxed(], select only the correct letter from A, B, C, (e.g., \boxed([C)
Reference data points that support your answer using the given format (e-g., [Data: Relationships (2, 3, 4); Entities (35, 36, 37, 39, 55, +more)]). If no relevant information from the table supports your answer, leave the reference empty (e.g., []).
14: GraphRAG Context Example
Relationships
source (relation) -> target chemotherapy (cause_ou) -> peripheral neuropathy due to and following chemotherapy
chemotherapy induced peripheral neuropathy (due_to) >> administration of antineoplastic agent
antineoplastic drugs (associated_vithh >> resistance antineoplastic drug
chemotherapy induced peripheral neuropathies (due_to) -> administration of antineoplastic
agent
hematologic malignancies - (isa) -> neoplastic digease
lymphoid leukaemia (associated bhaaciaaed__findnng.oo -> history of lymphoid leukemia leukemia (associated, fondddffiiinnnofffoo ~> history of leukemia
oral chemotherapy for malignant neoplasm cancer (associated_ with) -> restrictive cardicmyopathy secondary to malignancy cancer (associated_ -> cancer related fatigue
cancer (focus_ot
imatinib (plays_role) -> antineoplastic therapeutic role imatinib (isa) -> antineoplastic agent
vorinostat (plays_role) -> antineoplastic therapeutic role
nivolumab (plays_rold -> antineoplastic therapeutic role
rucaparib (plays_role) -> antineoplastic therapeutic role
70

---

Published in Transactions on Machine Learning Research (02/2026)
nivolumab (isa) antineoplastic agent
rucaparib (isa) -> antineoplastic agent
antineoplastic agents (inverse_isa) vorinostat
71

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: FActScore* KG evaluation with Qwen3-32B as a judge: helper LLM ablation.
Bold: better than Qwen3-32B baseline; underlined bold: best in column.
Context + General truth
KG type
Model size
#triples
Context only
Qwen3-32B (baseline)
32B
515,160
10.2
48.1
GRAPHMERT with Qwen3-32B helper LLM
80M
138,201
69.8
72.2
GRAPHMERT with Qwen3-4B as helper LLM
80M
9,000
61.5
62.0
Table 12: ValidityScore evaluation with Qwen3-32B judge: helper LLM ablation. Bold: better than Qwen3-32B baseline; underlined bold: best in column.
Model size
yes (Validity- maybe Score)
KG type
#triples
no missing
Qwen3-32B (baseline)
32B
515,460
43.0
24.1
31.4
1.5
GRAPHMERT with Qwen3-32B as helper LLM
80M
138,201
68.7
18.3
10.8
GRAPHMERT with Qwen3-1B as helper LLM
SOM
9,000
63.8
18.5
15.1
2.7
Helper LLM ablation
In this section, we investigate how much our gains in FActScore and ValidityScore come from GraphMERT versus helper LLM that converts its token predictions into text tails. We provide an evaluation of 9,000 triples combined with smaller Qwen3-4B LLM.
Tables I1 and 12 compare GRAPHMERT KG using Qwen3-32B a helper LLM vs. GRAPHMERT KG using a smaller Qwen3-4B helper. The result shows that with a small helper, GRAPHMERT still clearly out- performs the bascline, indicating that a substantial part of the improvement comes from the own predictions of GRAPIMERT rather than from the capacity of the helper. At the same time, scaling the helper LLM up reduces nonsensical completions and helps form more nuanced, semantically precise tails, and yields further gains.
Although the small Qwen3-4B helper LLM is prompled to merge (oken-level predictions into fluent English tails, in practice, it sometimes produces ungrammatical or clause-like spans (e.g., "hybrid closed loop tech- nology, has part, hybrid closed insulin delivery loop pump systems,," "Idl-c / apo b ratio associated with high"), which we do not want to accept as tails. To filter out such malformed completions while preserving genuine biomedical phrases, we use a small set of syntactic heuristics. First, we normalize the tail span and use domain lexicon (keywords and medical suffixes, e.g., "-emia," " "-itis," *-and kinase") whitelist clearly biomedical strings, guaranteeing they will not be misclassified non-nouns or filtered out. We then reject tail if it is very long (> 40 characters) dominated by stopwords (stopword ratio 0.3 for spans relations (e.g., plays_rola role,
with 2 4 tokens). For relations other than small set of "adjacency/rol" has tos_associated moreeeooo_ohhh)))) we also require a nominal structure: Multi-token tails must contain at least one noun/proper noun and have a nominal root, and spans containing a finite verb are rejected as clause-like rather than entity-like.
Further, to let readers directly inspect the GRAPHMERT prediction quality, we also include raw GRAPH- MERT predictions for four sequences prior to filtering by a helper I.I.M. Examples A-D display the full input text chunk and the raw top-20 GRAPIMERT token predictions. Tokens like ##oscler are subword token outputs. We include outputs from standard GRAPHMERT trained with MNM and GRAPHMERT trained with a simple MLM objective. The former is used throughout the article as creates a basis for nuanced multi-token tails, increasing coverage based on fine-grained tails. This is why the output of GRAPHMERT trained with the MNM objective includes many subword tokens. GRAPIMERT trained with the MLM objective leans on trivial words, which may boost accuracy on some test tasks, but neglect fine-grained details.
72

---

Published in Transactions on Machine Learning Research (02/2026)
Raw GraphMERT Output: Example
Input Context Sequence:
examine direct and indirect costs, early retirement, cardiovascular events and mortality over years people with atherosclerotic cardiovascular discase (ASCVD) and matched controls in Sweden. Methods Individuals aged 2-16 years living in Sweden on 01 January 2012 were identified in an existing database. Individuals with ASCVD were propensity score matched to controls without ASCVD by age, sex and educational status. We compared direct healthcare costs (inpatient, outpatient and drug costs), indirect costs (resulting from work absence) and the risk of stroke, myocardial infarction (MI) and early retirement. Results After matching, there were 231,411 individuals in each cohort. Total
Query 1: (atherosclerotic cardiovascular discase, cause_of, of, ?)
GRAPHMERT with MNM objective:
[disease, cardiovascular, atheroeclerotic, with, vascular, atherosclerosie, ##ooscler, ##vd, cerebrovascular, arteri, chronic, ischemic, risk, coronary, of, disorder, dementia, hyporcholesterolemiaa, ##osclerosis]
GRAPHMERT MLM objective:
(atherosclerosie, ischemic, disease, cardiovascular, atherosclerotic, coronary, artery, ather, stroke, vascular, infarction, carotid, cerebrovascular, ischaemic, plaque, angina, arteri, chronic, arterial, vessel]
Query 2: (myocardial infarction, due to, ?)
GRAPHMERT with MNM objective:
[ischemia, due, ischemic, injury, ischaemis, circulatory, revascularization, caused, myocardial, infarction, cerebrovascular, to, disease, heart, vascular, cardiovascular, complication, of, bypass, stroke]
GRAPEMERT with MLM objectiver
(accident, poisoning, ischemia, injury, diabetes, cardlovascular, infarction, ischsemis, hypertension, care, emergency, burn, cardiac, accidente, revascularization, medical, finding, nyocardial, sccidental]
Query 3: (stroke, to, ?)
GRAPHMERT with MNM objective:
[ischemis, artery, cerebrovascular, vascular, ischemic, stroke, disease, revascularization, fracture, ischaemis, af, bleeding, injury, due, neuropathy, complications, venous, cardiovascular, complication, to]
GRAPHMERT with MLM objective:
(ischemia, accident, ischemic, ischaemia, ischaemic, diabetes, due, cerebrovascular, emergency, coronary, of, poisoning, adverse, myocardial, care, hypoglycemia, angina, acute, heart]
73

---

Published in Transactions on Machine Learning Research (02/2026)
Raw GraphMERT Output: Example
Input Context Sequence:
widespread, fatal, infectious disease. Several antivirals against rabies virus (RABV) infection have been reported, no approved, RABV-specffic antiviral drugs that inhibit RABV infection the clinic after symptom onset are available. Therefore, more effective drugs to reduce rables fatalities are urgently needed. Bardoxolone methyl (CDDO-Me), an FDA- approved compound that has long been known as an antioxidant inflammatory modulator and one of the most potent muclear factor erythroid-derived 2-iikeyyhroiddderiieee 2-likects myelin, axons, and CNS neurons by Nrf2 activation. Therefore, we investigated the potency of its
Query 1: (bardoxolone methyl, has disposition,
GRAPHMERT with MNM objective:
(inhibitor, oxidoreductase, agenist, receptor, ##olone, reductase, ##ulator, proliferator, aryl, histone, alkaloid, methyltransterase, peroxisome, hydrolase,
protease, antine]
GRAPHMERT with MIM objective:
[globulin, inhibitor, ##olone, steroid, compound, protein, ligase, agonist, polysaccharide, flavonoid, tetracycline, alkylating, glucoside, hydrolase, paptide, reductase, lyase, protease, antibiotic, isothiocyanate]
Query 2: (bardoxolone methyl, plays Mole, ?)
GRAPHMERT with MNM objective:
[therapeutic, #toplastic, antine, antibacterial, antimalarial, antifungal, #Rolone, #ooar, medicinal, role, anth, #lanoo anti, antipar, oral, pharmacological, sntimicrobisl, ##osidd, drug, ##azinn]]
GRAPIMERT with MLM objective:
(antinalarial, antiviral, antibacterial, quercetin, therapeutic, antine, drug, antibiotic, compound, chloroquine, doxycycline, azithromycin, tetracycline, role, antioxident, protease, medicine, flavonoid, antifungal, penicillin]
74

---

Published in Transactions on Machine Learning Research (02/2026)
Raw GraphMERT Output: Example C
Input Context Sequence:
CVS-11 and CTN) in N2a cells. We also examined whether CDDO-Me activates the Nrf2-asocciated pathway upon infection with RABV straius of differing virulence. Nrf2, phosphorylated soquestosome (SQSTM1), SQSTM1, hemoglobin oxygenase (HO-1) and NAD(P)H dehydrogenase quinone (NQ01) expression in N2a cells increased to varying degress with CDDO-Me treatment, accompanied by Kelch-like ECHl-sssociated protein (Keapl) dissociation, upon infection with SC16, CVS-11 CTN. The activation SQSTM1 phosphorylation was significantly
Query 1: (nrf2, part of, ?)
GRAPHMERT with MNM objective:
[-, response, cellular, pathway, /. signaling, defense, process, stress, keapl, to, intracellular, protein, mrf2, of, activation, degradation, oxidative, related, inflammatory]
GRAPHMERT MLM objective:
[nrf2, oxidoreductase, protein, antioxidant, signaling, muclear, kinase, pathway, transcription, response, chaperone, signal, isomerase, cytoprotective, keapl, activator, stress, apoptosis, autophagy, factor)
Query 2: (sqstml, part_off, ?)
GRAPHMERT with MNM objective:
[degradation, procesa, of, reaponse, pathvay, complex, oxidoreductase, biosyathetic, reticulum, catabolic, seques, autophagy, assembly, protein, sq, acid, synthesis, phase, ##osomal]
GRAPHMERT MLM objective:
(protein, autophagy, kinase, mitochondrial, transcription, p62, oxidoreductase, dogradation, phoephorylated, mitophagy, ubiquitin, ribosomal, reticulum, muclear, dehydrogenase, response, seques, gene, transcript, 1ysosomal]
Query 3: (oddo- me, cause__of of, ?)
GRAPHMERT with MNM objective:
(##do, to, ##o, ##in, 2, ##on, ##oside, epi, cd, of, toxic, ##e, protein, disease, product, #ty, selen]
GRAPEMERT with MLM objective:
[fk, heme, protein, stress, oxidoreductase, endoplasmic, autophagy, zine, chaperone, serine, pignent, keapl, cysteine, compound, er, apoptosis, mitochondrial, reactive, oxidative, neutrophil]
75

---

Published in Transactions on Machine Learning Research (02/2026)
Raw GraphMERT Output: Example D
Input Context Sequence:
inteerleukin-11 R6, and receptor activator of nuclear factor kappa-B (RANK). Together, proteomic data suggest the largeting of several key regulators of inflamattinn, bone, and adipose lurnover, via transforming growth factor-beta/SS betoos-ee////AD. and Wingless-related integration site/be-catenin signaling pathways. To the best of the knowledge, this is first evidence of an intervention that drives against bone loss via RANK. Metatranscriptomic analyses of the gut microbiota show P7C3 increased Porphyromonadacese bacterium, Candidatus Melainshacteria, and Ruminococcacese bacterium abundance, potentially contributing to the favorable inflammatory
Query 1: (interleukin r6, plays_rolee role, ?)
GRAPEMERT with MNM objective: like, and, -, chemoattractant, of, /, necrosis, receptor, 1, cytokine, cytokines, chemotactic, by, on, including, inducible, in,
GRAPAMERT with MLM objective:
[interlenkin, cytokine, response, chemokine, receptor, inflammatory, matalloproteinnse,, cytokines, matrix, sigmaling, pathway, protein, process, factor, mediator, immunomed, signalling, system, secretion)
Query 2: (receptor activator of nuclear factor kappa b, cause_ of,
GRAPIMERT' with MNM objective:
[auclear, activator, and, the, tnf, -, transient, a, activation, p38, activating, phosphorylated, 2, factor, an, adiponectin, of, receptor]
GRAPEMERT with MLM objective:
(receptor, ligand, transcription, protein, nuclear, interleukin, signaling, cytokine, inhibitor, activator, chemokine, activity, activated, transducer, kinase, #tulator, pathway, binding, activation]
Query 3: (transforming growth
beta smad, part_aof"
GRAPEMERT with MNM objective:
[growth, differentiation, factor, fibroblast, process, signaling, signalling, proliferation, /, epidernal, ##oganesis, wnt, morphogenstic, cell, extracellular, matrix, tgi, resorption, developnent, regeneration]
GRAPEMERT with MLM objective:
[receptor, signaling, differentiation, protein, hormone, factor, matrix, cell, gene, transcription, grouth, collagen, cytokine, response, adipogenic, fibroblast, secretory, signalling, unt, pathuay]
Query 4: (wingless related integration site be catenin, part__f, ?)
GRAPEMERT with MNM objective:
[wing, unt, morphogenetic, morphogenesis, differentiation, regeneration, process, /, cell, signaling development, natrix, signalling, of, phosphatidylinositool. grouth, #oonyosin, kinase, pathway, nuclear)
GRAPHMERT with MLM objective:
[unt, signaling, transcription, receptor, hedgehog, differentiation, transition, protein, factor, hameobox, enhancer, pathuay, notch, kinase, cell, adipogenic, natrix, signalling, hippo, gene]
76


Reviews:
[{"review_id": "8cY5hdLHGL", "reviewer": "Reviewer_Lui9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces GraphMERT, a compact encoder-only model designed to extract reliable knowledge graphs (KGs) from unstructured text. By combining neural abstraction with symbolic reasoning, GraphMERT addresses the scalability and interpretability challenges of neurosymbolic AI.  A key contribution is the joint learning of semantic signals derived from a seed KG and syntactic signals from unstructured text. This is achieved through a proposed masked node modeling. \n\nThe authors present experiments on medical texts (e.g., PubMed diabetes papers)  to compare the proposed method with existing LLM-based models in accuracy and validity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "## *Strenghts*\nThe authors provide a clear and well-structured formulation of the problem, with sufficient motivation and a thoughtful positioning of their work in relation to existing literature. The connection to knowledge graphs (KGs) is articulated effectively, highlighting their relevance to the proposed approach. As I am not an expert in this specific domain, I cannot fully assess the completeness of the citations or the novelty of the contribution. Nevertheless, the paper is well organized, flows smoothly, and is accessible to readers.\n\nThe empirical evaluations seem to show a clear advantage of the proposed work (GraphMERT) compared to existing LLM-based methods. The adoption of metrics such as FActScore and ValidityScore appears appropriate and well-justified.\n\n\n## *Weakness*\nMy main concern regarding this work is that it relies on a helper LLM that introduces a potential limitation. The accuracy of the extracted relations is contingent on the helper model’s domain expertise. In cases where the helper LLM lacks sufficient knowledge of the target domain, the results may be incomplete or erroneous, thereby reducing the overall reliability of the proposed method based on the knowledge graph.", "audience_yn": "Yes", "audience_explanation": "The paper is of sufficient interest to some members of TMLR, especially those working on the knowledge graph.", "requested_changes": "1. A discussion on the limitations of the proposed model, especially when the helper LLM lacks sufficient knowledge of the target domain. \n2. Approaches to help mitigate the issue of a bad LLM helper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j4ND2kfbl5", "reviewer": "Reviewer_VVhw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\\\nThis paper introduces a compact encoder-only model GraphMERT, which is designed to extract structured knowledge directly from domain-specific text with mixing syntactic sequences and semantic relational cues. GraphMERT employs leafy chain graphs and H-GAT to fuse syntactic-semantic information and then trained with the objectives MLM + MNM to learn both contextual and structural signals. KG results show strong validity and reliability, significantly outperforming LLM-generated KGs in FActScore and ValidityScore.\n\n&nbsp;\n\n**Strengths**\n1. The problem setup is clear and timely. The paper effectively motivates the need to automatically derive reliable KGs from text corpora by highlighting limitations of LLM-based KG construction, including reliability issues and potential biases in triplet generation.\n2. The overall narrative flows well, and the related work section is well organized. The background and prior research are connected in a way that makes the method easy to follow.\n3. GraphMERT shows strong empirical gains over LLM-generated KGs (e.g., FActScore and ValidityScore), demonstrating the benefits of the structured modeling approach.\n\n&nbsp;\n\n**Weaknesses**\\\nI'm not very familiar with this specific research area, such as KG construction or graph-based Transformer architecture. Therefore, it is difficult for me to precisely assess the novelty of the proposed method relative to existing work.\n1. From my perspective, Some components appear somewhat close to prior work (e.g., the span masking resembles SpanBERT, and the attention decay idea feels similar to GradFormer). However, this impression may be due to my limited familiarity with this area.\n2. At times, the paper presents KGs as if they are universally beneficial (e.g., in Section 2.2.3 on reliability in critical fields). The claims could be toned down slightly to maintain balance.\n3. This is a minor comment, but the transition in the related work section feels slightly abrupt: Sections 2.1–2.6 provide general background, while Section 2.7 begins describing elements of the proposed method. These portions might fit more naturally within a unified preliminaries or method section, though this is not a major issue.\n4. While the method is described as domain-agnostic, the evaluation is limited to a single domain, where seed KG size and vocabulary diversity may not reflect other settings. Including an additional domain in the experiments would provide clearer evidence of the method’s generalizability.\n5. Because the final triplet relies on a helper LLM, the quality of the resulting relations may be sensitive to the helper model’s domain knowledge. If the helper LLM lacks sufficient expertise in the target domain, it may struggle to construct accurate or meaningful relations, potentially limiting the reliability of the extracted KG.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Although some aspects of the syntactic–semantic modeling are conceptually abstract, the paper presents clear and convincing qualitative and quantitative results, along with informative ablations, that effectively support its main claims.", "audience_yn": "Yes", "audience_explanation": "I believe that some members of TMLR’s audience would find the paper’s contributions interesting, particularly those working on knowledge graph construction or structured representations. That said, the paper is somewhat long in its current form, and presenting in a more concise and focused manner could make the work even more approachable and engaging for readers.", "requested_changes": "1. It may be helpful to moderate some of the broader claims regarding the universal advantages of KGs to present a more balanced perspective.\n2. As a minor suggestion, refining the structure of the related work and preliminaries to create a smoother transition could improve readability.\n3. Extending the experimental evaluation beyond a single domain would provide a stronger assessment of the method’s generalizability and overall impact.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RpCsavBYsM", "reviewer": "Reviewer_vaak", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces GraphMERT: a small encoder model that generates a Knowledge Graph (KG) from unstructured text. The KG is then used for performing various AI tasks. The paper then experiments on datasets such as FActScore in each case showing that use of KG outperforms the accuracy of base LLM. The paper also performs improvements on the KG starting from the original one obtained from the encoder model reporting its resulting accuracies. It proceeds to improve the KG over a few steps, at each step, reporting further improvements in the resulting accuracy. Based on these experiments, paper claims that GraphMERT leads to efficient, scalable, transparent, attributable, accountable, editable, auditable, and continually improvable state-of-the-art neurosymbolic AI.", "claims_and_evidence_yn": "No", "claims_explanation": "### Weak experiments:\n\nIn my view, the paper’s method is interesting, yet, its experiments are weak, and it over claims its contributions. \n\nThe long list of adjectives that paper uses to describe the contributions of its model seems to me quite overblown (efficient, scalable, transparent, attributable, accountable, editable, auditable, and continually improvable). Many of these adjectives apply to LLMs, too. First and foremost paper needs to compare its contributions to the SoTA models.\n\nFor its motivation, paper reports experimenting with powerful models such as Grok on a topic related to kidney disease and it reports that those LLMs are not quite accurate in finding the relationships between the concepts related to kidney disease. However, when it comes to its own main experiments on FActScore, the paper does not compare its results to the accuracy of models such as Grok. In the best case, the paper achieves an accuracy of 76.9% on FActScore while it does not report that Grok 4.0 achieves an accuracy of 90% and Grok 4.1 achieves an accuracy of 97% on FActScore.\n\n\n-------\n\n### Weak literature review:\n\nExtracting knowledge/representations from data and then leveraging that knowledge/representation is used all across the AI literature, but paper does not mention this clear enough. I give just two examples from a large number of papers that I am aware of.\n\nChen, C., Li, O., Tao, D., Barnett, A., Rudin, C. and Su, J.K., 2019. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32. – Extracts prototypes from the datasets and then processes the inputs by comparing the inputs against the prototypes.\n\nKadkhodaie, Z., Guth, F., Simoncelli, E.P. and Mallat, S., Generalization in diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations.\n\n\nSimilarly, extracting knowledge graphs from unstructured text is not a novel approach, neither leveraging the KG for boosting LLM accuracy. Below are only two papers that this paper has not cited/acknowledged.\n\nDeng, Y., Ye, C., Huang, Z., Ma, M.D., Kou, Y. and Wang, W., 2024. Graphvis: Boosting llms with visual knowledge graph integration. Advances in Neural Information Processing Systems, 37, pp.67511-67534.\n\nLiu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H. and Wang, P., 2020, April. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 03, pp. 2901-2908).\n\n\nI suggest adding a new section in the paper to review the above two lines of research in the literature and thoroughly citing the prior work, not just the two example that I gave.\n\n\n-------\n\nThe paper presents a case study about kidney which is too brief in my view. If paper has coauthors that are experts in kidney disease, that’s fine, otherwise, it would be best for a medical expert to be invited to review those claims and verify their correctness.\n\n\n-------\n\n\nMinor comment:\n\nSome of the language in the paper does not read sound to me. For example, when paper says “symbolic components provide abstraction while neural components provide generalization”, I find it rather naive, or in the best case, vague, in terms of how it views generalization. Abstraction itself is a very broad concept that is hard to disentangle from generalization when it comes to the applications that paper studies.", "audience_yn": "No", "audience_explanation": "No for this current version of the paper, in my view. But possibly yes, for a revised version of the paper.\n\nThe idea to include a set of useful keywords in the prompt to an LLM is not novel for the TMLR's audience.\n\nMuch of the paper is reviewing and explaining well known concepts such as Knowledge Graphs, Attention Networks, etc. Then it goes on to report the output of several models on a case study about kidney which might not be a clear example for many readers.\n\nThe output from Gemini Pro 2.5 without using the KG is quite brief and it is not clear whether a more detailed answer could have been obtained from this model without using the KG. At the end, it is not clear to me if there is any specific contribution in the method presented in the paper. If someone is specifically interested in a KG or a set of keywords, they can already include those keywords in their prompts to guide the LLMs. The encoder model itself is a simple method and there are already plenty of methods available in the literature for extracting KGs from unstructured text. Whether the KG leads to better accuracies, requires experiments on more advanced models, in my view.\n\nGiven the 29 pages preceding the experimental results in section 6, I was expecting a much stronger set of experiments. However, experiments are on only one dataset and one LLM (Qwen3-32B).", "requested_changes": "1. I would like to see experiments on several more LLMs and also on more datasets. SoTA LLMs, open source and closed source, needs to be covered in the experiments.\n\n2. Expanding the literature review and acknowledging what is already present in the recent AI literature and framing the contribution of this paper with respect to that literature..\n\n3. The paper needs to be structured better and more efficiently making it easier to read by the community. All the main contributions and experimental results need to come in the main body of the paper while trivial explanations, details, and extended analysis can go into the appendix. 38 pages for the main body of the paper seems too long in my view. Paper can assume its readers already know what is a knowledge graph or they can easily read about it in other sources, and so on.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "85d6WhZNzs", "reviewer": "Reviewer_94oo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces GraphMERT, a relatively small (80M parameters) encoder-only Transformer model designed to distill reliable, domain-specific Knowledge Graphs (KGs) from unstructured text. Unlike current trends that rely on large Generative LLMs for KG construction, GraphMERT operates by jointly encoding syntactic information (text) and semantic information (KG triples) into a unified data structure called \"Leafy Chain Graphs.\" Key contributions are below:\n\n1.\tLeafy Chain Graph Encoding: A novel input representation that grafts semantic relation edges and nodes (leaves) onto syntactic token sequences (roots), allowing the model to learn joint representations.\n\n2.\tArchitecture: An integration of a RoBERTa-style encoder with Hierarchical Graph Attention Networks (H-GAT) and a geodesic distance-based attention mask to handle the structural inputs.\n\n3.\tEfficiency and Reliability: The authors demonstrate that this compact model, when trained with a high-quality seed KG, outperforms significantly larger LLMs (e.g., Qwen-32B) in terms of ontology compliance (ValidityScore) and factuality (FActScore) within the medical domain (specifically Diabetes), while reducing hallucinations common in generative approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission provides a strong empirical evaluation to support its primary claim: that a specialized, smaller model can outperform general-purpose LLMs in extracting strictly ontology-compliant KGs.\n\n•\tMetrics: The use of FActScore and ValidityScore is appropriate for measuring the two key failure modes of LLMs: hallucinations and ontology violations.\n\n•\tComparison: The comparison against Qwen3-32B (a strong open-weights model) is fair and highlights the efficiency-performance trade-off. The results showing GraphMERT achieving ~69% FActScore vs. LLM's ~40% are compelling evidence of the method's effectiveness in constraining output to valid facts.\n\n•\tAblation Studies: Section 6.4 effectively isolates the contributions of the H-GAT module, span masking, and dropout, justifying the architectural choices.\n\nHowever, there is a nuance regarding the \"Helper LLM\" used in the inference pipeline (discussed in Requested Changes), which complicates the attribution of success solely to GraphMERT, though the evidence broadly supports the utility of the proposed framework.", "audience_yn": "Yes", "audience_explanation": "The paper sits at the intersection of Neurosymbolic AI, Knowledge Graph construction, and Efficient NLP. Given the current saturation of LLM-based research, TMLR readers will likely appreciate a method that advocates for smaller, structured models for high-stakes domains (like healthcare) where interpretability and strict ontology adherence are more critical than open-ended generation.", "requested_changes": "I find this work to be a solid contribution to the field of KG extraction. The methodology is distinct from the dominant \"prompt-the-LLM\" paradigm. However, to fully secure a recommendation for acceptance and clarify the technical contribution, I request the following adjustments:\n\n1. In Section 4.4 and 5.1.3, the pipeline uses a \"Helper LLM\" (Qwen3-32B) to combine GraphMERT-predicted tokens into coherent tails. To what extent is the FActScore/ValidityScore attributable to Qwen3's filtering capabilities rather than GraphMERT's predictions? If GraphMERT predicts a bag of tokens and Qwen3 reconstructs the meaning, Qwen3 is doing significant semantic heavy lifting. So please include an analysis or a baseline where a simpler, non-LLM method (e.g., a simple heuristic or a smaller language model) is used to combine tokens. Alternatively, provide qualitative examples of \"raw\" GraphMERT token outputs before the Helper LLM processes them, so readers can judge the raw signal quality.\n\n2. The paper relies heavily on \"LLM-as-a-judge\" (Qwen3-32B and GPT-5) for evaluating FActScore and ValidityScore. While this is standard practice, the paper targets the medical domain under the banner of \"Reliability.\" LLMs are known to have biases as judges. Please conduct a small-scale human evaluation (e.g., on 50-100 randomly sampled triples) to calibrate the automated metrics. Does human judgment align with the GPT-5/Qwen verdicts? This is crucial for a paper claiming \"reliable\" medical AI.\n\n3. The primary baseline is an LLM (Qwen) prompted to generate KGs. It would strengthen the paper to verify how GraphMERT compares to a traditional, fine-tuned BERT-based Relation Extraction (RE) model or a standard sequence-to-sequence extraction model (like T5-base fine-tuned on the triples). GraphMERT is essentially a specialized fine-tuning method. Readers might wonder if a vanilla fine-tuned RoBERTa or T5 (trained on the same text-triple pairs) would achieve similar results without the complex \"Leafy Chain Graph\" structure. Showing that GraphMERT outperforms a standard fine-tuned small model would solidly validate the architectural contribution (H-GAT + Unified Graphs).\n\n4. Section 4.1 and Figure 3 introduce the data structure of \"Leafy Chain Graph\". The description is technically precise but dense. Please add a brief intuition on why this structure helps. Is the main benefit that it forces the model to attend to the \"semantic\" leaves using the same mechanism it uses for \"syntactic\" text? A sentence or two clarifying the intuition behind the \"joint representation\" would help.\n\n5. The training data is ~125M tokens (Table 1). This is quite small by modern standards. Please explicitly discuss in the Limitations section whether this approach scales well to larger corpora or if it is specifically designed for \"low-resource, high-quality\" scenarios.", "broader_impact_concerns": "The authors have briefly touched upon reliability in medicine. However, a specific concern arises from the \"Helper LLM\" component. If the Helper LLM is a black-box commercial model or a large open model, it reintroduces the very \"hallucination\" and \"lack of provenance\" risks the paper argues against, just at a different stage of the pipeline (token combination rather than triple generation).\nI suggest adding a sentence in the Limitations or Broader Impact section acknowledging that while GraphMERT reduces hallucinations, the final assembly step still relies on a probabilistic model (the Helper LLM), which technically breaks the \"guaranteed\" chain of provenance, even if the risk is minimized. No specific Broader Impact Statement is missing, but this nuance is ethically relevant for medical AI.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "optXvAqZzb", "reviewer": "Reviewer_FEBJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes GraphMERT, an encoder-only transformer augmented with hierarchical graph attention and a combined MLM + MNM objective, designed to extract reliable domain-specific knowledge graphs (KGs) from small high-quality text corpora. The authors argue that existing LLM-based KG generation methods are unreliable due to hallucinations, prompt sensitivity, and poor domain alignment.\n\nStrength:\n\n1.Clear motivation and problem framing: The authors thoroughly articulate the shortcomings of current LLM-based KG generation—hallucinations, prompt brittleness, absence of provenance, and domain-misaligned relations. The motivational example using UMLS triples effectively demonstrates the brittle behavior of frontier LLMs.\n\n2.  Novel graph-based encoder architecture: GraphMERT introduces a meaningful architectural innovation:\n        1.  a leafy chain graph representation that merges syntactic (roots) and semantic (leaves) information;\n        2.   H-GAT fusion that injects relation-aware semantic representations;\n        3.  a distance-sensitive attention decay mask, integrating structural information.\n\nWeakness:\n\n1. FActScore* and ValidityScore rely on Qwen3-32B as the judge, even though Qwen3-32B also creates the baseline KG. This setup introduces risks: The judge may be more forgiving or harsher toward its own patterns.\n\n2. Limited discussion of computational cost: Although the model is compact, the pipeline includes heavy KG injection, H-GAT processing, span masking, entity linking with ANN search. There is no discussion present on the computational cost in the paper. \n\n3. The seed KG heavily defines relation vocabulary and tail semantics. This is briefly mentioned in limitations but deserves more rigorous treatment. However there should be a discussion what happens the seed KG is noisy or incomplete.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The major claims of the submission—regarding improved factuality, validity, provenance, and reliability are well-supported by clear, quantitative evidence and reinforced with diagnostic analysis and ablations", "audience_yn": "Yes", "audience_explanation": "I think the research problem addressed in this paper is interesting and of interest to TMLR.", "requested_changes": "Requested Changes\n1. The authors should provide clarity on training/inference time, GPU hours, and memory requirements.\n2. The paper provides detailed LLM failure examples but does not present. It would also be interesting to include failure modes of GraphMERT.\n\n\nOptional Changes:\nIf possible expand the evaluation beyond the medical domain to show the generalisability of the approach.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)