reguloGPT: Harnessing GPT for End-to-End Knowledge Graph Construction of Molecular Regulatory Pathways

Authors: xidong  wu
sumin  jo
yiming  zeng
arun  das
ting-he  zhang
parth  patel
yuanjing  wei
lei  li
shou-jiang  gao
jianqiu  zhang
dexter  pratt
yu-chiao  chiu
yufei  huang
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-07-05 20:44:21

Abstract: Molecular Regulatory Pathways (MRPs) are key to understanding biological functions. Knowledge Graphs (KGs) help organize and analyze MRPs by structuring complex interactions. Current methods for extracting KGs from biomedical literature struggle with hierarchical relationships and context. Large Language Models (LLMs) like GPT-4 show promise in addressing these issues but remain underexplored for end-to-end KG construction. We present reguloGPT, a novel GPT-4 based in-context learning prompt designed for the end-to-end extraction of a regulatory graph and context from a sentence that describes regulatory interactions. reguloGPT employs a context-aware relational graph to capture MRPs’ hierarchical structure and resolves semantic inconsistencies by embedding context directly within the relational edges. We created a benchmark dataset comprising four hundred annotated PubMed titles on N6-methyladenosine (m6A) regulations. Rigorous evaluations of reguloGPT on the benchmark dataset showed marked improvements over existing algorithms and other LLMs. We further developed a novel G-Eval scheme, leveraging GPT-4 for annotation-free performance evaluation that demonstrated agreement with evaluations on the benchmark dataset. Lastly, we constructed m6A-KG by applying reguloGPT to 1,396 m6A-related titles and demonstrated its utility in elucidating m6A’s regulatory mechanisms of cancer phenotypes across various cancers. These results underscore reguloGPT’s potential for advancing biological knowledge extraction. All reguloGPT works including source code, benchmark datasets, and m6A-KG are available at https://github.com/Huang-AI4Medicine-Lab/reguloGPT.

Main paper: 
 i. introduction: 
 Molecular Regulatory Pathways (MRPs) are central to our understanding of the molecular mechanisms controlling biological functions. Studying MRPs allows scientists to uncover disease-contributing dysregulations and guide the development of targeted therapies. For organizing and analyzing the ex-tensive data within MRPs, Knowledge Graphs (KGs) have become instrumental. These KGs offer structured representations of complex interactions among various entities such as genes, proteins, and biological processes within MRPs [1] , [2] . While databases like KEGG [3] have been established through meticulous human curation, the sheer volume and pace of new research publications pose a significant challenge to such manual efforts. To address this, automated Natural Language Processing (NLP) methods have been developed by combining rule-based and machine-learning strategies to improve the extraction of biomedical knowledge from literature, resulting in databases like INDRA [4] .MRPs are characterized by intricate relationships and hierarchical structures that span genes, proteins, and biological processes within a biological context. Text descriptions of MRPs, such as "METTL3-mediated m 6 A methylation of SPHK2 promotes gastric cancer progression by targeting KLF2," suggest a context-specific relational graph involving several entities like METTL3, m 6 A, SPHK2, KLF2, progression, and gastric cancer as context (Fig. 1A ). The relational graph encompasses both explicitly and implicitly mentioned relationships that collectively define the mechanism by which METTL3 regulates the progression of gastric cancer.Biomedical KGs are essential for integrating and analyzing complex data across various biomedical domains. KG can unify data from genomic databases, drug information, and research publications to provide a comprehensive view, enabling researchers to uncover insights that isolated data might miss and facilitating hypothesis generation by exploring connections between genes, proteins, diseases, and drugs, revealing new research opportunities and potential therapeutic targets. Additionally, KGs enhance the understanding of disease mechanisms and support context-specific predictions by comparing normal and disease-specific pathways, offering more precise insights for interventions. The advent of Large Language Models (LLMs) like GPT-4 represents a significant leap forward in Natural Language Processing (NLP), providing deep insights into the contextual dynamics of language. LLMs have transformed the traditional view of language from a static set of terms and rules into relational links between words [5] . This perspective aligns well with the core objective of KGs, which involves mapping out a network of relationships among entities. While LLM-based in-context learning (ICL) has demonstrated state-of-the-art performance in biomedical NLP tasks without expensive training or finetuning, its potential for end-to-end KG construction of MRPs remains largely unexplored, presenting a promising frontier in the field of biomedical research [5] . In this paper, we explore the capability of GPT-4 for the end-to-end construction of a context-aware relational graph to accurately delineate the context-specific MRPs of m 6 A mRNA methylation from a given sentence. Our contributions are:1) We proposed reguloGPT, a GPT-4 driven ICL prompt specifically designed for the end-to-end extraction of the regulatory graph and context to accurately interpret context-specific MRPs. We designed baseline, few-shot, and Chain-of-Thought (CoT) prompts for reguloGPT. 2) We introduced a context-aware relational graph representation of regulatory interactions in MRPs (Fig. 1A ). This graph uniquely incorporates the context as part of the relational edges, thereby addressing and resolving the semantic contradictions of relations that often arise when contexts are not considered (Fig. 1B ). It also possesses the inherent regulatory hierarchy of MRPs. 3) We annotated the context-aware relational graphs derived from 400 paper titles related to m 6 A MRPs to create a benchmark dataset. This dataset encompasses a diverse array of contexts, entities, and relationships, making it highly valuable for the proper evaluation of reguloGPT. 4) We thoroughly evaluated the performance of reguloGPT for predicting contexts, entity recognition, and extracting both explicit and implicit relationships. reguloGPT demonstrated significant improvement over several existing algorithms and LLMs. 5) To overcome the need for manual annotation in evaluating reguloGPT, we introduced a novel G-Eval scheme, which leverages the CoT prompt to evaluate extracted context and relational graphs. We demonstrated a strong similarity between G-Eval scores and annotation-based evaluations, suggesting its potential capability for annotationfree KG evaluation with larger datasets. 6) We applied reguloGPT to paper titles addressing m 6 A MRPs and constructed a comprehensive m 6 A-KG. We demonstrated its utility by exploring m 6 A-mediated pathways and delineated a mechanism that the m 6 A writer METTL3 regulates cancer-related phenotypes in breast cancer, lung cancer, and myeloid leukemia.Several methods have been devised to extract relationships from text. These methods can be largely categorized as rulebased [6] - [9] and machine-learning-based [10] . However, they are mostly pipelined approach, focusing on extracting individual triplet only [11] . Extracting detailed graphs containing more than two entities from MRP descriptions challenges these triplet-based approaches, as they fail to consider correlations between triplets within a graph. This limitation results in cascading errors from misidentified entities, causing the extraction of redundant or missed relationships, and leads to exponential computational complexity [12] . Although a few recent works have considered graph extractions with three entities, such as drug-gene-mutation relationships [13] , generalized approaches capable of extracting graphs of varying entity sizes are underexplored in the biomedical domain. Though the existing methods have their own advantage such as precision, control, interpretability, and adaptability, they are limited in capturing important contextual information like diseases and tissue types, potentially leading to inconsistencies and misinterpretations in biomedical KGs [14] . Recent advances in LLMs allowed researchers to address those limitations, providing a more integrated and adaptable approach to relationship extraction. Leveraging LLMs for knowledge graph construction provides advantages in handling complex and contextual relationships. Research and practical implementations are increasingly focusing on these capabilities to overcome the limitations of traditional triplet-based systems [15] - [18] . While LLMs have made noticeable strides in capturing multiple relationships, there are still challenges to address such as the effectiveness, scalability, and reliability of these methods in constructing detailed and accurate knowledge graphs [15] . Not only do these limitations in methods hinder efforts to address the challenges, but the lack of specialized benchmark datasets for large graphs of MRP also impedes progress. Existing datasets such as SemRep [6] have inspired new approaches for general biological relationships or chemical-disease relationships to some degree. However, they were not designed to represent the full spectrum of complex and context-dependent MRP graphs. Hence, there is an urgent need for annotated benchmark datasets and innovative approaches to extract these detailed, context-dependent relationships that are central to elucidating MRPs from the literature.reguloGPT is a GPT-4 prompt carefully designed and thoroughly assessed to extract a regulatory relational graph and its biological context from a sentence describing MRPs. The development of reguloGPT involves six key modules (Fig. 1C ): ICL reguloGPT prompt engineering, data collection and annotation of the benchmark dataset, G-Eval prompt engineering, performance evaluation, context-aware m 6 A-KG construction, and downstream analysis of m 6 A-KG. We engineered three reguloGPT prompts (baseline, few-shot, and CoT prompt) to harness the full potential of GPT-4 for end-to-end contextaware relational graph extraction. To assess the efficacy of reguloGPT, we comprehensively evaluated its performance against the benchmark dataset and compared it with six existing algorithms. In addition to traditional evaluation, we further introduced an annotation-free evaluation scheme, G-Eval, and demonstrated its capability for future applications. We then applied reguloGPT to all titles to construct a context-specific m 6 A-KG and analyzed m 6 A-associated regulatory pathways, delineating context-aware relational graph representation and novel regulatory mechanisms across various cancers.To harness the potential of ICL, we developed three distinct reguloGPT prompts (Fig. 2A, B, and C ). While the baseline prompt consists of instructions, definition, and output format (Fig. 2A ), the few-shot prompt adds demonstrations (Fig. 2B ) and the Chain-of-Thoughts (CoT) [19] prompt includes additional reasoning steps (Fig. 2C ) with the demonstrations.• Instruction specifies the task of reguloGPT for GPT-4• Definition defines the nodes, edges, and context of the context-aware relational graph. Each edge includes two nodes and a predicate. We also incorporate implicit edges since many relationships are logically derived but not explicitly stated (Fig. 2A ). Definition also outlines a set of constraints for extraction of nodes and edges.• Demonstration provides a few examples containing pairs of sentences and the relational graphs extracted from the sentences, along with their corresponding contexts. We included four examples in our prompt (one is illustrated in Fig. 2B ). • Chain-of-Thoughts adds a series of intermediate reasoning steps for each example (Fig. 2C ), encouraging a complex and logical response from LLM. • Output provides instructions for the desired output format.The lack of benchmark datasets for context-aware MRPrelated relational graphs is a primary obstacle to comprehensively assessing the proposed reguloGPT. To address this limitation, we developed an annotated benchmark dataset based on papers focusing on m 6 A mRNA methylation. m 6 A methylation, the most abundant mRNA modification in mammalian cells, exhibits highly dynamic and complex mechanisms. m 6 A methylation has emerged as a highly active research area, which has garnered much interest in recent years. Creating the annotated dataset focusing on m 6 A circumvents the challenge of information overload inherent in more established research domains, yet effectively covers both general and nuanced MRP-related descriptions. We specifically focused on the titles of these papers as they provide the most concise description of molecular regulations. To compile this dataset, we searched PubMed using PubTator [20] with the keyword 'm 6 A' to extract papers published from 2013, the inception of this area, to 2023. Subsequently, we selected titles that encompass complete sentences with genes.To facilitate the annotation of a benchmark dataset, we assembled a team of five subject-matter-expert annotators with backgrounds in computer science and biomedicine to annotate 400 titles containing MRPs from the m 6 A research papers. The annotation process consisted of three steps: a) Practice phase: We randomly selected 20 sentences as practice examples. Annotators followed the instructions to identify nodes, edges, and context, engaging in discussions to reach a consensus. They also documented special cases for further annotation; b) Group annotation phase: Guided by the annotation guidelines established during the practice phase, we divided the 400 titles among the five annotators. Each annotator reviewed and annotated a subset of titles, after which they cross-reviewed another annotator's work c) Adjudication phase: Annotations were considered final if both annotators agreed on them. In cases of disagreement, the annotations were discussed within the group until a consensus was reached.We used Gilda [21] to normalize nodes initially. Subsequently, we performed manual normalization to ensure consistency. Nodes were further categorized into five types: m 6 A, m 6 A-Writers/Erasers/Readers (m 6 A-WER), Gene Protein, GO Pathway, and Other. For edge normalization, we updated the ontological predicate definitions in SemRep [6] , resulting in a dictionary of 32 predicate types: administered to, affects, associated with, augments, causes, coexists with, compared with, complicates, converts to, diagnoses, disrupts, higher than, inhibits, interacts with, isa, locates, lower than, maintains, manifests, methods, occurs in, part of, precedes, predisposes, prevents, process of, produces, same as, sensitizes, treats, and uses. Similarly, we normalized contexts adopting cancer types based on definitions from The Cancer Genome Atlas (TCGA) [22] .We adopted relaxed entity matching scheme used in [23] and evaluated the performance as follows: a) Node: A node is considered true positive if it achieves more than 50% similarity with the ground truth. Nodes failing to meet this criterion are considered as false positives. Any ground truth nodes not predicted are counted as false negatives; b) Edge: After normalizing using the predicate dictionary, each edge is evaluated based on its predicate and the node alignment. An edge is true positive if both the predicate and node alignment matched the ground truth. False positives occur when node alignment is correct, but the predicate does not match. Any ground truth edges not predicted were false negatives; c) Context: Accuracy serves as the metric for context prediction. It is considered correct if achieving more than 50% similarity with the ground truth context.Since manual annotation is labor-intensive and costly, assessing context-aware KG construction based on human annotations at scale is practically infeasible. Recent research on annotation-free evaluation of Natural Language Generation has leveraged LLMs directly as evaluators [24] . For example, GPTScore utilized LLMs to evaluate candidate outputs, assigning scores based on generation probability without referencing any human annotations. Following a formfilling paradigm [25] , GPT-4 was also able to assess the coherence, consistency, fluency, and relevance of generated texts, with high agreement with human evaluators. Inspired by these exciting studies, we proposed G-Eval (Fig. 2D and E ), a novel annotation-free evaluator based on GPT-4 for reguloGPT output. G-Eval employs the form-filling paradigm, encompassing a context evaluation (Fig. 2D ) that assigns a score to the extracted context, and a graph evaluation (Fig. 2E ) that scores the final extracted graph from a sentence. The prompts for context and graph evaluations include four parts: 1) Introduction; 2) Definition, which describes the concept of context in the context evaluation, or the concepts of nodes and edges in the graph evaluation; 3) Evaluation Steps; and 4) Output Format. These concepts-contexts, nodes, and edges-are consistent with those defined in the reguloGPT prompts (Fig. 2A ). The evaluation steps were generated by GPT-4 based on the introduction and definition. Scores range from 1 to 5, and G-Eval repeats the evaluation five times to obtain an average score as the final output [26] .In addition to the annotated benchmark dataset, we processed 969 m 6 A-related titles with our reguloGPT-CoT prompt to extract context-aware graphs. We applied normalization to standardize the extracted nodes, edges, and contexts. These normalized relational graphs were integrated with those from our benchmark dataset by linking common nodes and edges to construct m 6 A-KG, a comprehensive KG detailing m 6 A functions across diverse contexts. A distinctive feature of m 6 A-KG is that each edge incorporates a set of associated contexts to inform the context under which the regulation defined by the edge occurs. Each edge also includes the extracted raw edge values and PubMed Identifier (PMID) of the associated titles, enabling traceability back to the original paper for reference. Neo4j was used [27] for visualizing and managing m 6 A-KG.Our query of PubMed yielded 1,369 research papers whose titles describe m 6 A MRPs. From these, we selected 400 titles and meticulously annotated their context-aware relational graphs addressing the regulatory mechanisms of m 6 A mRNA methylation to create the benchmark dataset. The benchmark data comprises 787 nodes, 1374 links with 153 predicates, and 243 contexts: equivalent to a graph with an average of 3.71 triplets and 0.86 context per title. Subsequent normalization resulted in 695 unique nodes, 1282 links with 21 predicates, and 160 unique contexts including 49 cancer contexts.We evaluated reguloGPT's performance against manual annotation using the benchmark dataset, comparing it to four established algorithms as baselines: OpenIE [8] , TRIPS [9] , REACH [7] , EIDOS [10] . Note that these algorithms do not output context information, so context evaluation results of these algorithms are not presented. Besides these baseline algorithms, we also evaluated the CoT prompt with Mixtral-8x22B and Llama3-70B, two widely used open-source LLMs. Mixtral-8x22B is the latest and largest mixture-of-experts LLM from Mistral AI, known for its efficient architecture and strong performance. Llama3-70B is the most advanced model in Meta's Llama series, featuring 70 billion parameters and optimized for instruction-following tasks. These models represent state-of-the-art performance in publicly available language models and provide robust comparisons against regu-loGPT. Overall, ICL prompts powered by LLMs demonstrated remarkable superiority over non-LLM algorithms (Table I ). reguloGPT-CoT exhibited the most effective performance, achieving an impressive accuracy of 0.89 for context prediction and F1 scores of 0.96 for node prediction and 0.60 for edge prediction. The relatively lower performance on edge prediction underscores the inherent complexity in accurately extracting a graph. Compared to other LLMs, reguloGPT-CoT achieved higher F1 score than both Mixtral-CoT and Llama3-CoT, confirming the consensus on the superior ability of GPT-4 for text understanding and reasoning over other LLMs [28] .Though EIDOS showed the highest performance among baseline algorithms, reguloGPT-CoT achieved substantial improvements with 22% and 23% higher node and edge F1 scores than EIDOS, respectively. The marked increase in edge recall highlights the advantage of reguloGPT's end-to-end strategy for graph extraction over methods focusing on individual triplets separately. The improvement is evident in the title "The m 6 A methyltransferase METTL3 promotes osteosarcoma progression by regulating the m 6 A level of LEF1." The benchmark annotations include four triplets under the context of "osteosarcoma," however, REACH only identified 'METTL3 -[STIMULATES] -level of LEF1.' Similarly, EIDOS extracted only one triplet 'm 6 A methyltransferase METTL3 -[STIMULATES] -osteosarcoma progression.' In contrast, all three reguloGPT ICL prompts successfully extracted all four relations with the correct context. Among the three reguloGPT ICL prompts, the advanced CoT prompt produced the most aligned output with our requirements. For instance, although we asked GPT-4 to introduce a dummy node to capture the state of an entity in the prompt, the baseline prompt ignored this guideline. In contrast, by adding one example in a demonstration with a similar case, the few-shot prompt was able to follow the requirement, even though this alignment was not stable. For instance, in the title "Suppression of m 6 A reader Ythdf2 promotes hematopoietic stem cell expansion," the few-shot prompt neglected the condition, but the CoT prompt successfully maintained this alignment.We next investigated the G-Eval evaluations of the three reguloGPT prompts on the benchmark dataset and assessed the consistency between G-Eval evaluations and benchmark annotations. G-Eval produced scores for context evaluation and graph evaluations for each title. Examining the average score across the 400 titles (Table II ) revealed a consistent trend with the annotation-based evaluation in Table I , i.e., the CoT prompt exhibited the best performance, followed by the few-shot and baseline prompts. To further validate the effectiveness of our G-Eval strategy, we analyzed the similarities between the G-Eval scores and annotation evaluation. Since the annotation evaluation for each sentence is binary, i.e., correct or incorrect, we first binarized G-Eval scores based on the score distribution, converting a score to correct if it was greater than 3 and incorrect otherwise. Furthermore, since G-Eval conducts graph evaluation instead of evaluating nodes and edges separately as the annotation-based evaluation does, we also generated a graph-level annotation evaluation where a sentence was deemed correct if more than 50% of the edges in the sentence were correctly predicted. We did not consider node prediction because their F1 scores are high, as shown in Table I . To compare the similarity between the G-Eval and annotation evaluations, we computed the Rand matching coefficient for each title. The similarity results demonstrate high similarities between the two evaluations, especially for CoT, where the Rand similarities reach 0.84 for context prediction and 0.81 for graph prediction. These results suggest that G-Eval is a promising annotation-free method for evaluating reguloGPT. D. m 6 A-KG, a context-aware-KG of m 6 A functions Next, we demonstrated the utility of reguloGPT to create a KG of the m 6 A-associated MRPs. m 6 A is one of the most predominant mRNA modifications in mammalian cells, present in over 40% of transcripts [29] . The dynamic m 6 A regulation involves various RNA binding proteins (RPBs) including writers (METTL3 & METTL14), which add methyl groups, erasers (ALKBH5 & FTO) which remove it, and readers, (e.g., YTH proteins), which bind to m 6 A sites to decode the regulatory signals to mediate gene expression. It achieves this by regulating stability, alternative splicing, nuclear export, and translation efficiency of mRNA. Additionally, it significantly influences cancer development and progression by modulating mRNA stability and splicing [30] . Despite growing interest, the roles of m 6 A and its writers, erasers, and readers in cancer through gene expression alterations are not fully understood.1) Construction of m 6 A-KG with reguloGPT: We applied reguloGPT to 969 titles without annotations, resulting in the extraction of context-aware relational graphs that depict functions related to m 6 A in diverse contexts. After normalizing the nodes, edges, and contexts, we integrated these relation graphs and the annotated graphs in the benchmark dataset into a comprehensive m 6 A knowledge graph (m 6 A-KG), denoting molecular regulatory pathways linked to m 6 A. The constructed m 6 A-KG comprises 2442 nodes, 4678 links with 200 edge predicates, and 376 contexts where each link encompassing an average of 0.91 contexts. The average node degree, calculated by aggregating incoming and outgoing edges, is 2.15 with 96% of nodes having less than five degrees and only eight nodes possessing more than 20 degrees. As expected, the m 6 A node emerged as the most connected, with a degree of 65, highlighting its centrality in the network. The top nodes by degree also include key m 6 A writers like METTL3 (degree 47) and METTL14 (degree 23), erasers such as ALKBH5 (degree 24) and FTO (degree 38), and readers like YTHDF1 (degree 28) and YTHDF2 (degree 22). These high degree nodes underscore their vital roles in the regulatory functions of m 6 A. Additionally, nodes like APOPTOSIS (degree 13), GLYCOLYSIS (degree 11), and METASTASIS (degree 10) also exhibited relatively high degrees, indicating m 6 A's associations with these tumor-related entities.2) The structure of m 6 A-KG reflects the architecture of molecular regulatory pathways.: Out-degree rates across each node category revealed a hierarchical structure aligned with that of a molecular pathway (Fig. 3A ). Specifically, m 6 A-WER and m 6 A have median out-degree rates of 0.72 and 0.52, respectively, suggesting that they occupy upstream positions and reaffirming their roles as key regulators. Gene Protein and GO Pathway, with a median out-degree rate of 0.5, serve as intermediate nodes bridging the upstream regulators with the downstream nodes. Interestingly, Other nodes exhibited a median out-degree rate of 0.00, indicating their positions at the extreme end of the pathway. Close inspection revealed that nodes with high out-degree rates include chemicals or environmental stimuli like simvastatin, propofol, hypoxia, hepatic microenvironment, and cigarette smoking, which are expected to be upstream in pathways (Fig. 3B ). Conversely, nodes with high in-degree rates were disease phenotypes or outcomes such as tumorigenesis, tumor growth, cancer progression, radiosensitivity, drug resistance/sensitivity, chemoresistance, or autophagic cell death, which naturally sit at the bottom of pathways (Fig. 3B ). The emergent structure of the m 6 A-KG, featuring various stimuli at the top followed by clear upstream m 6 A regulators and downstream phenotype outcomes, exhibits the hallmarks of an MRP.3) The m 6 A-KG reveals distinct mechanisms of m To gain insights into cancer-specific m 6 A-mediated functions, we extracted cancer-specific sub-KGs for breast cancer, leukemia, and lung cancer (Fig. 3C, D, and E ). These sub-KGs presented clear hierarchies of MRPs, with m 6 A regulators at the top and disease phenotype nodes downstream. METTL3's widespread association across cancers prompted further examination of pathways centering on this regulator. We focused on pathways supported by edges spanning multiple titles because they could reveal novel functions. The breast cancer sub-KG (Fig. 3C ) delineates a complex dual-pathway mechanism, with evidence from five titles [26] , [31] - [34] , suggesting METTL3's involvement in tumor metastasis through two distinct routes: regulation of COL3A1, crucial for extracellular matrix structure, and alteration of cancer cell metabolism via the glycolytic pathway. This duality suggests that therapeutic targeting METTL3 could simultaneously disrupt key structural and metabolic routes essential to cancer metastasis, offering a promising avenue for multifaceted therapeutic intervention. Moreover, cancer-dependent regulations of MEG3, a tumor suppressor gene, were revealed in lung and leukemia sub-KGs. The leukemia sub-KG (Fig. 3D ) suggests that MEG3 modulates miR-493-5p to suppress myeloid leukemia by inhibiting METTL3-mediated m 6 A methylation [35] , [36] . Conversely, in lung cancer sub-KG (Fig. 3E ), METTL3 methylates MEG3, which facilitates tumorigenesis and metastasis [37] . These distinct regulatory mechanisms were corroborated through a detailed examination of the literature associated with the extracted pathways [38] , validating the m 6 A-KG's utility in uncovering new functions.In this study, we developed reguloGPT, a novel GPT-4 application for the end-to-end KG construction for MRPs. We devised ICL prompting to extract context-aware relational graphs for MRPs. We annotated 400 titles on m 6 A methylation, encompassing various regulations applicable to broader studies. We benchmarked reguloGPT, showing significant performance improvements over existing algorithms, including Mixtral-8x22B and Llama3-70B. Also, we found a strong similarity between manual evaluation and our proposed annotation-free G-Eval evaluation. reguloGPT successfully built a detailed m 6 A-KG with 2442 nodes, 4678 links with 200 predicates, providing a rich map of m 6 A regulatory functions by featuring its unique context-aware edges with associated contexts and PubMed IDs. This design not only allowed us to understand context-specific regulations but also enhances traceability and data verification. The m 6 A-KG revealed distinct mechanisms of m 6 A functions across various cancer types, facilitating a deeper understanding of m 6 A's role in cancer and opening avenues for targeted cancer research and therapy development. The hierarchical structure of the m 6 A-KG mirrors the architecture of MRPs, offering a more intuitive understanding of the complex interactions and roles within these pathways. Future studies will explore a more systematic G-Eval assessment and relationship extraction, along with improved normalization schemes for edges and contexts. A systematic and effective approach to elucidate novel regulatory functions from the KG will be further developed. 
 
 Appendix:  



Reviews: review: title: title: reguloGPT: Harnessing GPT for End-to-End Knowledge Graph Construction of Molecular Regulatory Pathways
paper_summary: paper_summary: This paper proposes the use of ChatGPT to extract knowledge graphs about molecular regulatory pathways. To do that, the authors first collected a dataset of paper titles containing regulatory pathways for m6A. Some of this titles were used by experts to generate knowledge graphs that are then used to compare with the results provided by ChatGPT. Additionally, the chatGPT was also used to automatically evaluate the generated knowledge graphs and compared with the other LLMs
strength_weakness: strength_weakness: The text on most figures is too small to be read
There is a lot of works using ChatGPT, comparison or at least discussion about related or similar works is needed
questions: questions: Why the four established algorithms were selected for baselines? The criteria used for the selection should be discussed
How were the known problems of LLMs (hallucinations, etc) addressed? Or at least, how do they impact the results? This should also be discussed
There is a huge amount of works using LLMs, some disuussion about this is needed.
title: title: Reviews of Submission361
paper_summary: paper_summary: This paper introduces an approach leveraging GPT-4 for the automated construction of Knowledge Graphs (KGs) for Molecular Regulatory Pathways (MRPs). The tool, reguloGPT, uses in-context learning (ICL) to generate context-aware relational graphs from textual descriptions of MRPs, particularly focusing on m6A mRNA methylation. This approach addresses the limitations of current methods in capturing complex, hierarchical relationships and contextual nuances within MRPs. The paper details the creation of a benchmark dataset from 400 annotated PubMed titles and evaluates reguloGPT's performance in extracting accurate and contextually relevant information compared to existing algorithms and other large language models (LLMs). The study also explores the application of reguloGPT to a larger dataset, resulting in a comprehensive m6A-KG that elucidates m6A's regulatory roles across various cancers.
strength_weakness: strength_weakness: 1. The effectiveness of reguloGPT is likely contingent on the quality and specificity of the input data. The paper does not sufficiently explore how data quality or variations in textual descriptions might influence the output KGs. Additional ablation studies on this aspect would be helpful.
2. Could you provide further details on the design of the prompt?
3. The experimental design as presented is deterministic. It may be advantageous to adopt a probabilistic approach, such as updating experimental designs based on specific criteria using Bayesian methods or incorporating active learning to expedite the experimentation process.
questions: questions: See the weakness.
title: title: ReguloGPT and Its Application in Constructing Knowledge Graphs for Targeted Cancer Research
paper_summary: paper_summary: The authors developed reguloGPT, a novel GPT-4 application that implements instruction-following conversational (ICL) prompting to generate context-specific relational graphs detailing MRP interactions. They built a benchmark dataset of 400 m6A methylation titles and used it to evaluate and prove the efficacy of the proposed reguloGPT. The authors also proposed an annotation-free evaluation method, G-Eval, and demonstrated its consistency with benchmark evaluations. Finally, they successfully applied reguloGPT to create a comprehensive and detailed knowledge graph, m6A-KG, which maps diverse regulatory mechanisms across cancer types, enhancing our understanding of m6A's role in cancer and supporting targeted research and therapy development.
strength_weakness: strength_weakness: The prompts are based on GPT-4 and other cutting-edge large language models (LLMs), which, although state-of-the-art at the time of drafting, may quickly become outdated due to rapid model iterations.
questions: questions: What is the significance of mining knowledge graphs from biomedical literature? Could you elaborate on this? 
I noticed some relevant explanations in Result.D: 'm6A-KG, a context-aware KG of m6A functions.' Please consider providing a more detailed summary and highlighting these aspects in the introduction