Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notes
Authors: Vahan Arsenyan, Spartak Bughdaryan, Fadi S Shaya, Kent Wilson Small, Davit Shahnazaryan


Reviewer paper summaries: This paper presents an end-to-end machine learning solution utilizing large language models (LLMs) for constructing knowledge graphs (KGs) from electronic medical record (EMR) notes. The study evaluates the performance and safety of 12 different LLMs, comparing encoder-only, encoder-decoder, and decoder-only models. The results show that encoder-decoder models, particularly with guided prompt design, outperform others in extracting clinical relations needed for KG construction. The study demonstrates the method's application on age-related macular degeneration, highlighting the potential of LLMs to automate and enhance medical knowledge discovery.

This paper proposes a method for constructing biomedical knowledge graphs (KGs) from electronic medical record (EMR) notes using large language models (LLMs). The automatic construction of KGs has far-reaching applications in medicine, including drug discovery and clinical trial design. The paper focuses on extracting information from EMR notes to identify interactions between medical entities, such as diseases, factors, treatments, and manifestations. The authors experiment with different LLMs, including encoder-only, decoder-only, and encoder-decoder architectures, and evaluate their performance and suitability for clinical relation extraction. They introduce a guided prompt design to improve the performance of decoder-only LLMs and analyse the challenges and limitations of different models. The paper concludes with a case study on age-related macular degeneration (AMD) as a qualitative example.
Summary of strengths: `Comprehensive evaluation`: The study conducts an extensive evaluation of 12 LLMs, encompassing various architectures, including encoder-only, decoder-only, and encoder-decoder models. This thorough comparison provides valuable insights into the strengths and weaknesses of each model type in the context of clinical relation extraction. The evaluation criteria include both quantitative metrics and qualitative assessments, such as the ability to generate structured outputs and the tendency to hallucinate. By manually annotating a dataset and conducting rigorous benchmarking, the authors highlight which models are most effective and safe for use in medical applications. 

`Practical demonstration`: The practical demonstration of the proposed methodology on age-related macular degeneration (AMD) showcases the real-world applicability and utility of the approach. By constructing a KG for AMD, the study illustrates how LLMs can extract and organize valuable clinical information from EMR notes.

The paper addresses an important and challenging task in the biomedical domain by utilising LLMs for knowledge graph construction from EMR notes. The comprehensive evaluation of 12 LLMs with varying architectures contributes to the understanding of their performance and suitability for clinical relation extraction. The paper highlights the potential of encoder-decoder models and the effectiveness of the guided prompt design for decoder-only LLMs. The case study on AMD provides a practical example of the application of the proposed method.
Summary of weaknesses: `Lack of in-context learning novelty`: The guided prompt design lacks novelty for ACL proceedings.
The paper's main contribution is the benchmarking of KGs and the comprehensive evaluation of different LLMs, rather than introducing new models. The focus on implementation details makes it more suitable for a system demo or industrial track paper.

`Insufficient details on KG statistics`: The paper does not provide detailed statistics of the final KG, such as the number of nodes and edges, or information on how the KG is stored and maintained.

`Unclear safety measures`: The paper claims to address safety but it’s unclear to me how this is achieved within the framework. Using open-source LLMs instead of commercial ones? 

`Writing and presentation improvements`: The writing and presentation could be more concise and clear. There are some typos and formatting errors. Some statements are vague or overly complex, i.e., ``We propose providing larger context information than that which is solely present in the prompt."

A significant concern is the authors' choice not to use publicly available datasets. They opted for a private dataset to avoid potential biases from models trained on common public data. However, this decision impacts reproducibility and transparency, making it difficult for other researchers to validate or compare the findings. While their intent was to ensure unbiased results, this approach limits the ability of the broader research community to replicate the experiments. Providing a clearer justification for this choice and discussing its specific implications would strengthen the paper.
Comments, suggestions and typos: `Suggestions`:

The caption for Table 1 would be better placed below the table.

`Typos`:

Use \citet instead of \cite in line 70, 146, ect. Use \citet when you want to mention the author’s name as part of the sentence without parentheses. 

Line 78: use `’ instead of ‘’

Line 117: a -> an

Missing period . in line 119, 123, 127

Line 182: large language models -> LLMs (be consistent with abbreviations)

* 240: "[...] 'a' set of questions"
Overall assessment scores: 3

2.5