Key Verbatim Extraction from Clinical Notes: A Hierarchical Multimodal Cross-Attention Approach
Authors: Feiyu E, Hanwen Shi, Kunpeng Zhang, Jin Zhang, Geoffrey Kim


Reviewer paper summaries: The paper introduces a knowledge-enhanced hierarchical multimodal cross-attention to integrate textual knowledge with the patient network to synthesize information across multiple data levels.
The proposed model is trained to generate these sentence embeddings and select key sentences in clinical notes.

The paper introduces a method for extracting key sentences from clinical notes, focusing on oncology. The proposed approach uses a knowledge-enhanced hierarchical multimodal cross-attention mechanism that combines textual knowledge from ClinicalBERT with patient network knowledge using Graph Attention Transformers (GATs). This method synthesizes information across multiple data levels (word, sentence, note, and patient) to efficiently highlight key sentences in clinical notes. Validated on a large real-world dataset, the model significantly outperforms baseline models, demonstrating improvements in F1 score and accuracy.
Summary of strengths: The proposed sentence embedding method is a novel approach that incorporates a knowledge base from the patient network into the construction of within and between sentence embeddings.
Experimental results show the superiority of the proposed method over the baselines.

`Useful methodology`: The paper introduces a hierarchical multimodal cross-attention approach that combines textual knowledge from ClinicalBERT with patient network knowledge using Graph Attention Transformers (GATs). This innovative method effectively integrates multiple data levels, including word, sentence, note, and patient levels, to enhance the extraction of key sentences from clinical notes. The proposed model demonstrates significant performance improvements over existing baseline models. The results show an increase of up to 4.17% in F1 score and 2.79% in accuracy.

`Well-organized presentation`: The paper is well-organized and clearly presents the methodology and experimental results.
Summary of weaknesses: Because the dataset used for experiments is closed, it is difficult to reproduce the results.
The target domain is focused on the biomedical, and the generalizability is limited.

`Comparison with baseline methods`: The paper compares the proposed model with LMs like BART and T5, and LLMs like GPT-3.5 and GPT-4. However, the detailed task settings (input, encoder, hyperparameters) are not discussed, making it difficult to ensure a fair comparison. The implementation details can be added in the Appendix. Also, the limitations of LLMs mentioned in lines 205-209 are not convincing to the audience. Domain knowledge can be incorporated using RAG, hallucinations can be controlled with in-context learning, and many LLMs now support longer context windows than traditional LMs. 

`Lack of discussion on EHR extraction work`: Section 2.2 mentions classic methods (cTAKES, CNN, LSTM) for information extraction (IE) from EHRs but lacks discussion on state-of-the-art frameworks that use attention mechanisms and Graph Neural Networks (GNNs) for IE tasks in clinical notes.  Including a discussion of recent advancements and how they relate to or differ from the proposed approach would provide a more comprehensive background of this work.
Comments, suggestions and typos: (A) LL421-426 and LL429-434 are duplicated text.
(B) The dimension information is required for each vector and matrix defined in section 3.
(C) Is the identity of each patient preserved when constructing patient embedding P?
(D) If the patient's identity is preserved, how can the proposed method be used for unknown patients whose reports are not included in the training?
(E) In the cross-attention module in section 3.4,  shouldn't masking of patient embedding be needed because each patient is relevant to some but not all sentences?

What’s your reproduction policy? Will you release code and dataset?

The title refers to "key verbatim extraction," and line 56 defines key verbatim as the extraction of exact words, phrases, or sentences from a longer text. However, the Methodology section focuses on the extraction of key sentences from clinical notes. This inconsistency between the title and the actual task needs clarification to avoid confusion.

Why only patient embeddings are used for the model? There are other node embeddings like oncology and medication trained on GAT that could be useful for key sentence extraction.

Figure 1 can be embedded within the text column width. 

According to Line 390-394, the patient embedding is not directly fed into the FFNN layer, while the graph indicates patient embedding is used for FFNN.

The model name GEHE first appears at line 450 in the experiment section, proposed it early in introduction to keep the consistency.
Overall assessment scores: 2.5

3