# Improving Medical X-ray Report Generation by Using Knowledge Graph
## 1 Introduction
However, their prior knowledge is manually predefined and, therefore, requires domain experts to be closely involved in the design and implementation of the system, which is a waste of time and effort. Due to the nature of the graph, their method can usually achieve high accuracy, but may miss some important findings. While it is feasible to manually identify and implement a high-quality knowledge graph to obtain good accuracy, it is often impractical to exhaustively encode all nodes and relationships in this way. Our work uses prior knowledge from text mining to build a generic knowledge graph to alleviate some of these concerns. We propose an innovative framework for automatic report generation based on prior knowledge, which seamlessly integrates prior and linguistic knowledge at different levels. First, we investigate a data-driven approach to automatically obtain associations between disease labels in radiology reports. This prior knowledge is a natural extension of human-designed knowledge. Disease labels are defined as nodes in the knowledge graph, which are related and influence each other during the propagation of the graph. Secondly, we establish a graph convolutional neural network to aggregate and transmit information between each node to obtain prior knowledge [7]. Specifically, a set of multi-view chest X-ray images are sent to the convolutional neural network for image feature extraction, and then the content of the clinical document instructed by the doctor is used for text feature extraction using Transformer [8]. The two extracted features are summed and normalized to wound together to obtain a hybrid image-text feature, called a contextualized embedding. The image-text hybrid features and the adjacency matrix constructed according to the knowledge graph are transferred to the three-layer graph convolutional network, and the special features of each knowledge graph node are learned to obtain the episodic representation with prior knowledge. Then, these node features are transferred to two branches, a linear classifier for disease classification, and a generator made of a transformer to generate reports. After generating the report, the generated report is passed into the text classifier again to fine-tune the generated report. Unlike previous studies, additional text mining concepts are added to the model as labels for classification as well as nodes in the knowledge graph, and the expressive power of the model is enriched by training on the chest X-ray image dataset with structured labels. We evaluate our proposed method on the publicly accessible Open-I [9] and MIMIC-CXR [10] datasets, where we employ Natural Language Generation (NLG) and clinical efficacy (CE) metrics to analyze the quality of clinically generated reports. The results show that the proposed method achieves good performance in both natural language generation and clinical efficacy indicators. It is also shown that the addition of prior knowledge helps to improve the quality and accuracy of the automatic generation of radiology reports. Our contributions are outlined below:  
1. We mine and model the text, and according to the mined information, we use prior knowledge to build the knowledge graph and construct the corresponding adjacency matrix (Section 3.1.1). 2. We combine text-image hybrid features with knowledge reasoning based on the knowledge graph to improve the quality and accuracy of radiology report generation (Section 3.1.4). 3. Our experiments show that our proposed model outperforms state-of-the-art methods. The knowledge graph composed of prior knowledge of patients plays a crucial role in improving the quality of generated reports.