# Improving Medical X-ray Report Generation by Using Knowledge Graph
## 3 Our Approach
### Classifier
#### 3.1.3 Text Encoder and Disease Contextualized Representation  
Let \(T\) be a text document of length \(l\) with word embedding \(\{\omega_{1},\omega_{2},\dots,\omega_{l}\}\), where \(\omega_{i}\in\mathbb{R}^{c}\) denotes the \(i\)-th word in the text and \(e\) is the embedding dimension. We use the Transformer encoder as our text feature extractor to extract a set of hidden states \(H=\{\mathsf{h}_{1},\mathsf{h}_{2},\dots,\mathsf{h}_{l}\}\), where \(\mathsf{h}_{i}\in\mathbb{R}^{c}\) denotes the attention feature of the \(i\)-th word to other words in the text. \[\mathsf{h}_{i}\in Encoder(\omega_{i}|\omega_{1},\omega_{2},\dots,\omega_{l}) \tag{1}\]  
\(n\) disease-related topics are queried from the whole document \(T\), which is summarized as \(Q=\left\{\mathsf{q}_{1},\mathsf{q}_{2},\dots,\mathsf{q}_{n}\right\}\). We refer to this retrieval process as text summarization embedding \(\mathsf{D}_{txt}\in\mathbb{R}^{n\times c}\):  
\[\mathsf{D}_{txt}=Softmax\Big{(}QH^{T}\Big{)}H \tag{2}\]  
Figure 2: Construction of the knowledge graph. The knowledge graph is built based on the co-occurrence of the tags in the radiology report, and only the tag names of some nodes are shown. In the figure, part (**a**) represents all node names in the knowledge graph; part (**b**) represents the knowledge graph, where the green nodes represent the 14 common disease labels defined by CheXpert, and the blue nodes represent the additional disease labels defined by us. \(\mathfrak{q}_{i}\in\mathbb{R}^{e}\) is randomly initialized and updated by the attention mechanism. \(Softmax\big{(}QH^{T}\big{)}\) represents the word-attention heat-map of \(n\) query diseases in the document. As shown in Figure 1, we decouple the multi-view latent features \(\mathcal{X}\in\mathbb{R}^{e}\) extracted by the image encoder into a low-dimensional disease representation \(\text{D}_{img}\in\mathbb{R}^{n\times e}\), where each row is a vector \(\varphi_{j}(\mathbf{x})\in\mathbb{R}^{e},j=1,2,\ldots,n\). \(\varphi_{j}(\mathbf{x})\) is defined as follows:  
\[\varphi_{j}(\mathbf{x})=A_{j}^{T}\mathbf{x}+b_{j} \tag{3}\]  
where \(A_{j}\in\mathbb{R}^{e\times e}\) and \(b_{j}\in\mathbb{R}^{e}\) are trainable parameters for the \(j\)-th class of disease representation. \(n\) denotes the number of disease representations and \(e\) is the dimension of the embedding. Then, the visual embedding \(\text{D}_{img}\) and the text summary embedding \(\text{D}_{txt}\) are twisted together to form the disease situational representation \(\text{D}_{fused}\in\mathbb{R}^{n\times e}\):  
\[\text{D}_{fused}=LayerNorm\big{(}\text{D}_{img}+\text{D}_{txt}\big{)} \tag{4}\]  
The fusion of visual and textual information allows our model to simulate the workflow of a hospital to screen the visual manifestations of a disease based on a patient's clinical history. 1.4 Graph Convolutional Networks and Contextualized Representations of Diseases with Prior Knowledge  
We use the GCN to model the intrinsic association between diseases or topics, and the adjacency matrix is built based on the knowledge graph detailed above. The GCN updates its node representation via message passing, and graph convolution is represented as [37]:  
\[\hat{H}^{l}=ReLU\Big{(}BN\Big{(}ConvId\Big{(}H^{l}\Big{)}\Big{)}\Big{)} \tag{5}\]  
\[m=ReLU\Big{(}D^{-1/2}\hat{A}D^{-1/2}H^{l}W^{l}\Big{)} \tag{6}\]  
\[H^{l+1}=ReLU(BN(ConvId(concat(\hat{H}^{l},m)))) \tag{7}\]  
where \(H^{l}\) is the state in layer \(l\) and \(H^{0}\) is initialized using the disease contextualized representation. \(\hat{A}=A+I_{N}\) is the adjacency matrix with self-connection, \(A\) is the adjacency matrix of the knowledge graph, \(I_{N}\) is the identity matrix of order \(N\), \(D=\text{diag}\sum_{j}A_{ij}\) is the degree matrix of the graph, \(BN\) is the batch normalization, and \(W^{l}\) is a trainable layer-specific weight matrix. We extract \(\text{D}_{fused}\) to the semantic information between nodes by the GCN, and obtain the disease situational representation \(\text{D}_{pri\text{-}fused}\in\mathbb{R}^{n\times e}\) with prior knowledge.