Introduction
The availability of massive electronic health records (EHR) data and the advances of deep learning technologies have provided unprecedented resource and opportunity for predictive healthcare, including the computational medication recommendation task. A number of deep learning models were proposed to assist doctors in making medication recommendation BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . They often learn representations for medical entities (e.g., patients, diagnosis, medications) from patient EHR data, and then use the learned representations to predict medications that are suited to the patient's health condition.
To provide effective medication recommendation, it is important to learn accurate representation of medical codes. Despite that various considerations were handled in previous works for improving medical code representations BIBREF4 , BIBREF2 , BIBREF3 , there are two limitations with the existing work:
To mitigate the aforementioned limitations, we propose G-BERT that combines the pre-training techniques and graph neural networks for better medical code representation and medication recommendation. G-BERT is enabled and demonstrated by the following technical contributions:
Related Work
Medication Recommendation Medication Recommendation can be categorized into instance-based and longitudinal recommendation methods BIBREF1 . Instance-based methods focus on current health conditions. Among them, Leap BIBREF9 formulates a multi-instance multi-label learning framework and proposes a variant of sequence-to-sequence model based on content-attention mechanism to predict combination of medicines given patient's diagnoses. Longitudinal-based methods leverage the temporal dependencies among clinical events, see BIBREF10 , BIBREF11 , BIBREF12 . Among them, RETAIN BIBREF10 uses a two-level neural attention model to detect influential past visits and significant clinical variables within those visits for improved medication recommendation.
Pre-training Techniques The goal of pre-training techniques is to provide model training with good initializations. Pre-training has been shown extremely effective in various areas such as image classification BIBREF13 , BIBREF14 and machine translation BIBREF15 . The unsupervised pre-training can be considered as a regularizer that supports better generalization from the training dataset BIBREF16 . Recently, language model pre-training techniques such as BIBREF5 , BIBREF6 , BIBREF7 have shown to largely improve the performance on multiple NLP tasks. As the most widely used one, BERT BIBREF7 builds on the Transformer BIBREF8 architecture and improves the pre-training using a masked language model for bidirectional representation. In this paper, we adapt the framework of BERT and pre-train our model on each visit of the EHR data to leverage the single-visit data that were not fit for model training in other medication recommendation models.
Graph Neural Networks (GNN) GNNs are neural networks that learn node or graph representations from graph-structured data. Various graph neural networks have been proposed to encode the graph-structure information, including graph convolutional neural networks (GCN) BIBREF17 , message passing networks (MPNN) BIBREF18 , graph attention networks (GAT) BIBREF19 . GNNs have already been demonstrated useful on EHR modeling BIBREF20 , BIBREF1 . GRAM BIBREF20 represented a medical concept as a combination of its ancestors in the medical ontology using an attention mechanism. It's different from G-BERT from two aspects as described in Section "Input Representation" . Another work worth mentioning is GAMENet BIBREF1 , which also used graph neural network to assist the medication recommendation task. However, GAMENet has a different motivation which results in using graph neural networks on drug-drug-interaction graphs instead of medical ontology.
Problem Formalization
Definition 1 (Longitudinal Patient Records) In longitudinal EHR data, each patient can be represented as a sequence of multivariate observations: $ \mathcal {X}^{(n)} = \lbrace  \mathcal {X}_1^{(n)}, \mathcal {X}_2^{(n)}, \cdots , \mathcal {X}_{T^{(n)}}^{(n)} \rbrace  $ where $n\in \lbrace 1,2,\ldots , N\rbrace $ , $N$ is the total number of patients; $T^{(n)}$ is the number of visits of the $n^{th}$ patient. Here we choose two main medical code to represent each visit $\mathcal {X}_t = \mathcal {C}_d^t \cup \mathcal {C}_m^t$ of a patient which is a union set of corresponding diagnoses codes $\mathcal {C}_d^t \subset \mathcal {C}_d$ and medications codes $\mathcal {C}_m^t \subset \mathcal {C}_m$ . For simplicity, we use $\mathcal {C}_\ast ^t$ to indicate the unified definition for different type of medical codes and drop the superscript $(n)$ for a single patient whenever it is unambiguous. $n\in \lbrace 1,2,\ldots , N\rbrace $0 denotes the medical code set and $n\in \lbrace 1,2,\ldots , N\rbrace $1 the size of the code set. $n\in \lbrace 1,2,\ldots , N\rbrace $2 is the medical code.
Definition 2 (Medical Ontology) Medical codes are usually categorized according to a tree-structured classification system such as ICD-9 ontoloy for diagnosis and ATC ontology for medication. We use $\mathcal {O}_d, \mathcal {O}_m$ to denote the ontology for diagnosis and medication. Similarly, we use $\mathcal {O}_\ast $ to indicate the unified definition for different type of medical codes. In detial, $\mathcal {O}_\ast = \overline{\mathcal {C}_\ast } \cup \mathcal {C}_\ast $ where $\overline{\mathcal {C}_\ast }$ denotes the codes excluding leaf codes. For simplicity, we define two function $pa(\cdot ), ch(\cdot )$ which accept target medical code and return ancestors' code set and direct child code set.
Problem Definition (Medication Recommendation) Given diagnosis codes $\mathcal {C}_d^t$ of the visit at time $t$ , patient history $\mathcal {X}_{1:t} = \lbrace \mathcal {X}_1, \mathcal {X}_2, \cdots , \mathcal {X}_{t-1}\rbrace $ , we want to recommend multiple medications by generating multi-label output $\hat{\mathbf {y}}_t \in \lbrace 0,1\rbrace ^{|\mathcal {C}_m|}$ .
Method
The overall framework of G-BERT is described in Figure 2 . G-BERT first derives the initial embedding of medical codes from medical ontology using graph neural networks. Then, in order to fully utilize the rich EHR data, G-BERT constructs an adaptive BERT model on the discarded single-visit data for visit representation. Finally we add a prediction layer and fine-tune the model in the medication recommendation task. In the following we will describe G-BERT in detail. But firstly, we give a brief background of BERT especially for the two pre-training objectives which will be later adapted to EHR data in Section "Pre-training" .
Background of BERT Based on a multi-layer Transformer encoder BIBREF8 (The transformer architecture has been ubiquitously used in many sequence modeling tasks recently, so we will not introduce the details here), BERT is pre-trained using two unsupervised tasks:
A typical input to BERT is as follows ( BIBREF7 ):
Input = [CLS] the man went to [MASK] store [SEP] he bought a gallon [MASK] milk [SEP]
Label = IsNext
where [CLS] is the first token of each sentence pair to represent the special classification embedding, i.e. the final state of this token is used as the aggregated sequence representation for classification tasks; [SEP] is used to separate two sentences; [MASK] is used to mask out the predicted words in the masked language model. Using this form, these inputs facilitate the two tasks described above, and they will also be used in our method description in the following section.
Input Representation
The G-BERT model takes medical codes' ontology embeddings as input, and obtains intermediate representations from a Transformer encoder as the visit embeddings. It is then pre-trained on EHR from patients who only have one hospital visit. The derived encoder and visit embedding will be fed into a classifier and fine-tuned to make predictions.
Ontology Embedding We constructed ontology embedding from diagnosis ontology $\mathcal {O}_d$ and medication ontology $\mathcal {O}_m$ . Since the medical codes in raw EHR data can be considered as leaf nodes in these ontology trees, we can enhance the medical code embedding using graph neural networks (GNNs) to integrate the ancestors' information of these codes. Here we perform a two-stage procedure with a specially designed GNN for ontology embedding.
To start, we assign an initial embedding vector to every medical code $c_\ast \in \mathcal {O}_\ast $ with a learnable embedding matrix $\mathbf {W}_e \in \mathbb {R}^{|\mathcal {O}_\ast | \times d}$ where $d$ is the embedding dimension.
Stage 1. For each non-leaf node $c_\ast \in \overline{\mathcal {C}_\ast }$ , we obtain its enhanced medical embedding $\mathbf {h}_{c_\ast } \in \mathbb {R}^{d}$ as follows:
$$\mathbf {h}_{c_\ast } = g(c_\ast , ch(c_\ast ), \mathbf {W}_e)$$   (Eq. 11)
where $g(\cdot ,\cdot ,\cdot )$ is an aggregation function which accepts the target medical code $c_\ast $ , its direct child codes $ch(c_\ast )$ and initial embedding matrix. Intuitively, the aggregation function can pass and fuse information in target node from its direct children which result in the more related embedding of ancestor' code to child codes' embedding.
Stage 2. After obtaining enhanced embeddings, we pass the enhance embedding matrix $\mathbf {H}_e \in \mathbb {R}^{|\mathcal {O}_\ast | \times d}$ back to get ontology embedding for leaf codes $c_\ast \in \mathcal {C}_\ast $ as follows:
$$\mathbf {o}_{c_\ast } = g(c_\ast , pa(c_\ast ), \mathbf {H}_e)$$   (Eq. 12)
where $g(\cdot ,\cdot ,\cdot )$ accepts ancestor codes of target medical code $c_\ast $ . Here, we use $pa(c_\ast )$ instead of $ch(c_\ast )$ , since utilizing the ancestors' embedding can indirectly associate all medical codes instead of taking each leaf code as independent input.
The option for the aggregation function $g(\cdot ,\cdot ,\cdot )$ is flexible, including sum, mean. Here we choose the one from graph attention networks (GAT) BIBREF19 , which has shown efficient embedding learning ability on graph-structured tasks, e.g., node classification and link prediction. In particular, we implement the aggregation function $g(\cdot ,\cdot ,\cdot )$ as follows:
$$g (c_\ast , p(c_\ast ), \mathbf {H}_e) = \operatornamewithlimits{\scalebox {1}[1.5]{\parallel }}_{k=1}^K \sigma \left( \sum _{j \in \lbrace c_\ast \rbrace  \cup pa(c_\ast )} \alpha _{i,j}^k \mathbf {W}^k \mathbf {h}_j \right)$$   (Eq. 13)
where $\operatornamewithlimits{\scalebox {1}[1.5]{\parallel }}$ represents concatenation which enables the multi-head attention mechanism, $\sigma $ is a nonlinear activation function, $\mathbf {W}^k \in \mathbb {R}^{m \times d}$ is the weight matrix for input transformation, and $\alpha _{i,j}^k$ are the corresponding $k$ -th normalized attention coefficients computed as follows:
$$\alpha _{i,j}^k = \frac{e^{\left(\text{LeakyReLU}(\mathbf {a}^\intercal [\mathbf {W}^k \mathbf {h}_i || \mathbf {W}^k \mathbf {h}_j])\right)}}{\sum _{k \in \mathcal {N}_i} e^{\left(\text{LeakyReLU}(\mathbf {a}^\intercal [\mathbf {W}^k \mathbf {h}_i || \mathbf {W}^k \mathbf {h}_k])\right)}}$$   (Eq. 14)
where $\mathbf {a} \in \mathbb {R}^{2m}$ is a learnable weight vector and LeakyReLU is a nonlinear function. (we assume $m = d/K$ ).
As shown in Figure 2 , we construct ICD-9 tree for diagnosis and ATC tree for medication using the same structure. Here the direction of arrow shows the information flow where ancestor nodes can get information from their direct children (in stage 1) and similarly leaf nodes can get information from their connected ancestors (in stage 2).
It is worth mentioning that our graph embedding method on medical ontology is different from GRAM BIBREF20 from the following two aspects:
Initialization: we initialize all the node embeddings from a learnable embedding matrix, while GRAM learns them using Glove from the co-occurrence information.
Updating: we develop a two-step updating function for both leaf nodes and ancestor nodes; while in GRAM, only the leaf nodes are updated (as a combination of their ancestor nodes and themselves).
Visit Embedding Similar to BERT, we use a multi-layer Transformer architecture BIBREF8 as our visit encoder. The model takes the ontology embedding as input and derive visit embedding $\mathbf {v}_\ast ^t \in \mathbb {R}^d$ for a patient at $t$ -th visit:
$$\mathbf {v}_\ast ^t = \mathrm {Transformer}(\lbrace \textit {[CLS]}\rbrace  \cup \lbrace \mathbf {o}_{c_\ast }^t| c_\ast \in \mathcal {C}_\ast ^t\rbrace )[0]$$   (Eq. 17)
where [CLS] is a special token as in BERT. It is put in the first position of each visit of type $\ast $ and its final state can be used as the representation of the visit. Intuitively, it is more reasonable to use Transformers as encoders (multi-head attention based architecture) than RNN or mean/sum to aggregate multiple medical embedding for visit embedding since the set of medical codes within one visit is not ordered.
It is worth noting that our Transformer encoder is different from the original one in the position embedding part. Position embedding, as an important component in Transformers and BERT, is used to encode the position and order information of each token in a sequence. However, one big difference between language sentences and EHR sequences is that the medical codes within the same visit do not generally have an order, so we remove the position embedding in our model.
Pre-training
We adapted the original BERT model to be more suitable for our data and task. In particular, we pre-train the model on each EHR visit (within both single-visit EHR sequences and multi-visit EHR sequences). We modified the input and pre-training objectives of the BERT model: (1) For the input, we built the Transformer encoder on the GNN outputs, i.e. ontology embeddings, for visit embedding. For the original EHR sequence, it means essentially we combine the GNN model with a Transformer to become a new integrated encoder. In addition, we removed the position embedding as we explained before. (2) As for the pre-training procedures, we modified the original pre-training tasks i.e., Masked LM (language model) task and Next Sentence prediction task to self-prediction task and dual-prediction task. The idea to conduct these tasks is to make the visit embedding absorb enough information about what it is made of and what it is able to predict.
Thus, for the self-prediction task, we want the visit embedding $v^1_\ast $ to recover what it is made of, i.e., the input medical codes $\mathcal {C}_\ast ^t$ for each visit as follows:
$$\begin{aligned} &\mathcal {L}_{se}(\mathbf {v}^1_\ast , \mathcal {C}_\ast ^1) = -\log p(\mathcal {C}_\ast ^1 | \mathbf {v}^1_\ast ) \\ & = - \sum _{c \in \mathcal {C}_\ast ^1}\log p(c_\ast ^1 | \mathbf {v}^1_\ast ) + \sum _{c \in \mathcal {C}_\ast \setminus \mathcal {C}_\ast ^1}\log p(c_\ast ^1 | \mathbf {v}^1_\ast ) \end{aligned}$$   (Eq. 19)
we minimize the binary cross entropy loss $\mathcal {L}_s$ , and in practise, $\text{Sigmoid}(f(\mathbf {v}_\ast ))$ should be transformed by applying a fully connected neural network $f(\cdot )$ with one hidden layer. With an analogy to the Masked LM task in BERT, we also used specific symbol [MASK] to randomly replace the original medical code $c_\ast \in \mathcal {C}_\ast ^1$ . So there are $15\%$ codes in $\mathcal {C}_\ast $ which will be replaced randomly and the model should have the ability to predict the masked code based on others.
Likewise, for the dual-prediction task, since the visit embedding $\mathbf {v}_\ast $ carries the information of medical codes of type $\ast $ , we can further expect it has the ability to do more task-specific prediction as follows:
$$\begin{aligned} \mathcal {L}_{du} = -\log p(\mathcal {C}_d^1 | \mathbf {v}_m^1) - \log p(\mathcal {C}_m^1 | \mathbf {v}_d^1) \end{aligned}$$   (Eq. 20)
where we use the same transformation function $\text{Sigmoid}(f_1(\mathbf {v}_m^1))$ , $\text{Sigmoid}(f_2(\mathbf {v}_d^1))$ with different weight matrix to transform the visit embedding and optimize the binary cross entropy loss $\mathcal {L}_{du}$ expanded same as $\mathcal {L}_{se}$ in Eq. 19 . This is a direct adaptation of the next sentence prediction task. In BERT, the next sentence prediction task facilitates the prediction of sentence relations, which is a common task in NLP. However, in healthcare, most predictive tasks do not have a sequence pair to classify. Instead, we are often interested in predicting unknown disease or medication codes of the sequence. For example, in medication recommendation, we want to predict multiple medications given only the diagnosis codes. Inversely, we can also predict unknown diagnosis given the medication codes.
Thus, our final pre-training optimization objective can simply be the combination of the aforementioned losses, as shown in Eq. 21 . It is used to train on EHR data from all patients who only have one hospital visits..
$$\begin{aligned} \mathcal {L}_{pr} = \frac{1}{N} \sum _{n=1}^N (( &\mathcal {L}_{se}(\mathbf {v}_d^{1, (n)}, \mathcal {C}_d^{1, (n)}) + \mathcal {L}_{se}(\mathbf {v}_m^{1, (n)}, \mathcal {C}_m^{1, (n)})\\ & + \mathcal {L}_{du}^{(n)}) ) \end{aligned}$$   (Eq. 21)
Fine-tuning
After obtaining pre-trained visit representation for each visit, for a prediction task on a multi-visit sequence data, we aggregate all the visit embedding and add a prediction layer for the medication recommendation task. To be specific, from pre-training on all visits, we have a pre-trained Transformer encoder, which can then be used to get the visit embedding $\mathbf {v}_*^\tau $ at time $\tau $ . The known diagnosis codes $\mathcal {C}_d^t$ at the prediction time $t$ is also represented using the same model as $\mathbf {v}_*^t$ . Concatenating the mean of previous diagnoses visit embeddings and medication visit embeddings, also the last diagnoses visit embedding, we built an MLP based prediction layer to predict the recommended medication codes as in Equation 23 .
$$\mathbf {y}_t = \mathrm {Sigmoid}(\mathbf {W}_1 [(\frac{1}{t}\sum _{\tau < t} \mathbf {v}_d^\tau ) || (\frac{1}{t}\sum _{\tau < t} \mathbf {v}_m^\tau ) ||\mathbf {v}_d^t] + b)$$   (Eq. 23)
where $\mathbf {W}_1 \in \mathbb {R}^{|\mathcal {C}_m| \times 3d} $ is a learnable transformation matrix.
Given the true labels $\hat{\mathbf {y}}_t$ at each time stamp $t$ , the loss function for the whole EHR sequence (i.e. a patient) is
$$\mathcal {L} = - \frac{1}{T-1} \sum _{t=2}^T (\mathbf {y}_t^\intercal \log (\hat{\mathbf {y}}_t) + (1 - \mathbf {y}_t^\intercal ) \log (1 - \hat{\mathbf {y}}_t))$$   (Eq. 24)
Experiment
Data We used EHR data from MIMIC-III BIBREF21 and conducted all our experiments on a cohort where patients have more than one visit. We utilize data from patients with both single visit and multiple visits in the training dataset as pre-training data source (multi-visit data are split into visit slices). In this work, we transform the drug coding from NDC to ATC Third Level for using the ontology information. The statistics of the datasets are summarized in Table 2 .
Baseline We compared G-BERT with the following baselines. All methods are implemented in PyTorch BIBREF22 and trained on an Ubuntu 16.04 with 8GB memory and Nvidia 1080 GPU.
We also evaluated three G-BERT variants for model ablation.
Metrics To measure the prediction accuracy, we used Jaccard Similarity Score (Jaccard), Average F1 (F1) and Precision Recall AUC (PR-AUC). Jaccard is defined as the size of the intersection divided by the size of the union of ground truth set $Y_t^{(k)}$ and predicted set $\hat{Y}_t^{(k)}$ .
$$\text{Jaccard} = \frac{1}{\sum _k^N \sum _t^{T_k} 1}\sum _k^N \sum _t^{T_k} \frac{|Y_t^{(k)} \cap \hat{Y}_t^{(k)}|}{|Y_t^{(k)} \cup \hat{Y}_t^{(k)}|}\nonumber $$   (Eq. 36)
where $N$ is the number of patients in test set and $T_k$ is the number of visits of the $k^{th}$ patient.
Implementation Details We randomly divide the dataset into training, validation and testing set in a $0.6 : 0.2 : 0.2$ ratio. For G-BERT, the hyperparameters are adjusted on evaluation set: (1) GAT part: input embedding dimension as 75, number of attention heads as 4; (2) BERT part: hidden dimension as 300, dimension of position-wise feed-forward networks as 300, 2 hidden layers with 4 attention heads for each layer. Specially, we alternated the pre-training with 5 epochs and fine-tuning procedure with 5 epochs for 15 times to stabilize the training procedure.
For LR, we use the grid search over typical range of hyper-parameter to search the best hyperparameter values which result in L1 norm penalty with weight as $1.1$ . For deep learning models, we implemented RNN using a gated recurrent unit (GRU) BIBREF24 and utilize dropout with a probability of 0.4 on the output of embedding. We test several embedding choice for baseline methods and determine the dimension for medical embedding as 300 and thershold for final prediction as 0.3 for better performance. Training is done through Adam BIBREF25 at learning rate 5e-4. We fix the best model on evaluation set within 100 epochs and report the performance in test set.
Results
Experimental Results Table. 3 compares the performance on the medication recommendation task. For variants of G-BERT, $\texttt {G-BERT}_{G^-,P^-}$ performs worse compared with $\texttt {G-BERT}_{G^-}$ and $\texttt {G-BERT}_{P^-}$ which demonstrate the effectiveness of using ontology information to get enhanced medical embedding as input and employ an unsupervised pre-training procedure on larger abundant data. Incorporating both hierarchical ontology information and pre-training procedure, the end-to-end model G-BERT has more capacity and achieve comparable results with others.
As for baseline models, LR and Leap are worse than our most basic model ( $\texttt {G-BERT}_{G^-,P^-}$ ) in terms of most metrics. Comparing $\texttt {G-BERT}_{P^-}$ and GRAM, which both used medical ontology information without pre-training, the scores of our $\texttt {G-BERT}_{P^-}$ is slightly higher in all metrics. This can demonstrate the validness of using Transformer encoders and the specific prediction layer for medication recommendation. Our final model G-BERT is also better than the attention based model, RETAIN, and the recently published state-of-the-art model, GAMENet. Specifically, even adding the extra information of DDI knowledge and procedure codes, GAMENet still performs worse than G-BERT.
In addition, we visualized the pre-training medical code embeddings of $\texttt {G-BERT}_{G^-}$ and G-BERT to show the effectiveness of ontology embedding using online embedding projector shown in (https://raw.githubusercontent.com/jshang123/G-Bert/master/saved/tsne.png/).
Conclusion
In this paper we proposed a pre-training model named G-BERT for medical code representation and medication recommendation. To our best knowledge, G-BERT is the first that utilizes language model pre-training techniques in healthcare domain. It adapted BERT to the EHR data and integrated medical ontology information using graph neural networks. By additional pre-training on the EHR from patients who only have one hospital visit which are generally discarded before model training, G-BERT outperforms all baselines in prediction accuracy on medication recommendation task. One direction for the future work is to add more auxiliary and structural tasks to improve the ability of code representaion. Another direction may be to adapt our model to be suitable for even larger datasets with more heterogeneous modalities.
Acknowledgment
This work was supported by the National Science Foundation award IIS-1418511, CCF-1533768 and IIS-1838042, the National Institute of Health award 1R01MD011682-01 and R56HL138415.

Question: IS the graph representation supervised?
Answer: The graph representation appears to be semi-supervised. It is included in the learning pipeline for the medical recommendation, where the attention model is learned. (There is some additional evidence that is unavailable in parsed text)

Question: Is the G-BERT model useful beyond the task considered?
Answer: There is nothing specific about the approach that depends on medical recommendations. The approach combines graph data and text data into a single embedding.

It learns a representation of medical records. The learned representation (embeddings) can be used for other predictive tasks involving information from electronic health records.