# Ontology Attention Layer for Medical Named Entity Recognition
## 3 Proposed Methods
### Ontology Encoder
#### 3.3.1 Serialization and Tokenization  
The primary challenge is how to serialize the ontology to preserve relationship information between the concepts in the resulting sequence. We hypothesize that since a BERT-based language model takes the position of each input token into account by exploiting the position vectors [25], if the position of each class tag or property in the ontology matches the conditions as follows, the language model is able to learn the corresponding relationship information by fine-tuning:  
1. The relative position of two tokens is able to show the relationship between the corresponding class tags or properties.
2. The positions are fixed in every training epoch.  
Treating the ontology as a graph, the relationship of the classes and their properties can be represented by their topological location. Thus, we use alphabetically pre-order depth-first traversal (DFS) to traverse the ontology and output and concatenate the tags of each visited node. In our preliminary study, we found that using pre-order traversal is slightly better than post-order traversal or the combination of both of them. The comparison of pre-order traversal, post-order traversal and the combination of them is provided in Appendix A.  
For the small example in Figure 2, which shows a part of the ontology in our experiment (to be described in Section 4.3), the subtrees are sorted alphabetically from left to right. In the example, \"i\" is before \"r\", so the subtree of \"disease\" is to the left of that of \"drug\". We start from the root, and then the \"disease\" subtree. For the root, we output nothing. For the \"disease\" subtree, at first, we obtain the tag of the node \"disease\". Let us assume that all the tags are equal to the English node name here for simplicity; thus, the sequence is now \"disease\". Then we go to the left-most subtree of the \"disease\" subtree, which has only one node, \"cause\". We obtain its tag and update the sequence to \"disease cause\". Similarly, we obtain \"disease cause clinical manifestations diagnostic criteria disease name\" for the \"disease subtree\". Then we process the \"drug\" subtree and update the sequence to \"disease cause clinical manifestations diagnostic criteria disease name drug contraindication dosage drug component drug name indications\".  
In this way, each node will have a meaningful position that implicitly corresponds to its topological position in the ontology, which is also fixed in every training epoch. We can see that \"clinical manifestations\", which are properties of a \"disease\", are closer to \"disease\" than to \"drug\". Furthermore, the class \"disease\" is closer to its own \"disease name\" than the \"drug name\" property of the class \"drug\".  
Formally, the ontology is considered as a directed tree, where the classes and their properties are treated as nodes, and the relationships, e.g., \"is_property\", are treated as unweighted edges. Denote the tree as \(G=(V,E)\), where \(V\) and \(E\) refer to the nodes and edges, respectively. Depth-first traversal is employed, utilizing alphabetically ordered depth-first search to traverse the ontology and generate a sequence of node tags \(S=(s_{1},s_{2},\ldots,s_{i},\ldots,s_{n})\), where \(i\) is the position and \(n\) is the number of nodes in the ontology. Note that the relationship tags are dropped here because of the limitation of the proposed method.  
Another issue is how to separate the node in the resulting sequence. It depends on the target language. In the experiment, the tags of nodes were separated by a space, because most experiments were performed on a Chinese dataset where space was rarely used.