# Web Interface of NER and RE with BERT for Biomedical Text Mining
## 3 Materials and Methods
### Fine-Tuning for Relation Extraction  
In this section, we will explain the BioBERT fine-tuning task of RE. RE is the task of determining the relationship between named entities. Identifying the relationship between words in a sentence can be highly helpful in interpreting meaning or purpose. For example, it can benefit natural-language-processing applications such as sentiment analysis, question answering, and summarization. In addition, structural triples can be extracted from sentences to summarize information and identify key components. Our RE task determines whether there is a relationship between (gene-disease) entities from different domains in a biomedical corpus sentence. Identifying entities from the NER model comes first, before performing the RE task. In Figure 6, the NER model identified  
Figure 6: Word tokenizer and word-piece tokenizer. \"BACE1\" as a genes/proteins entity and \"AD\" as a diseases entity in the input sentence. To prevent model overfitting, genes/proteins' entities are replaced with the \"@GENES\" mask and diseases' entities with the \"@DISEASES\" mask in that sentence. The actual input sentence for RE in that sentence is \"We identified 15 novel @GENES substrates that we specifically altered in @DISEASES\". Given input sentences as text, the model tokenizes them and breaks them into words. Each word is pre-trained by BERT and mapped to a defined word ID. Through this process, token embeddings of the sentence are created, and semantic information of each token is extracted. Then, based on the object recognition results, we can extract the object's embedding through the object's surrounding context. In the process of extracting relationships between entities, additional models are constructed using the [CLS] tokens and their embeddings from the output of the BERT model to further understand the contextual information between entities. A single output layer is then used to output the highest probabilities of possible relationship types in the input sentence. We performed RE in this way. With a sentence sequence of extracted named entities as input, the BioBERT RE model outputs whether the two genes/proteins and diseases objects are related in each sense. To visualize RE results, we use NetworkX. It is a Python package that makes it simple to create and analyze complex networks [17]. This allows the result data to be handled and manipulated in a graph format, as well as output in various formats such as json and kt. A named entity extracted from the NER model and found to be related to other types of objects in the RE model is added as a node in the graph. Each node's attributes were set to the recognized types of each entity, genes/proteins, and diseases. Graph types include directed and undirected graphs. Because we do not extract directed results between entities, our RE results are represented using undirected graphs. After all of the graph's nodes have been initialized, the undirected graph is completed by adding edges between nodes and saving the results in json format. This saved result is fed into Cytoscape for visualization. Cytoscape is a bioinformatics package for visualizing biological networks [18]. We loaded the previously saved json data into the javascript language and used the Cytoscape library to visualize the graph so that we could interact with the user. The process of extracting and analyzing relationships between entities is as follows. First, we extract sentences from the entire abstract that contain one or more disease and gene category entities. \"However, despite numerous BACE1 inhibitors entering clinical trials, none have successfully improved AD pathogenesis, despite effectively lowering A\(\beta\) concentrations.\" is an example of one of the extracted sentences. If there is more than one entity in each category, we perform the task of finding a number of sentences in all cases where a gene-disease link can be found. In the example sentence, two entities, \"BACE1\" and \"A\(\beta\)\", were extracted as genes, and \"AD\" was extracted as a disease.