## 3 Materials and Methods  
The NER and RE web interfaces are web pages showing results of BioBERT NER and RE. The proposed system incorporates the NER dataset generation module of the tagging tool system, as depicted in Figure 4, into the existing web interface of NER and RE with BERT. PubMed identifier (PMID) received based on the user's input is used to extract information such as title, abstract, and authors using the Entrez-NCBI search engine. The BERT-based training model performs data normalization and outputs the NER and RE results for each category. **BERT-based model:** BERT was released by Google in 2017 and is based on a transformer and a deep learning model comprising encoder and decoder structures [30,31]. The transformer preprocesses the input data, transmits it to the encoder, and then inputs the data  
Figure 3: Overview of the ezTag system (cited from [23]). into the decoder. A padding mask is applied to the encoder data to reduce the amount of computation required. Next, a look-ahead mask is applied to the decoder data to hide the prediction word in advance. Then, the padding mask is applied to the data and the existing input data are used as inputs to the encoder. The decoder receives the decoder input data, the result calculated by the encoder, and the look-ahead mask and padding mask values as inputs. Finally, the following word is predicted based on the value calculated by the decoder. Figure 5 depicts the encoder and decoder structures within the transformer. We focus solely on the encoder as [PERSON] is trained using only this component of the transformer. Both the encoder and the decoder are responsible for three main tasks: positional encoding, multi-head self-attention, and position-wise feed-forward networks. Positional encoding considers positional information of words as word order is essential to comprehension of sentence structures in language. By incorporating positional encoding of the same dimension as the existing embedding, embeddings with time signals can be used as input. Attention can be classified as scaled dot-product attention or multi-head attention. Multi-head attention yields better values by concatenating multiple scaled dot-product attentions. Multi-head attention is used because it reduces the size of vectors and enables parallel processing when multiple attention functions are used. \[\text{MultiHead}(\text{Q},\text{K},\text{V})=\text{Concat}(\text{head}_{1}, \dots,\text{head}_{\text{h}})\text{W}^{\text{O}} \tag{1}\]  
\[\text{head}_{\text{i}}=\text{Attention}\Big{(}\text{Q}\text{W}_{\text{i}}^{ \text{Q}},\text{Ki}^{\text{K}},\text{V}\text{W}_{\text{i}}^{\text{V}}\Big{)} \tag{2}\]  
The aforementioned equations give the formula for multi-head attention. The attention of each head (keys, values, queries) is divided by h to reduce the dimension and concatenated. Finally, position-wise feed-forward networks are layers that extract information  
Figure 4: Overview of the proposed web interface of NER and RE with BERT. Figure 5: Structure of the transformer. from sentences. They are implemented alongside an attention layer and are composed of two linear transformations, using ReLU as an activation function. Generative Pretrained Transformer (GPT) is another model based on the decoder of the aforementioned transformer. It exhibits weak contextual understanding because it only reads sentences from left to right, i.e., in one direction. BERT, which reads text bidirectionally using only the encoder of the aforementioned transformer, can understand the context more naturally. Figure 6 depicts input representation of BERT. Two tokens stand out in the input section: [CLS] and [SEP]. The first token of every sentence is [CLS]; the classification value determined by BERT is dependent on the operation of this token. The [SEP] token is used at the end of each sentence to distinguish it from the subsequent one. BERT consists of three embedding layers: token embedding layer (word-piece embedding layer), segment embedding layer, and position embedding layer.