# Hybridizing Fuzzy String Matching and Machine Learning for Improved Ontology Alignment
## 3 Preliminaries
### Bidirectional Encoder Representations from Transformers  
The Bidirectional Encoder Representations from Transformers (BERT) is a language representation model introduced by [PERSON] et al. [47] at Google; it is a versatile tool that can be applied to a wide range of tasks such as language inference and question answering. Its architecture is founded on the original implementation described in [48]. Unlike the early versions of the Generative Pre-Training of Graph model (GPT) [49], BERT utilizes bidirectional self-attention, allowing each token to attend to both its preceding and subsequent context. Conversely, GPT utilizes unidirectional self-attention, enabling each token to attend solely to the context preceding it.  
BERT employs a two-step process for executing a range of Natural Language Processing (NLP) tasks, which are pre-training and fine-tuning, illustrated in Figure 1. In the pre-training phase, the model is trained on an extensive text corpus to acquire a comprehensive understanding of language. The objective of pre-training is to create a model that can be fine-tuned for various downstream tasks, including text classification, question answering, and natural language generation. Fine-tuning refers to the process of training a pre-trained BERT model for a specific downstream NLP task.  
During the pre-training phase, the BERT model is trained using a masked language modeling (MLM) task and a next sentence prediction (NSP) task. In the MLM task, a percentage of tokens in the input text are randomly masked, and the model is trained to predict the original token based on the context of the surrounding words. The NSP task involves providing two sentences as input and the model needs to predict whether the second sentence follows the first sentence in the original text. By training on these tasks, BERT learns to represent words and sentences in a way that captures their contextual meaning and relationships, which is important for many downstream NLP tasks. Once pre-trained, the BERT model can be fine-tuned on specific tasks with relatively small amounts of task-specific labeled data to achieve state-of-the-art performance [47].  
Fine-tuning involves substituting the final layer of the pre-trained model with a task-specific layer, which is then trained on labeled data for the new task. This approach allows the BERT model to leverage its pre-training to acquire general language understanding and then use the fine-tuning step to adapt to the specificities of the new task [47]. For instance, if we are using BERT for text classification, we would replace the pre-training task's final layer with a classification layer and fine-tune the model on the text classification task with labeled data. Likewise, if we aimed to apply BERT for a question-answering task, we would substitute the final layer with a question-answering layer and fine-tune the model with labeled data specific to that task. The primary advantage of fine-tuning is that it enables the BERT model to be utilized for a diverse range of NLP tasks without necessitating significant modifications to the architecture or training methodology. By fine-tuning the pre-trained model for a particular task, the model can attain state-of-the-art performance even with limited amounts of task-specific labeled data.