# On the Use of Knowledge Transfer Techniques for Biomedical Named Entity Recognition +
### Transfer Learning  
Transferring the learned information from one domain to another domain is referred to as transfer learning [26]. Usually, a model is trained on a task in one domain, which is then reused on another related domain or related task [27]. The MTL can also be seen as transfer learning, but in MTL the tasks are learned simultaneously. In contrast, in transfer learning, the tasks are learned sequentially. The transfer learning is done in two stages: pretraining and domain adaptation. The pretraining stage involves the training of the base model, which is then reused on the target task in the adaptation phase. The pretraining phase is expensive, but it is usually required to be performed once. Therefore, it is best practice to choose the source task that can exhibit general representations for many target tasks. In transfer learning, the model is trained on an auxiliary task, which is then reused on the main task. Similarly, the model can be trained on a source domain which can then be reused on the target domain. For instance, the model can be trained on book reviews and then reused on hotel reviews; in this case, the source and target domains are different, but the source and target tasks are the same. Similarly, the source and domains can be the same while the source and target tasks are different, e.g., the object detection model can be used for image classification. In a third case, both the domains and the tasks are different, e.g., spam classifier is used for radiology text report classification. In transfer learning, a pretrained model can also be used as a feature extractor or model weight's initialization. Feature extraction is a feature engineering process that is performed using deep learning models instead of performing manually. When using a pretrained model as a feature representation, some of the layers (usually the early/shallow ones) are kept frozen. In this way, the base model works as an input feature for the target model. Moreover, feature extraction is found to be effective for similar tasks in the transfer learning approach. In this sense, the transfer learning technique can be considered a one time feature engineering method that extracts input features (mostly low-level features, e.g., dots or lines in an image) to other similar tasks. Transfer learning also involves a fine-tuning method [28], where the weights of the pretrained layers are used in the main task model, and then the whole model is fine-tuned. In this case, the weights of the layers are not kept frozen. The idea is to relearn new features rather than learn them from scratch. The naive example could be to learn how to count numbers coming after five (5), so instead of relearning numbers again from one (1), we start learning from the number six (6). The fine-tuning method actually helps a target model to adopt a task-specific representation from the general-purpose representation of a base model. This approach is useful when the objective is to implement a pretrained model for various tasks. [PERSON] et al. [29] fine-tuned a pretrained transformer-based language model for task-specific input transformations during an MTL approach. [PERSON] et al. [30] trained a model on a huge dataset to extract the features for a dataset with few training instances. [PERSON] and [PERSON] [31] empirically showed that the performance of a model could be improved using transfer learning for an imbalanced labels dataset. [PERSON] et al. [32] used a pretrained POS tagging model for word segmentation. [PERSON] et al. [33] used a high-resource language pair to pretrain a machine translation model, which was then applied to a low-resource language pair. [PERSON] et al. [27] performed experiments to compare the feature extractor and the fine-tuning techniques. They found that, for the feature extractor, the performance of the main task model depended on where the layers were cut.