# Can We Survive without Labelled Data in NLP? Transfer Learning for Open Information Extraction
## 2 Transfer Learning in NLP  
Formerly, there was a misconception that a machine learning framework will achieve the desirable results only if the testing data and training data have similar distribution and feature space. Thus, a new framework was required for data with different distribution properties and features, making the collection of labelled training data expensive and difficult. Transfer learning lessens the demand of gathering an immense amount of labelled training data by reemploying the knowledge gained from a different task to tackle new tasks faster and constructively.  
[PERSON] and [PERSON] introduced a transfer learning taxonomy [3]. Additionally, they categorized transfer learning into three classes:  
Inductive transfer learning: labelled data are accessible in source and target domain.  
Transductive transfer learning: labelled data are only available in the source domain.  
Unsupervised transfer learning: No labelled data are both source and target domain.  
Transfer learning has been implemented in various different machine learning tasks, achieving notable results, for instance, textual summarization [4], named entity recognition [5], question answering [6; 7], and text classification [8].  
BERT (Bidirectional Encoder Representations from Transformers) [9] was a breakthrough in transfer learning on a range of language-based tasks, not only due to the fact that BERT was pretrained  
\begin{table}
\begin{tabular}{c c} \hline \hline \multirow{2}{*}{**Sentence**} & **[PERSON] Was Born on 9 October 1940, in Liverpool and Gained** \\  & **Worldwide Fame as the Founder of the Beatles.** \\ \hline \multirow{4}{*}{**OIE Tuples**} & \textless{} [PERSON], Born, 9 October 1940\textgreater{} \\  & \textless{} [PERSON], Born, Liverpool\textgreater{} \\  & \textless{} [PERSON], founder, Beatles\textgreater{} \\ \cline{2-3}  & Person-Born-On: \textless{} [PERSON], Born, 9 October 1940\textgreater{} \\ \cline{2-3}  & Person-Born-In: \textless{} [PERSON], Born, Liverpool \textgreater{} \\ \cline{2-3}  & Person-Organization: \textless{} [PERSON], founder, Beatles\textgreater{} \\ \hline \hline \end{tabular}
\end{table}
Table 1: Open information extraction and relation extraction example.  
on an immense dataset, but also because it has a substantial number of transformer blocks (encoder layers) and feed-forward networks. Later on, many transfer learning models built on BERT were introduced, for example ULMFiT [10] and OpenAI transformer [11]. This novel development also affected the way words are encoded, with more elaboration being found in Section 4.2.  
As shown in Figure 1, in our work, two transductive transfer learning experiments were carried out. The first one transfers knowledge learned from the OIE news domain to the OIE bio-medical domain--this is referred to as domain adaptation. In contrast to transfer learning, domain adaption entails adapting a model trained on one domain to other different domains on the same task. The default process of supervised domain adaptation for neural models involves pre-training the network on data from the source domain followed by fine-tuning hyperparameters on data from the target domain. The second experiment transfers information from the OIE news domain to the RE news domain. Moreover, a small percentage of OIE bio-medical data were added to OIE news data to experiment with inductive transfer learning. Similarly, a small amount of RE training data were inputted to the neural model along with OIE news corpus, with both experiments being referred to as multi-task learning.