Transfer Learning In NLP
By: Pratik Bhavsar
Tags: Artificial Intelligence, Naturallanguageprocessing, Machine Learning, Programming, Data Science

Transfer learning

At present we have achieved fantastic results for many tasks like speech recognition, machine comprehension, object detection, and machine translation due to massive efforts of collecting data. These huge models are extremely data-hungry and require an immense amount of labelled data.

Most of the time there is a huge difference between these datasets available for training and its target application — for example training data available for speech recognition of USA but the target region is India.

In the last few years, we have learnt the art of making great machine learning models for audio, image, and text data with large amounts of data. Unfortunately every time we train a model on a dataset and try to use it for a different dataset, the performance deteriorates. This happens as the model fails to generalize and understand the basic patterns in the data. Even a slight difference can catch it off-guard.

This problem is important to tackle as the real-world data is ever-changing and it’s impractical to keep retraining model frequently from scratch for new scenarios. This is where transfer learning comes to the rescue.

Fig:1.1 Traditional supervised learning

Suppose we have a sentiment analysis task for the domain of Indian stock news. We have enough labelled data for this supervised task and train a model for it. Now when we apply the model to the task in the same domain, it will probably behave as expected. But, as soon as we apply it for the same task in another domain, such as cryptocurrencies, it will behave unexpectedly.

If we have to use the same paradigm of supervised learning, we need to collect labelled data for cryptocurrency news and train a new model. Hopefully, it will now perform great for cryptocurrency news, although who can predict crpytos! 😂

But what if we don’t have enough labelled data or cannot afford to collect it? Doesn’t it make sense to leverage the older model and train it for small labelled data we have for the cryptos?

But what if we want to add another prediction class to the model, such as neutral, while the earlier model was just trained to predict positive and negative? Transfer learning allows us to deal with these scenarios and use knowledge learned from a previous task/domain for a new one.

Let’s give a formal definition to transfer learning. Given a source domain Ds, a corresponding source task Ts, as well as a target domain Dt and a target task Tt, the objective of transfer learning now is to enable us to learn the target conditional probability distribution P(Yt|Xt) in Dt with the information gained from Ds and where Ds ≠Dt or Ts ≠ Tt.

In most cases, a limited number of labelled target examples, which is quite smaller than the number of labelled source examples are assumed to be available. The following figure illustrates the process of transfer learning:

Fig: 1.2 Transfer learning

Types of transfer learning

Domain adaptation

Cross-lingual learning

Multi-task learning

Sequential transfer learning

Based on domain and task there can be many variations which can be tackled using transfer learning:

Xs ≠ Xt : The feature space of source and target is different. For example, the domain-specific words of stocks are different from those of cryptos. Word initial coin offering ( ICO ) which is specific to cryptos will never occur in the context of stocks. Also, if the languages differ, there can be a complete mismatch of feature space. This scenario is referred to as cross-lingual learning or cross-lingual adaptation.

: The feature space of source and target is different. For example, the domain-specific words of stocks are different from those of cryptos. Word ( ) which is specific to cryptos will never occur in the context of stocks. Also, if the languages differ, there can be a complete mismatch of feature space. This scenario is referred to as cross-lingual learning or cross-lingual adaptation. Ps(Xs) ≠ Pt(Xt) : The marginal probability distribution of words is different for source and target. The word ledger will be used more frequently for cryptos while it will be used rarely for stocks. This scenario is generally known as domain adaptation.

: The marginal probability distribution of words is different for source and target. The word ledger will be used more frequently for cryptos while it will be used rarely for stocks. This scenario is generally known as domain adaptation. Ys ≠ Yt : Labels differ for source and target. Source had positive and negative labels while the target has neutral as well.

: Labels differ for source and target. Source had positive and negative labels while the target has neutral as well. Ps(Ys) ≠ Pt(Yt) : The marginal probability distribution of labels is different for source and target. Positive labels occur more than negative in training but the market crashed and the real data has more negative labels than positive.

: The marginal probability distribution of labels is different for source and target. Positive labels occur more than negative in training but the market crashed and the real data has more negative labels than positive. Ps(Ys|Xs) ≠ Pt(Yt|Xt): The condition probability distribution of labels is different. This can happen if the same words can mean different or the data imbalance is different for source and target, for example, cold storage means totally different in the world of cryptos.

Now we define taxonomy as per Pan and Yang [2010]. They segregate transfer learning mainly into transductive and inductive. It is further divided into domain adaption, cross-lingual learning, multi-task learning and sequential transfer learning.