Fine-tuning
Author: shainaraza
Pull request: no
Labels: name: wontfix

I read lot of articles on pre-training vs fine-tuning but I am still not able to get the meaning in the context of transformer models, I understand the pre-training is training the dataset from scratch , while (point of confusion) the fine-tuning is using the pre-trained model/data and add on the top of that with our own custom dataset?
Then it comes the downstream tasks, any fine line of distinctions among these three terms pre-training, finetuning and downstream tasks
please clarify

Comments:
Both pre-training and fine-tuning involve _training_ a model (i.e., updating the weights of the model using backpropagation).

* In a first step, a model is (pre-)trained on a very large dataset (such as all English Wikipedia articles). BERT for example, is pre-trained on 2 tasks: masked language modeling (MLM) and next sentence prediction (NSP). Masked language modeling is, for example, given the sentence "I went to the [MASK] to buy a bread", the model must predict the word "bakery" in this case. For sentence prediction, given 2 sentences A and B, the model must predict whether sentence B follows sentence A in the dataset, or is just a random sentence. Note that both MLM and NSP are self-supervised learning (we don't need to manually annotate the dataset, because we can just use Wikipedia and mask out some words or randomize the sentences). 

The reason they call it pre-training is because it's training a model before you train it on another, second, dataset. 

* In a second step, a model can be fine-tuned on a task of interest, usually called the downstream task. This can be text classification for example, or named-entity recognition, question-answering, summarization,... Fine-tuning is also training a model, but we start with the model that was already (pre-)trained on the large dataset in the first step. The reason this is done is because the model has already learned a lot about language in general (just by predicting masked words and interpreting the order of sentences). So the weights of these model are already quite good, they contain some "knowledge". We can now just use this model, and train it further on our own (usually way smaller) dataset. Note that this is supervised learning (we need to collect a labelled dataset). In case the downstream task is text classification (for example determining whether a movie review is positive or negative), then we need to collect a dataset of movie reviews and label each individual review with either "positive" or "negative".  

This picture by [Jay Allamar](http://jalammar.github.io/illustrated-bert/) illustrates this very well (note that in the figure below, the downstream task is also text classification):

![](http://jalammar.github.io/images/bert-transfer-learning.png)

BTW, please post any questions which are not bugs/new features you would like to see added on the [forum](https://discuss.huggingface.co/) rather than here.



> Both pre-training and fine-tuning involve _training_ a model (i.e., updating the weights of the model using backpropagation).
> 
> * In a first step, a model is (pre-)trained on a very large dataset (such as all English Wikipedia articles). BERT for example, is pre-trained on 2 tasks: masked language modeling (MLM) and next sentence prediction (NSP). Masked language modeling is, for example, given the sentence "I went to the [MASK] to buy a bread", the model must predict the word "bakery" in this case. For sentence prediction, given 2 sentences A and B, the model must predict whether sentence B follows sentence A in the dataset, or is just a random sentence. Note that both MLM and NSP are self-supervised learning (we don't need to manually annotate the dataset, because we can just use Wikipedia and mask out some words or randomize the sentences).
> 
> The reason they call it pre-training is because it's training a model before you train it on another, second, dataset.
> 
> * In a second step, a model can be fine-tuned on a task of interest, usually called the downstream task. This can be text classification for example, or named-entity recognition, question-answering, summarization,... Fine-tuning is also training a model, but we start with the model that was already (pre-)trained on the large dataset in the first step. The reason this is done is because the model has already learned a lot about language in general (just by predicting masked words and interpreting the order of sentences). So the weights of these model are already quite good, they contain some "knowledge". We can now just use this model, and train it further on our own (usually way smaller) dataset. Note that this is supervised learning (we need to collect a labelled dataset). In case the downstream task is text classification (for example determining whether a movie review is positive or negative), then we need to collect a dataset of movie reviews and label each individual review with either "positive" or "negative".
> 
> This picture by [Jay Allamar](http://jalammar.github.io/illustrated-bert/) illustrates this very well (note that in the figure below, the downstream task is also text classification):
> 
> ![](https://camo.githubusercontent.com/1a96f1ed09104b2d0267c8cced10fbdc9da74987/687474703a2f2f6a616c616d6d61722e6769746875622e696f2f696d616765732f626572742d7472616e736665722d6c6561726e696e672e706e67)
> 
> BTW, please post any questions which are not bugs/new features you would like to see added on the [forum](https://discuss.huggingface.co/) rather than here.

thanks @NielsRogge  for the splendid information

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.