Add example for fine tuning BERT language model
Author: ghost
Pull request: yes

We are currently working on fine-tuning the language model on a new target corpus. This should improve the model, if the language style in your target corpus differs significantly from the one initially used for training BERT (Wiki + BookCorpus), but is still too small for training BERT from scratch. In our case, we apply this on a rather technical english corpus.

The sample script is loading a pre-trained BERT model and fine-tunes it as a language model (masked tokens & nextSentence) on your target corpus. The samples from the target corpus can either be fed to the model directly from memory or read from disk one-by-one. 

Training the language model from scratch without loading a pre-trained BERT model is also not very difficult to do from here. In contrast, to the original tf repo, you can do the training with multi-GPU instead of TPU.

We thought this might be also helpful for others. 

Comments:
This looks like a great addition!

Is it a full re-implementation of the pre-training script?

The implementation uses the same sampling parameters and logic, but it's not a one-by-one re-implementation of the original pre-training script. 

**Main differences:**
- In the original repo they first create a training set of TFrecords from a raw corpus ([create_pretraining_data.py](https://github.com/google-research/bert/blob/master/create_pretraining_data.py)) and then perform model training using [run_pretraining.py](https://github.com/google-research/bert/blob/master/run_pretraining.py). We decided 
against this two step procedure and do the conversion from raw text to sample "on the fly" (more similar to [this repo from codertimo](https://github.com/codertimo/BERT-pytorch)). With this we can actually generate new samples every epoch.
- We currently feed in pair of lines (= sentences) as one sample, while the [original repo](https://github.com/google-research/bert/blob/master/create_pretraining_data.py#L229) fills 90% of samples up with more sentences until max_seq_length is reached (for our use case this did not make any sense)

**Main similarities:**
- All sampling / masking probabilities and parameters
- Format of raw corpus (one sentence per line & empty line as doc delimiter) 
- Sampling strategy: Random nextSentence must be from another document
- The data reader [of codertimo](https://github.com/codertimo/BERT-pytorch) is similar to our code, but didn't really match the original method of sampling.

Happy to clarify further details!

Hi @deepset-ai this is great and, just a suggestion, maybe if this makes it to the repo it would be great to include something in the README too about this functionality in this pull request?

Just added some basic documentation to the README. Happy to include more, if @thomwolf thinks that this makes sense.

Yes, I was going to ask you to add some information in the readme, it's great. The more is the better. If you can also add instructions on how to download a dataset for the training as in the other examples it would be perfect. If your dataset is private, do you have in mind another dataset that would let the users try your script easily? If not it's ok, don't worry.

Another thing is that the `fp16` logic has now been switched to NVIDIA's [apex module](https//github.com/nvidia/apex) and we have gotten rid of the `optimize_on_cpu` option (see the [relevant PR](#116) for more details). You can see the changes in the current examples like `run_squad.py`, it's actually a lot simpler since we don't have to manage parameters copy in the example and it's also faster. Do you think you could adapt the fp16 parts of your script similarly?

This is something I'd been working on as well, congrats on a nice implementation!

One question, though: I noticed you stripped out the code for evaluating on a test set, but when fine-tuning the LM on a smaller corpus, would it be worth keeping that in? Overfitting is much more of a risk in a smaller corpus.

> This is something I'd been working on as well, congrats on a nice implementation!
> 
> One question, though: I noticed you stripped out the code for evaluating on a test set, but when fine-tuning the LM on a smaller corpus, would it be worth keeping that in? Overfitting is much more of a risk in a smaller corpus.

@Rocketknight1, you are right that we will probably need some better evaluation here. Currently, I have the feeling though that the evaluation on down-stream tasks is more meaningful (see also Jacob Devlin's comment [here](https://github.com/google-research/bert/issues/95#issuecomment-437599265)). But in addition, some better monitoring of the loss during and after training would be nice. 

Do you already have something in place and would like to contribute on this? Otherwise, I will try to find some time during the upcoming holidays to add this.

> 
> 
> > This is something I'd been working on as well, congrats on a nice implementation!
> > One question, though: I noticed you stripped out the code for evaluating on a test set, but when fine-tuning the LM on a smaller corpus, would it be worth keeping that in? Overfitting is much more of a risk in a smaller corpus.
> 
> @Rocketknight1, you are right that we will probably need some better evaluation here. Currently, I have the feeling though that the evaluation on down-stream tasks is more meaningful (see also Jacob Devlin's comment [here](https://github.com/google-research/bert/issues/95#issuecomment-437599265)). But in addition, some better monitoring of the loss during and after training would be nice.
> 
> Do you already have something in place and would like to contribute on this? Otherwise, I will try to find some time during the upcoming holidays to add this.

I don't have any evaluation code either, unfortunately! It might be easier to just evaluate on the final classification task, so it's not really urgent. I'll experiment with LM fine-tuning when I'm back at work in January. If I get good benefits on classification tasks I'll see what effect early stopping based on validation loss has, and if that turns out to be useful too I can submit a PR for it?

Have you thought about extending the vocabulary after fine-tuning on custom dataset.  This could be useful if the custom dataset has specific terms related to that domain.  

> Have you thought about extending the vocabulary after fine-tuning on custom dataset. This could be useful if the custom dataset has specific terms related to that domain.

Adjusting the vocabulary before fine-tuning could be interesting, but you would need some smart approach to exchange "less important" tokens from the original byte pair vocab with "important" ones from your custom corpus (while maintaining the pre-trained embeddings for the rest of the vocab meaningful). 
We don't work on this at the moment. Looking forward to a PR, if you have time to work on this. 

> > Have you thought about extending the vocabulary after fine-tuning on custom dataset. This could be useful if the custom dataset has specific terms related to that domain.
> 
> Adjusting the vocabulary before fine-tuning could be interesting, but you would need some smart approach to exchange "less important" tokens from the original byte pair vocab with "important" ones from your custom corpus (while maintaining the pre-trained embeddings for the rest of the vocab meaningful).
> We don't work on this at the moment. Looking forward to a PR, if you have time to work on this.

Yes I am working on it. The idea is to add more items to the pretrained vocabulary.  Also will adjust the model layers: bert.embeddings.word_embeddings.weight, cls.predictions.decoder.weight with the mean weights and also update cls.predictions.bias with mean bias for additional vocabulary words.

Will send out a PR once I test it.

Ok this looks very good, I am merging, thanks a lot @tholor!