"Easy" path for classifier training / pre-training
Author: Rocketknight1
Pull request: no
Labels: name: Discussion

name: wontfix

I've noticed quite a few issues from people outside research who want to fine-tune a pre-trained BERT model to solve a task they're working on, but there's a steep learning curve. Right now, the workflow for someone who wants to use this repo for a custom task is something like this:

1) Understand how DataProcessors work and write a custom DataProcessor (or read the code for the existing data processors and hack your training data into a format that works for them)
2) Understand the `examples/run_classifier.py` script and modify it to use the custom DataProcessor
3) Write a script (or another modification of `run_classifier.py`) that loads unlabelled data and performs inference

This is a lot of work, especially for people who aren't familiar with the codebase! The Tensor2Tensor TF BERT repo is even worse - it's even harder for newcomers to do anything without understanding the code in detail. But it's possible to make BERT accessible to a lot more people with just a few changes:

1) Make a generic DataProcessor and clearly describe the expected input format in the docs so that people don't have to read the code to understand it. For example, the GenericDataProcessor class could expect one training example per line, and one label per line in a separate file. We could also add a GenericPairedDataProcessor, where the classifier reads two sequences as input instead of just one (e.g. for entailment tasks).

2) Add an inference script that loads a saved model state file and performs classifications and writes a file of its predictions. It should read data using the same GenericDataProcessor class, but will not require a label file to be present. If labels are present, it can also write evaluation metrics like accuracy.

3) Optionally, modify `run_classifier.py` to allow loading of fine-tuned BERT language models from the `lm_finetuning/` scripts

4) Document the whole workflow with example calls so that people can use BERT and get state-of-the-art results without needing to read any code!

To make it even easier, we could add a functional interface so people could just call something like `pytorch_pretrained_bert.train_classifier()`

Do you think this is a good idea? Or is it too end-user focused - would it work better as a separate repo that used this one as a dependency, especially if this repo is moving away from being BERT-specific and turning into more of a large set of PyTorch Transformer model implementations?

Comments:
I join this issue.

Also I have a question related to the p.3
> Optionally, modify run_classifier.py to allow loading of fine-tuned BERT language models from the lm_finetuning/ scripts

`finetune_on_pregenerated.py` script uses `BertForPreTraining` with 2 heads and this is like vanilla training from the original paper. But `run_classifier.py` uses `BertForSequenceClassification` which is being learned only for predicting labels, not masked tokens and isNextSeq. Am I right?
If so, how can I merge these two approaches? I want to fine-tune the pretrained bert for my dataset and also train a classifier on the top of it.

Thank you.

It's a good question and a good discussion @Rocketknight1.

I think your suggestion of "splitting" the present repo in two by extracting the examples in a separate repo and refactoring them to have a better workflow is a good idea.

At the present stage, it's important to keep the core library stable as it is used in downstream libraries but the examples are an area where many people would like to contribute to the community and it would be nice to have a more flexible structure that allows such contributions more easily than the present monolithic setup.

So if you want to start a new repo splitting the "usage-part" of this repo it could be a very interesting idea I think. I'm also happy to help and host it close to the present repo if it's what you had in mind (maybe shoot me an email in that case).

Understood! I'm tight on time right now, but if I find time I'll try to build an interface like that and contact you to ensure we sync things up between the two repos.

Keeping the discussion open on this

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.