Add BORT
Author: stefan-it
Pull request: yes
Labels: name: New model

Hi,

this PR adds the recently introduced BORT model from @adewynter and Daniel J. Perry from the Alexa team into Transformers.

----

BORT was introduced in the [Optimal Subarchitecture Extraction For BERT](https://arxiv.org/abs/2010.10499).

Details about BORT:

> We extract an optimal subset of architectural parameters for the BERT architecture from Devlin et al. (2018) by applying recent breakthroughs in algorithms for neural architecture search. This optimal subset, which we refer to as "Bort", is demonstrably smaller, having an effective (that is, not counting the embedding layer) size of 5.5% the original BERT-large architecture, and 16% of the net size. Bort is also able to be pretrained in 288 GPU hours, which is 1.2% of the time required to pretrain the highest-performing BERT parametric architectural variant, RoBERTa-large (Liu et al., 2019), and about 33% of that of the world-record, in GPU hours, required to train BERT-large on the same hardware. It is also 7.9x faster on a CPU, as well as being better performing than other compressed variants of the architecture, and some of the non-compressed variants: it obtains performance improvements of between 0.3% and 31%, absolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.

This should fix #8135 :hugs: 

---

ToDo tasks:

* [x] Upload models (both PyTorch and TensorFlow model) to model hub
* [x] Add conversion script from Gluonnlp to Transformers
* [x] Enable unit tests (they are working and just wait for the model upload)

Comments:
🔥 Looking forward to taking a look at the conversion script from GluonNLP/mxnet!

@patrickvonplaten I added some examples for both `modeling_bort.py` and modeling_tf_bort.py` :hugs: 

@julien-c The conversion script is also added - you just need to install  `gluonnlp==0.8.3` and `mxnet==1.5.0`.

These versions are defined in the BORT [requirements file](https://github.com/alexa/bort/blob/master/requirements.txt). The conversion script also performs a version check.

We'll have to think a bit how to advertise this. Let me draft up a "Contribution Proposal" for the fine-tuning algorithm.

Hey @stefan-it,

I've discussed a bit with @LysandreJik and @sgugger offline and I do agree with @LysandreJik after having thought about it again. I think it's better if we actually don't add any new code (besides the conversion script) that should be added to `src/transformers/models/bert/` and the docs page. I'm very sorry to have you asked to go down this road! I think however it does make more sense to not add any "tokenizer" or "model" code as those are exact copies of the `RobertaTokenizer` and `BertModel`. It's probably most efficient to open a new PR and only add the required files. Super sorry again!

Are we planning to implement the architectural optimization (FPTAS) or just the pre-trained models?

> Are we planning to implement the architectural optimization (FPTAS) or just the pre-trained models?

Great question! For now, we'll just add the model weights - see: #9813. A community contribution showing how to do FPTAS in a notebook would be extremely valuable though.

Closing in favor of #9813