Add ALBERT to the library
Author: LysandreJik
Pull request: yes

This PR adds ALBERT to the library. It offers two new model architectures:

- AlbertModel
- AlbertForMaskedLM

AlbertModel acts in a similar way to BertModel as it returns a sequence output as well as a pooled output. AlbertForMaskedLM exposes an additional language modeling head.

A total of four pre-trained checkpoints are available, which are the checkpoints discussed in the official ALBERT paper, available on the TensorFlow hub page:

- albert-base
- albert-large
- albert-xlarge
- albert-xxlarge

These are currently available on the S3 bucket: an ALBERT model may be loaded like other models with the following code.

```py
from transformers import AlbertTokenizer, AlbertModel

tokenizer = AlbertTokenizer.from_pretrained("albert-base")
model = AlbertModel.from_pretrained("albert-base")
```

What is left to implement:

- ~PyTorch model & tests~
- ~Tokenizer & tests~
- ~Export PyTorch checkpoints~
- ~TensorFlow 2 model & tests~
- ~Export TensorFlow 2 checkpoints~
- Replicate the results obtained in the paper; **currently obtained 81 acc on MNLI with albert-base**

# Workflow for including a model from [README.md](https://github.com/huggingface/transformers/blob/master/templates/adding_a_new_model/README.md)

Here an overview of the general workflow: 

- [ ] add model/configuration/tokenization classes
- [ ] add conversion scripts
- [ ] add tests
- [ ] finalize

Let's details what should be done at each step

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the files to replace `XXX` (with various casing) with your model name
- [x] copy-past or create a simple configuration class for your model in the `configuration_...` file
- [x] copy-past or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0)
- [x] copy-past or create a tokenizer class for your model in the `tokenization_...` file

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [x] copy the conversion script (`convert_...`) from the present folder to the main folder.
- [x] edit this script to convert your original checkpoint weights to the current pytorch ones.

# Adding tests:

Here is the workflow for the adding tests:

- [x] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the tests files to replace `XXX` (with various casing) with your model name
- [x] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] add import for all the relevant classes in `__init__.py`
- [x] add your configuration in `configuration_auto.py`
- [x] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [x] add your tokenizer in `tokenization_auto.py`
- [x] [high-level-API] add your models and tokenizer to `pipeline.py`
- [x] [high-level-API] add a link to your conversion script in the main conversion utility (currently in `__main__` but will be moved to the `commands` subfolder in the near future)
- [x] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
- [x] add a mention of your model in the doc: `README.md` and the documentation it-self at `docs/source/pretrained_models.rst`.
- [x] upload the pretrained weigths, configurations and vocabulary files.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1683?src=pr&el=h1) Report
> Merging [#1683](https://codecov.io/gh/huggingface/transformers/pull/1683?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fa735208c96c18283b8d2f3fcbfc3157bbd12b1e?src=pr&el=desc) will **increase** coverage by `0.9%`.
> The diff coverage is `87.16%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1683/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1683?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master    #1683     +/-   ##
=========================================
+ Coverage   85.08%   85.99%   +0.9%     
=========================================
  Files          94       98      +4     
  Lines       13920    14713    +793     
=========================================
+ Hits        11844    12652    +808     
+ Misses       2076     2061     -15
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1683?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl94bG5ldC5weQ==) | `90.24% <ø> (ø)` | :arrow_up: |
| [transformers/configuration\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fYWxiZXJ0LnB5) | `100% <100%> (ø)` | |
| [transformers/modeling\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2FsYmVydC5weQ==) | `81.73% <81.73%> (ø)` | |
| [transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX2FsYmVydC5weQ==) | `84.46% <84.46%> (ø)` | |
| [transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9hbGJlcnQucHk=) | `89.74% <89.74%> (ø)` | |
| [transformers/tests/modeling\_tf\_albert\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2FsYmVydF90ZXN0LnB5) | `94.39% <94.39%> (ø)` | |
| [transformers/tests/modeling\_albert\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2FsYmVydF90ZXN0LnB5) | `95.04% <95.04%> (ø)` | |
| [transformers/tests/tokenization\_albert\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL3Rva2VuaXphdGlvbl9hbGJlcnRfdGVzdC5weQ==) | `97.43% <97.43%> (ø)` | |
| [transformers/tests/modeling\_bert\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2JlcnRfdGVzdC5weQ==) | `96.38% <0%> (-0.54%)` | :arrow_down: |
| [transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fdXRpbHMucHk=) | `98.66% <0%> (-0.02%)` | :arrow_down: |
| ... and [20 more](https://codecov.io/gh/huggingface/transformers/pull/1683/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1683?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1683?src=pr&el=footer). Last update [fa73520...afef0ac](https://codecov.io/gh/huggingface/transformers/pull/1683?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Will there be models for classification?

Yes, at the moment there is AlbertForSequenceClassification and there may be more soon

@LysandreJik Thanks for adding this :+1: 

I've one question: the ALBERT team did release version 2 of their models yesterday, see: 

https://github.com/google-research/google-research/commit/2ba150bef51fcedcfda31f16321264300f201a8d

Are these updated models available on S3 yet 🤔

V2 just use 0 dropout and lr to 1e-5, the architecture didn’t change, so
maybe it just need time to transfer model to here.

On Sat, Nov 2, 2019 at 21:06 Stefan Schweter <notifications@github.com>
wrote:

> @LysandreJik <https://github.com/LysandreJik> Thanks for adding this 👍
>
> I've one question: the ALBERT team did release version 2 of their models
> yesterday, see:
>
> google-research/google-research@2ba150b
> <https://github.com/google-research/google-research/commit/2ba150bef51fcedcfda31f16321264300f201a8d>
>
> Are these updated models available on S3 yet 🤔
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/1683?email_source=notifications&email_token=AIEAE4FY2TZTBZ5YV6P5ARLQRV3ONA5CNFSM4JHPMB22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC43PFI#issuecomment-549042069>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIEAE4BGB37TAZRZI6CQ3HLQRV3ONANCNFSM4JHPMB2Q>
> .
>


I'm not sure how heavily you want to take advantage of Apex when available, but Apex does provide a fused implementation of Lamb. https://nvidia.github.io/apex/optimizers.html#apex.optimizers.FusedLAMB

@stefan-it the ALBERT v2 models are now available on the S3. You can access them using `albert-{base,large,xlarge,xxlarge}-v2` identifiers!

@BramVanroy Indeed, thanks! For now we're focusing more on the model implementation rather than the optimizers; the optimizers can be obtained from other libraries (such as apex) and used with the models from `transformers` so it is not a priority right now.

Hi @LysandreJik thanks for the model versioning :)

Just a few notes from my (early) experiments with this ALBERT implementation.

I used a feature-based approach in Flair for NER on English CoNLL dataset. More precisely I used embeddings from all layers (incl. word embedding layer) + scalar mix over all layers to get an embedding for the first subtoken of each token. Results for the base model are "ok": 93.13 (dev) and 89.17 (test) compared to BERT base: 94.74 (dev) and 91.38 (test).

After work I implemented an `AlbertForTokenClassification` class and added it to the `run_ner.py` example script. With default parameters 88.06 (dev) and 82.94 (test) could be achieved (so there's large room for improvement in my implementation 😅).

But: I also tested the `large` and `xlarge` models. Using Flair (and all 24 + 1 layers with scalar mix) the F-score dropped to 45% on test set?! The fine-tuning experiment (with `run_ner.py`) yields 0% for F-score 😂

I'm not sure what's going on with the > `large` models 🤔 (I did experiments for NER only)

HI @stefan-it,
I won't suggest ALBERT for NER task. As of now, all the released weights are trained using lowering the sentence.  NER model is usually built using  Cased models. BERT NER is based on bert-base/large-cased. 

For BERT the difference was ~ 0.2 to 0.3% on CoNLL (base and large model, feature-base approach) - but I'll further investigate the large ALBERT models 😅

Hi @stefan-it, thanks for your study! I could indeed replicate the issue with ALBERT-large which has very bad results on SQuAD after being fine-tuned on it. I'm looking into it today and I'll update you on the progress.

I curious about how worse for squad

On Wed, Nov 6, 2019 at 21:57 Lysandre Debut <notifications@github.com>
wrote:

> Hi @stefan-it <https://github.com/stefan-it>, thanks for your study! I
> could indeed replicate the issue with ALBERT-large which has very bad
> results on SQuAD after being fine-tuned on it. I'm looking into it today
> and I'll update you on the progress.
>
> —
> You are receiving this because you commented.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/1683?email_source=notifications&email_token=AIEAE4G7RYHSXRPZ5Y2MWJ3QSLENDA5CNFSM4JHPMB22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDGTPTQ#issuecomment-550320078>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIEAE4AOIEITNSQM6EO55FDQSLENDANCNFSM4JHPMB2Q>
> .
>


I've looked into it and there seems to be an error with the models `large`, `xlarge` and `xxlarge` version 2. The `base` models of both versions as well as the larger models of version 1 seem to work correctly (there was an issue that was fixed this morning).

@pohanchi based on a single epoch just to check the models were learning, `albert-base-v2` obtains 82.5 exact and 89.9 F1 and `albert-large-v1` obtains 82.8 exact and 90 F1

I'm looking into the V2 models now.

Side question: for how long are you planning to support Python 2? Considering it's as good as EOL and all that.

@BramVanroy, as long as Google outputs models in Python 2 we'll continue to maintain it, and probably for a few months after that!

(That's only for the core code though. Examples and scripts are already Python 3 only AFAIK)

@LysandreJik Great job! Could you elaborate on why you added and removed the Lamb optimizer? Is there any issue with this implementation?

Great work all. Tried it and noticed a few things, that may or may not be issues, but I'll post the details here just in case.:

- doesn't work in pytorch 1.1.0, does in 1.2.0. This is probobly OK as 1.2.0 is the version listed in requirements.dev.txt
  - The error is for line [`w = self.dense.weight.T` "Parameter self.dense.weight has not attribute T"](https://github.com/huggingface/transformers/blob/06fc337815/transformers/modeling_albert.py#L206)
- You may be aware of this but it doesn't work with fp16 O1 yet 
  - `RuntimeError: Expected object of scalar type Half but got scalar type Float for argument #2 'mat2'`
  - reffering to line [`projected_context_layer = torch.einsum("bfnd,ndh->bfh", context_layer, w) + b`](https://github.com/huggingface/transformers/blob/06fc337815/transformers/modeling_albert.py#L209). Specifically context_layer is half, w while b are float.
  - these changes fix fp16 O1:
    - `w = self.dense.weight.T.view(self.num_attention_heads, self.attention_head_size, self.hidden_size).to(context_layer.dtype)`
    -  `b = self.dense.bias.to(context_layer.dtype)`
- it does run without fp16 :)

Thank you very much for your great work!@LysandreJik

I have tried running with the run_glue.py file to obtain the test accuracy for MNLI task. 
**Without training, just evaluation**. Using the **albert-base-v1** model from the S3, I have obtained **31.8% accuracy** for MNLI, which differs greatly from the ALBERT paper. However, after training with the default hyperparameters specified in the run_glue.py file, I obtained an accuracy which is similar to the paper. 

I am a new guy to NLP, previously working in CV. I am wondering does the S3 model contains the pretrained weight for ALBERT? Since without training, the result differs greatly from the papers.



@panaali LAMB was first added but I couldn't manage to make it work immediately, so as the authors said that there was not a huge difference between ADAM and LAMB, I removed it and fine-tuned with ADAM instead. As I told Bram a few messages ago: "For now we're focusing more on the model implementation rather than the optimizers; the optimizers can be obtained from other libraries (such as apex) and used with the models from transformers so it is not a priority right now.". I believe you can use existing LAMB implementations with our models and it will work out of the box, such as [cybertronai's implementation.](https://github.com/cybertronai/pytorch-lamb), or from apex.

@wassname Thank you for your comments, I'm looking into that.

@astrongstorm the model as it is saved on our S3 only contains the base model, without the classification head (similarly to most of the models hosted on our S3). Before using them, it is essential to fine-tune them so that the classification head may be trained on the actual dataset.

@LysandreJik Thanks for your reply! There is still one confusing point about the S3 model. I am wondering in S3 model, does it contain both hyperparameter and the parameters for the model, or it only contains one of them. 

The S3 holds several files: 
- The configuration files which holds what you might call the hyper-parameters: number of inner group, hidden size, vocabulary size, etc.
- The model files which contain parameters for pytorch (albert-xxx-pytorch_model.bin) and for tensorflow (albert-xxx-tf_model.h5)
- the tokenizer vocabulary files

Thanks @LysandreJik for the great work! I am looking forward to use it. When will this branch be merged to the master or is there a timeline?

Hi @jimmycode, it should be merged at the end of the week.

I think the v1 models are looking good (v2 are currently very bad) - I did some comparisons for NER (CoNLL-2003):

| Model                    | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg.
| ------------------------ | ----- | ----- | ----- | ----- | ----- | ---------
| BERT large, cased (Dev)  | 95.69 | 95.47 | 95.77 | 95.86 | 95.91 | 95.74
| BERT large, cased (Test) | 91.73 | 91.17 | 91.77 | 91.22 | 91.46 | **91.47**
| ALBERT xxlarge, uncased, v1 (Dev) | 95.35 | 95.42 | 95.17 | 95.16 | 95.39 | 95.30
| ALBERT xxlarge, uncased, v1 (Test) | 91.49 | 91.60 | 91.69 | 90.88 | 91.27 | 91.39

(although cased vs. uncased is not really a fair comparison)

I'll prepare a PR when the ALBERT code was merged to support a "for-token-classification" interface that can be used in the `run_ner.py` example.

Hi Thanks for the quick addition.

Does ALBERT require the usage of AlbertTokenizer? or can we simply use BERTTokenizer?

Because otherwise, there might be a need to re-process all data using AlbertTokenizer.

Hi @jshin49. yes the `AlbertTokenizer` should be used: BERT uses word pieces, ALBERT uses a sentence piece model. The output of the tokenizer implementations is totally different:

```python
from transformers import AlbertTokenizer, BertTokenizer

bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
albert_tokenizer = AlbertTokenizer.from_pretrained("albert-base-v1")

sentence = "neuschwanstein near munich"
```

Outputs:

```python
In [9]: bert_tokenizer.tokenize(sentence)
Out[9]: ['ne', '##us', '##ch', '##wan', '##stein', 'near', 'munich']

In [10]: albert_tokenizer.tokenize(sentence)
Out[10]: ['▁neu', 'sch', 'wan', 'stein', '▁near', '▁munich']
```

Thank you @LysandreJik for the great work! , Do you have any plans to add multilingual ALBERT?

> After work I implemented an `AlbertForTokenClassification` class and added it to the `run_ner.py` example script.

@stefan-it could you add this as PR?


Oh, I totally forgot that 😅

I can look into it the next days :)