Add MultiBERTs conversion script
Author: gchhablani
Pull request: yes

# What does this PR do?

This PR adds MultiBERTs checkpoint conversion script to the bert model. This PR closes #13069.

Currently the issue is that some weights required in HuggingFace `BertForPreTraining` are not present in the classifier layer.

The `cls` keys present in `BertForPreTraining`:

```python
cls.predictions.bias
cls.predictions.transform.dense.weight
cls.predictions.transform.dense.bias
cls.predictions.transform.LayerNorm.weight
cls.predictions.transform.LayerNorm.bias
cls.predictions.decoder.weight
cls.predictions.decoder.bias
cls.seq_relationship.weight
cls.seq_relationship.bias
```
Name splits starting with `cls` in the MultiBERTs checkpoint `seed_0.zip`:

```python
['cls', 'predictions', 'output_bias']
['cls', 'predictions', 'transform', 'LayerNorm', 'bias']
['cls', 'predictions', 'transform', 'LayerNorm', 'kernel']
['cls', 'predictions', 'transform', 'dense', 'bias']
['cls', 'predictions', 'transform', 'dense', 'kernel']
['cls', 'seq_relationship', 'output_bias']
['cls', 'seq_relationship', 'output_weights']
```

The following weights are not present:

```python
cls.predictions.decoder.weight
cls.predictions.decoder.bias
```

How do I handle this?

EDIT:
------
I checked for the original BERT checkpoints present in the table [here](https://github.com/google-research/bert) , the same issue happens there as well.

EDIT 2:
------
The BERT conversion skip also skips the final layers (pre-training). Specifically, this script does not handle MLM/NSP heads.

Comments:
Thanks for working on this. Where are the MultiBERT checkpoints in the README you link to?

Nvm, found them here: https://github.com/google-research/language/tree/master/language/multiberts

And question out of interest: was it not possible to use the existing conversion script?

@NielsRogge Thanks for taking a look at this. 

The existing scripts do not consider NLP/MLM heads - which are probably randomly initialized for downstream models, including pre-training, MLM and NSP. I feel, for MultiBERTs, we might want the heads too? I'm not 100% sure of this requirement.

Maybe @yanaiela can share the exact requirement he has in mind if that is not the case?

What do you think?

So, although in my specific use case I don't need the MLM heads, it would be nice to have.

Also, I don't think that this is what this PR is doing, but I think it would be beneficial to integrate such function into the standard model loading. There's an increasing interest of researchers in studying the intermediate steps of models, and there's more and more publications that release these checkpoints. It would be good to be able to load these checkpoint seamlessly through the regular api, rather than converting each checkpoint on its own.

@yanaiela Are you saying that the checkpoints should be available on the Hub for easy use?

For example:

```python

from transformers import BertModel
model = BertModel.from_pretrained('multiberts-seed-0')
```
?

Or, add a custom definition in the modeling file which does something like:

```python
from transformers import BertModel
model = BertModel.from_pretrained_original_tf_url(<URL>)
```

Well, the first option would be super convenient (btw, it should also contain the checkpoint step).

I think this conversion script will be needed in order to convert. I can put these models up on the hub after that so the first option can be used.

ah sure, but I'm not sure if the best option would be to upload all of them to the hub?
It may accumulate to a lot of storage (relatively) with all the checkpoints, so maybe a good option would be to integrate the conversion when calling the `from_pretrained` function locally.
What do you think?

@yanaiela If that is the requirement, then is there an issue with downloading the checkpoint, using conversion script and then using the model? 

I could add another method which downloads from the URL if not present in cache and returns the model.

For example
```python
from transformers.models.bert.convert_original_multiberts_tf2_checkpoint_to_pytorch import get_pretrained_multiberts_checkpoint

seed_0_model = get_pretrained_multiberts_checkpoint(seed=0, force_download=True)
```


Ah I mainly was commenting on the comment of putting models on the hub.
Integrating the script sounds like a good idea though!

:P Okay. Not sure which is the best way to go, wdyt @NielsRogge?

Hi,

> ah sure, but I'm not sure if the best option would be to upload all of them to the hub?

Storage is not a problem on the hub. All MultiBERT checkpoints can be uploaded there. In that way, people can do the following:

```
from transformers import BertModel

model = BertModel.from_pretrained("google/multibert-...")
```

Regarding the conversion script, I wonder whether we could update the existing conversion script in `modeling_bert.py` to also include the MLM and NSP heads, such that we don't need to add a new one. 

> The following weights are not present:

> cls.predictions.decoder.weight
> cls.predictions.decoder.bias

=> is that because MultiBERT checkpoints use weight tying (i.e. same embedding layer at the input and output)?

@NielsRogge

Sorry I didn't check this earlier. You are right, they [use the input embeddings](https://github.com/google-research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/run_pretraining.py#L140) to the [`get_masked_lm_output` method](https://github.com/google-research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/run_pretraining.py#L257).

Should I edit the bert conversion script with an option to include MLM/NSP heads?


@NielsRogge I checked. The current `convert_bert_original_tf_checkpoint_to_pytorch` script does exactly what I'm doing ;-; Sorry, I was looking at the `tf2` checkpoint script earlier. :/ My bad.

Should I start pushing the multiberts checkpoints to the hub, then?

Yes, you can upload all checkpoints to the hub (under the "Google" namespace), and then close this PR.

Thanks a lot for working on this @gchhablani!

I have pushed all final checkpoints to the hub as `multiberts-seed-x` where x ranges from 0 to 24.

For intermediate, I'm thinking something like `multiberts-seed-x-10k` for the 10k-th checkpoint. Does this sound okay?

Yes, that's fine!

---
language: en
tags:
- exbert
- multiberts
license: apache-2.0
datasets:
- bookcorpus
- wikipedia
---
# MultiBERTs Seed 0 (uncased)
Seed 0 pretrained BERT model on English language using a masked language modeling (MLM) objective. It was introduced in
[this paper](https://arxiv.org/pdf/2106.16163.pdf) and first released in
[this repository](https://github.com/google-research/language/tree/master/language/multiberts). This model is uncased: it does not make a difference
between english and English.

Disclaimer: The team releasing MultiBERTs did not write a model card for this model so this model card has been written by [gchhablani](https://hf.co/gchhablani).
## Model description
MultiBERTs models are transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
was pretrained with two objectives:
- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
  recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
  GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
  sentence.
- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
  they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
  predict if the two sentences were following each other or not.
This way, the model learns an inner representation of the English language that can then be used to extract features
useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
classifier using the features produced by the BERT model as inputs.
## Intended uses & limitations
You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for
fine-tuned versions on a task that interests you.
Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
generation you should look at model like GPT2.
### How to use
Here is how to use this model to get the features of a given text in PyTorch:
```python
  from transformers import BertTokenizer, BertModel
  tokenizer = BertTokenizer.from_pretrained('multiberts-seed-0')
  model = BertModel.from_pretrained("bert-base-uncased")
  text = "Replace me by any text you'd like."
  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)
```
### Limitations and bias
Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
predictions. This bias will also affect all fine-tuned versions of this model. For an understanding of bias of this particular
checkpoint, please try out this checkpoint with the snippet present in the [Limitation and bias section](https://huggingface.co/bert-base-uncased#limitations-and-bias) of the [bert-base-uncased](https://huggingface.co/bert-base-uncased) checkpoint.

## Training data
The MultiBERTs models were pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
headers).
## Training procedure
### Preprocessing
The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
then of the form:
```
  [CLS] Sentence A [SEP] Sentence B [SEP]
```
With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus and in
the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
"sentences" has a combined length of less than 512 tokens.
The details of the masking procedure for each sentence are the following:
- 15% of the tokens are masked.
- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
- In the 10% remaining cases, the masked tokens are left as is.
### Pretraining
The model was trained on 16 Cloud TPU v2 chips for two million steps with a batch size
of 256. The sequence length was set to 512 throughout. The optimizer
used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
learning rate warmup for 10,000 steps and linear decay of the learning rate after.
### BibTeX entry and citation info
```bibtex
  @article{DBLP:journals/corr/abs-2106-16163,
    author    = {Thibault Sellam and
                 Steve Yadlowsky and
                 Jason Wei and
                 Naomi Saphra and
                 Alexander D'Amour and
                 Tal Linzen and
                 Jasmijn Bastings and
                 Iulia Turc and
                 Jacob Eisenstein and
                 Dipanjan Das and
                 Ian Tenney and
                 Ellie Pavlick},
    title     = {The MultiBERTs: {BERT} Reproductions for Robustness Analysis},
    journal   = {CoRR},
    volume    = {abs/2106.16163},
    year      = {2021},
    url       = {https://arxiv.org/abs/2106.16163},
    eprinttype = {arXiv},
    eprint    = {2106.16163},
    timestamp = {Mon, 05 Jul 2021 15:15:50 +0200},
    biburl    = {https://dblp.org/rec/journals/corr/abs-2106-16163.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
  }
```
<a href="https://huggingface.co/exbert/?model=multiberts">
	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
</a>


Adding this README to all, will just replace `seed-0` with `seed-x` and `Seed 0` with `Seed X`.

That looks great @gchhablani !

@NielsRogge @patrickvonplaten I have added all the MultiBERTs checkpoints to `google` organization on the hub.

@yanaiela You can access them like:

```python
from transformers import BertModel
model = BertModel.from_pretrained('google/multiberts-seed-0')
intermediate_checkpoint = BertModel.from_pretrained('google/multiberts-seed-0-20k')
```

Please let me know in case of any issues!

Awesome, thanks!
btw, based on the documentation, it seems like the tokenizer uses the `bert-base-uncased` tokenizer. Is there a reason not to allow the same name with the tokenization?

Looks good to me!

@gchhablani - just to follow-up here...did you manage to correctly convert the mulit-bert checkpoints with an existing conversion script (that also took into account MLM and NSP)? In this case, I might have merged the PR too quickly :sweat_smile: 

Hi @patrickvonplaten 
Sorry, I didn't check this PR was merged. Yes, I had updated the script.

However, we don't need this script as the `load_tf_checkpoint` method works fine for the conversion. Should I create another PR to remove the script? Or should I revert the merge?

I used the following script for pushing intermediate checkpoints: [MultiBERTs Pushing Script](https://gist.github.com/gchhablani/070d41ec7b02a0b3b0429d04cadee557) and a similar one for the final checkpoints. We don't need a new conversion script.


@yanaiela No, there's no reason why we cannot add the tokenizer files as well to the checkpoints. I can do that if needed. 

Wdyt @patrickvonplaten @NielsRogge?

Sure it would be nice to add the tokenizer files as well!

> Hi @patrickvonplaten Sorry, I didn't check this PR was merged. Yes, I had updated the script.
> 
> However, we don't need this script as the `load_tf_checkpoint` method works fine for the conversion. Should I create another PR to remove the script? Or should I revert the merge?
> 
> I used the following script for pushing intermediate checkpoints: [MultiBERTs Pushing Script](https://gist.github.com/gchhablani/070d41ec7b02a0b3b0429d04cadee557) and a similar one for the final checkpoints. We don't need a new conversion script.

Yeah it would be great if you could open a new PR to delete the conversion file in this case then :-) Thanks a lot!

@patrickvonplaten @yanaiela
I have added the tokenizer files to the checkpoints and updated the model card accordingly.
Please let me know if you find any issues.

It works great btw. Well done, and thanks!