bert: add conversion script for BERT Token Dropping TF2 checkpoints
Author: stefan-it
Pull request: yes

Hi,

this PR adds a conversion script for BERT models, that were trained with the recently introduced "Token Dropping for Efficient BERT Pretraining" approach, introduced in [this paper](https://arxiv.org/abs/2203.13240):

![Unbenannt](https://user-images.githubusercontent.com/20651387/167600822-fe0bb6ab-0aac-4624-ab63-9cb9db91ee88.png)

Models are trained with the TensorFlow 2 implementation from the TensorFlow models repository, which can be found [here](https://github.com/tensorflow/models/tree/master/official/projects/token_dropping). Note: The model architecture only needs changes during pre-training, but the final pre-trained model is compatible with the original BERT architecture!

Unfortunately, the authors do not plan to release pre-trained checkpoints.

But I have pre-trained several models with their official implementation and I've also released the checkpoints and the PyTorch converted model weights on the Hugging Face Model Hub:

https://huggingface.co/dbmdz/bert-base-historic-multilingual-64k-td-cased

This is a multi-lingual model, that was trained on ~130GB of historic and noisy OCR'ed texts with a 64k vocab.

## Conversion Script Usage

In order to test the conversion script, the following commands can be used to test the conversion:

```bash
wget "https://huggingface.co/dbmdz/bert-base-historic-multilingual-64k-td-cased/resolve/main/ckpt-1000000.data-00000-of-00001"
wget "https://huggingface.co/dbmdz/bert-base-historic-multilingual-64k-td-cased/resolve/main/ckpt-1000000.index"
wget "https://huggingface.co/dbmdz/bert-base-historic-multilingual-64k-td-cased/resolve/main/config.json"
python3 convert_bert_token_dropping_original_tf2_checkpoint_to_pytorch.py --tf_checkpoint_path ckpt-1000000 --bert_config_file config.json --pytorch_dump_path ./exported
```

This outputs:

```bash
All model checkpoint weights were used when initializing BertForMaskedLM.

All the weights of BertForMaskedLM were initialized from the model checkpoint at ./exported.
If your task is similar to the task the model of the checkpoint was trained on, you can already use BertForMaskedLM for predictions without further training.
```

##  Masked LM Predictions

The masked lm predictions are pretty good and are comparable with the multil-lingual model, that was trained with the official BERT implementation. Just use the inference widget on the Hugging Face Model Hub.

In this example, the sentence `and I cannot conceive the reafon why [MASK] hath` is used to test the model. For a good comparison, the [32k hmBERT](https://huggingface.co/dbmdz/bert-base-historic-multilingual-cased) is used that was trained with the official BERT implementation on the same corpus:

```json
[
  {
    "score": 0.3564337193965912,
    "token": 1349,
    "token_str": "she",
    "sequence": "and I cannot conceive the reafon why she hath"
  },
  {
    "score": 0.21097686886787415,
    "token": 903,
    "token_str": "it",
    "sequence": "and I cannot conceive the reafon why it hath"
  },
  {
    "score": 0.10645408183336258,
    "token": 796,
    "token_str": "he",
    "sequence": "and I cannot conceive the reafon why he hath"
  },
  {
    "score": 0.0170532688498497,
    "token": 1049,
    "token_str": "we",
    "sequence": "and I cannot conceive the reafon why we hath"
  },
  {
    "score": 0.01265314407646656,
    "token": 45,
    "token_str": "I",
    "sequence": "and I cannot conceive the reafon why I hath"
  }
]
```

With the 64k hmBERT model that was trained with the Token Dropping approach, the output is:

```json
[
  {
    "score": 0.5147836804389954,
    "token": 796,
    "token_str": "he",
    "sequence": "and I cannot conceive the reafon why he hath"
  },
  {
    "score": 0.1566970944404602,
    "token": 1349,
    "token_str": "she",
    "sequence": "and I cannot conceive the reafon why she hath"
  },
  {
    "score": 0.08448878675699234,
    "token": 903,
    "token_str": "it",
    "sequence": "and I cannot conceive the reafon why it hath"
  },
  {
    "score": 0.020168323069810867,
    "token": 45,
    "token_str": "I",
    "sequence": "and I cannot conceive the reafon why I hath"
  },
  {
    "score": 0.01774059422314167,
    "token": 3560,
    "token_str": "God",
    "sequence": "and I cannot conceive the reafon why God hath"
  }
]
```

## Downstream Task Performance

We have also used this model when participating in the HIPE-2022 Shared Task and the BERT model pre-trained with Token Dropping approach achieved really good results on the NER downstream task, see [results here](https://github.com/dbmdz/clef-hipe/tree/main/experiments/clef-hipe-2022#final-models):

| Backbone LM                  | Configuration       | F1-Score (All, Development) | F1-Score (German, Development) | F1-Score (English, Development) | F1-Score (French, Development) | Model Hub Link
| ---------------------------- | ------------------- | --------------------------- | ------------------------------ | ------------------------------- | ------------------------------ | -----------------------------------------------------------------
| hmBERT (32k)                 | `bs4-e10-lr5e-05#4` | 87.64                       | 89.26                          | 88.78                           | 84.80                          | [here](https://huggingface.co/dbmdz/flair-hipe-2022-ajmc-all)
| hmBERT (64k, token dropping) | `bs8-e10-lr3e-05#3` | 87.02                       | 88.89                          | 86.63                           | 85.50                          | [here](https://huggingface.co/dbmdz/flair-hipe-2022-ajmc-all-64k)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @LysandreJik 

thanks for the approval! I have also added a model card and the model is also mentioned in our new  ["hmBERT: Historical Multilingual Language Models for Named Entity Recognition"](https://arxiv.org/abs/2205.15575) - which is used as the backbone language model for our winning NER models (English and French) :)


/cc @sgugger :hugs: 

Sorry this slipped through the cracks! Thanks a lot for your contributrion!