Add CharacterBERT model [WIP]
Author: helboukkouri
Pull request: yes
Labels: name: WIP

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9061

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
*I'm still missing the updates in the general documentation.*

- [x] Did you write any new necessary tests?
*Some of the tests are currently failing. This is due to CharacterBERT having a different input shape than bert (batch size, seq length, token length) instead of (batch size, seq length). Also, there are some other tests that are related to reshaping the embedding which fail for the same reason. I did not fix these test as that would mean changing things in the way the common tests are currently working. For all other cases, I tried my best to implement tests that by adapting those from the BERT suite (these pass).*

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors which may be interested in your PR.

@LysandreJik please have a look at this PR. 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
I will, thanks for the ping @helboukkouri!

Hi @helboukkouri, how are you doing? What do you think of the proposed changes above? Would you like us to take over from here?

> Hi @helboukkouri, how are you doing? What do you think of the proposed changes above? Would you like us to take over from here?

Hi @LysandreJik, sorry for the delay. No need to take over, I have been working on another topic (pre-training code for CharacterBERT) but I'll go back to the PR as soon as possible (beginning of next week at the latest).

I will fix the documentation then move on to the tests - about which I wanted to know if it is okay to change some of the code that is common to all models. I think that most tests that do not pass with CharacterBERT don't because they expect the input to be shaped as `(batch size, seq length)` instead of `(batch size, seq length, token length)`.

Cheers!

Really cool, looking forward to it!

For tests in the common tests that you believe don't apply to CharacterBERT, I would recommend you override them in the CharacterBERT test file directly. For example TAPAS's tokenizer didn't fit to most common tests as it is a table-based model/tokenizer, so we reimplemented most tests in the test file directly:

https://github.com/huggingface/transformers/blob/master/tests/test_tokenization_tapas.py#L49

This is an example for the tokenizer, but you can apply the same for the model!

I still need to fix the tests. I'll try to progress on that asap 😊

Not a bug, just a general comment:

This is perfectly working when using a (potential future) CharacterBERT model, that is cased:

Tokenizer config:

```json
{
    "do_lower_case": false,
    "strip_accents":false
}
```

(place it under a folder e.g. `./convbert-tokenizer-test`.

Then:

```python
In [1]: from transformers import CharacterBertTokenizer

In [2]: tokenizer = CharacterBertTokenizer.from_pretrained("./convbert-tokenizer-test")

In [3]: tokenizer.tokenize("Nice weather in Munich töday!")
Out[3]: ['Nice', 'weather', 'in', 'Munich', 'töday', '!']
```

is perfectly working: no lowercasing and accent stripping (as defined in tokenizer configuration) is done.

Hi @LysandreJik, so I made sure all the tests in `tests/test_modeling_character_bert.py` pass by:

- Fixing some issues regarding the input shape
- Removing the inheritance to ModelTesterMixin

The command I used to run the tests was:
```python
python -m pytest ./tests/test_modeling_character_bert.py
```

Now I guess I could copy/paste and adapt the tests from `ModelTesterMixin` to include them back. But this seems against the whole point of having common tests (and will make the test file for CharacterBERT pretty verbose). Should I do it anyway ? Is it necessary ?

Wanted your input before moving forward. 😊

Also, at some point I will probably need to add the hardcoded hyperparameters like the maximum token length and the character embedding dimension (basically all of [this](https://github.com/helboukkouri/transformers/blob/add-character-bert/src/transformers/models/character_bert/modeling_character_bert.py#L232-L249)) to the `CharacterBertConfig` class.

Hi @helboukkouri, that's fantastic news!!

Regarding the `ModelTesterMixin`, a better approach would be to keep your class as a child of it, but to re-implement directly in that class the tests that need to receive changes. By overriding those tests, you can choose what happens in them.

Regarding the configuration, by all means do! You can add as many CharacterBERT specific configuration properties as you need, and remove the ones that you don't need, of course.


> Regarding the `ModelTesterMixin`, a better approach would be to keep your class as a child of it, but to re-implement directly in that class the tests that need to receive changes.

Of course! Should've thought of that :)

So, I added the tests from `ModelTesterMixin`. They all pass with the exception of those related to embedding tying which I bypass as `CharacterBERT` does not have a WordPiece embedding layer.

I will now complete `CharacterBertConfig` to include the parameters from the CharacterCNN module.
Is there anything else to do ? Please let me know 😊

Hey @helboukkouri ,

do you think it makes sense to add the `max_word_length` value to that configuration :thinking: 

As we discussed this parameter in the corresponding CharacterBERT issue, I'm not sure if it ever will be changed/adjusted :thinking: 

On the other side it is included in the ELMo [configuration files](https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway_5.5B/elmo_2x4096_512_2048cnn_2xhighway_5.5B_options.json) and it also can be seen as a "magic number" in the code...

> do you think it makes sense to add the `max_word_length` value to that configuration 🤔

Hi @stefan-it,

I'm actually working on it right now. It will be a parameter in `tokenizer_config.json` as well as a possible argument for `CharacterBertTokenizer`. This way, it possible for anybody to choose to train models that handle shorter or longer words.

I'm also adding a whole bunch of parameters in the model configuration like the character embeddings dimension and number of highway layers in the `CharacterCnn` module. The only fixed value will be the character "vocabulary size". This will stay at 263 for the 256 possible utf-8 bytes (I'm not sure I'm using the right term here) + the special symbols for [CLS], [SEP], [MASK], [PAD], Beginning/EndOfWord, CharacterPadding.

@LysandreJik Please let me know if there is anything else I can do. 😊

It seems most of the tests failing are solved on `master`, do you mind rebasing on `master`? I'll give a deeper look ASAP!

Sorry Lysandre, I'm not really used to doing merges and rebases. I guess this is good practice ^^
Please let me know if I somehow managed to do what you needed me to do 😊

Flair team here 😅

This is currently not working:

```python
from transformers import CharacterBertTokenizer

tokenizer = CharacterBertTokenizer() 

tokenized_string = tokenizer.tokenize("Hello from Munich!")

encoded_inputs = tokenizer.encode_plus(tokenized_string, max_length=1024, 
                                                                   truncation=True, stride=512, return_overflowing_tokens=True)
```

Problem comes with the `return_overflowing_tokens` argument, it throws the following error message:

```
/mnt/character-bert-pretraining/external/transformers/src/transformers/tokenization_utils_base.py in encode_plus(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2357         )
   2358
-> 2359         return self._encode_plus(
   2360             text=text,
   2361             text_pair=text_pair,

/mnt/character-bert-pretraining/external/transformers/src/transformers/tokenization_utils.py in _encode_plus(self, text, text_pair, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
    440         second_ids = get_input_ids(text_pair) if text_pair is not None else None
    441
--> 442         return self.prepare_for_model(
    443             first_ids,
    444             pair_ids=second_ids,

/mnt/character-bert-pretraining/external/transformers/src/transformers/tokenization_utils_base.py in prepare_for_model(self, ids, pair_ids, add_special_tokens, padding, truncation, max_length, stride, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, prepend_batch_axis, **kwargs)
   2807         # Padding
   2808         if padding_strategy != PaddingStrategy.DO_NOT_PAD or return_attention_mask:
-> 2809             encoded_inputs = self.pad(
   2810                 encoded_inputs,
   2811                 max_length=max_length,

/mnt/character-bert-pretraining/external/transformers/src/transformers/tokenization_utils_base.py in pad(self, encoded_inputs, padding, max_length, pad_to_multiple_of, return_attention_mask, return_tensors, verbose)
   2635
   2636         batch_size = len(required_input)
-> 2637         assert all(
   2638             len(v) == batch_size for v in encoded_inputs.values()
   2639         ), "Some items in the output dictionary have a different batch size than others."

AssertionError: Some items in the output dictionary have a different batch size than others.
```

When a user wants to use the `encode_plus` function we should maybe add some additional checks to avoid these errors :thinking: 


It seems that `attention_scores` and `attention_mask` have different shapes, I just tried the following example from the PR:

```python
from transformers import CharacterBertTokenizer, CharacterBertForNextSentencePrediction
import torch

tokenizer = CharacterBertTokenizer.from_pretrained('helboukkouri/character-bert')
model = CharacterBertForNextSentencePrediction.from_pretrained('helboukkouri/character-bert')

prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
next_sentence = "The sky is blue due to the shorter wavelength of blue light."

encoding = tokenizer(prompt, next_sentence, return_tensors='pt')
outputs = model(**encoding, labels=torch.LongTensor([1]))
logits = outputs.logits

assert logits[0, 0] < logits[0, 1] # next sentence was random
```

This throws:

```bash
  File "nsp.py", line 11, in <module>
    outputs = model(**encoding, labels=torch.LongTensor([1]))
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 744, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/character-bert-pretraining/external/transformers/src/transformers/models/character_bert/modeling_character_bert.py", line 1611, in forward
    outputs = self.character_bert(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 744, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/character-bert-pretraining/external/transformers/src/transformers/models/character_bert/modeling_character_bert.py", line 1149, in forward
    encoder_outputs = self.encoder(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 744, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/character-bert-pretraining/external/transformers/src/transformers/models/character_bert/modeling_character_bert.py", line 742, in forward
    layer_outputs = layer_module(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 744, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/character-bert-pretraining/external/transformers/src/transformers/models/character_bert/modeling_character_bert.py", line 629, in forward
    self_attention_outputs = self.attention(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 744, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/character-bert-pretraining/external/transformers/src/transformers/models/character_bert/modeling_character_bert.py", line 557, in forward
    self_outputs = self.self(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 744, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/character-bert-pretraining/external/transformers/src/transformers/models/character_bert/modeling_character_bert.py", line 480, in forward
    attention_scores = attention_scores + attention_mask
RuntimeError: The size of tensor a (35) must match the size of tensor b (50) at non-singleton dimension 3

```

Hi @helboukkouri, the rebase was exactly what I wanted you to do, so that's great! How is it going? Can we help in any way regarding the failing tests/Stefan's comments?

> Hi @helboukkouri, the rebase was exactly what I wanted you to do, so that's great! How is it going? Can we help in any way regarding the failing tests/Stefan's comments?

Hi @LysandreJik, so right now `CharacterBertTokenizer` works well if you simply do tokenize/convert_token_to_ids, but I still need to make sure other methods work well (e.g. encode_plus - i.e. stefan's comments). I'll work on it when I get the chance. 😊

Other than that I'm not really sure why there are still tests that do not pass. After rebasing, only:

- run_tests_templates
- build_doc
- check_code_quality 

had issues. But now I see that more tests break... I'll try to investigate the other tests but in the meantime, do you have any pointers for solving the three tests listed above ?

Cheers!

> I'll try to investigate the other tests

Is there any chance  that I need to rebase again ? There seems to be some conflicts in

- src/transformers/__init__.py
- src/transformers/models/auto/configuration_auto.py
- src/transformers/models/auto/modeling_auto.py

You might have to rebase once again indeed, the issue here is because of torch 1.8.0. Let me know if you would want me to handle it, happy to!

- Regarding the templates, that's actually the same issue as the code quality, so you can ignore that
- The build_doc gives you a bit of information when you open the failure:
```
/home/circleci/transformers/src/transformers/models/character_bert/configuration_character_bert.py:docstring of transformers.CharacterBertConfig:17:Unexpected indentation.
```
- And finally, you have some code quality issues. In order to do that, you should:
    - Install the code quality tools with `pip install -e .[quality]` at the root of the repo
    - Run `make fixup` which should tell you the issues with the style in your code

Please let me know if I can be of any help, happy to!

Hi, I've been playing around with the implementation as well, specifically the `CharacterBertForMaskedLM`. I keep running into issues with the decoding, which (as I've read in the code) is not currently implemented. Specifically, I'm having a hard time to understand how you are aligning the MLM vocab size (100k in the available model snapshot) with the character-level representations, and how you would (schematically) re-label predictions from your model.

If there is any way to help out with the MLM setup specifically, let me know!

> I'm having a hard time to understand how you are aligning the MLM vocab size (100k in the available model snapshot) with the character-level representations

Hi @dennlinger, glad to hear you're interested in CharacterBERT. I added a MLM vocabulary just as a workaround to allow me to do masked language modeling since CharacterBERT does not have a wordpiece vocab, which in the case of BERT is re-used at the output layer during MLM. So in my case, it is only used for this purpose.

How are you trying to use CharacterBERT ? In a seq2seq context ? When do you need decoding ? 

The specific use case is literally just to predict masked tokens, which I'm using in the following example right now:

```python
from transformers import CharacterBertTokenizer, CharacterBertForMaskedLM, BertTokenizer
import torch

if __name__ == "__main__":
    tokenizer = CharacterBertTokenizer.from_pretrained("helboukkouri/character-bert")
    model = CharacterBertForMaskedLM.from_pretrained("helboukkouri/character-bert")
    
    tokens = tokenizer.tokenize("[CLS] This is a [MASK] [SEP]")
    input_tensor = torch.tensor(tokenizer.convert_tokens_to_ids(tokens)).unsqueeze(0)

    with torch.no_grad():
        outputs = model(input_tensor)
        predictions = outputs[0]

    # How can we interpret the output idx from this?
    predicted_index = torch.argmax(predictions[0, 4, :]).item()
    # This fails currently with NotImplementedError
    predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
```

I think the main issue I have is the [empty vocab file](https://huggingface.co/helboukkouri/character-bert/blob/main/vocab.txt), since I am assuming that you had a specific 100k vocab during your training, right?

> I am assuming that you had a specific 100k vocab during your training, right?

Oh I see now. So, the first thing to know is that the checkpoint in "helboukkouri/character-bert" only has weights for the `CharacterBertModel` part. So, no pre-trained weights for the MLM/NSP parts. So, even if the tokenizer worked properly, you would have had meaningless outputs 😊

On the other hand, the `convert_ids_to_tokens` method from the tokenizer is indeed "missing", and that is because there are no "token ids" with CharacterBERT as each token is seen as a sequence of character/byte ids. However, I think I should implement anyway but in a way where it takes a tensor `(batch, token sequence, character ids)` and returns `(batch, token sequence)`. I'll add it to my todo list :)

Also, I think I can manage to recover the mlm vocab / checkpoints and add them at some point to "helboukkouri/character-bert" so that the entire `CharacterBertForPretraining` model can be loaded and give meaningful output. I'll also need to find a solution for easily recovering the tokens from the predicted MLM ids (maybe add a `convert_mlm_ids_to_tokens` to the tokenizer ?)

Hope this is helpful

Ah, it's making more sense now, the detailed explanation did help a lot! :)
Maybe as a last question: It seems the MLMHead is only able to return `(batch, seq_len, MLM_vocab_size)`, and (as far as I can tell) not the required `(batch, seq_len,character_ids)`. How would one acquire the necessary character predictions from the MLM model?

 I'll continue to watch the PR, thanks for all the effort!

> Please let me know if I can be of any help, happy to!

Thanks @LysandreJik! I don't mind continuing to work on PR but since I have some pressing things to handle first, progress may be slow for some time. If you think you can help solve any issues in the meantime please don't hesitate 😊

Here's my conversion script that I've used to convert my pre-trained model into a masked lm model:

```python
import torch

from transformers import CharacterBertConfig, CharacterBertForMaskedLM

orig_model = torch.load("./ckpt_1184.pt")

orig_state_dict = orig_model["model"]

# wget https://huggingface.co/helboukkouri/character-bert/resolve/main/config.json
config = CharacterBertConfig.from_pretrained("./")

model = CharacterBertForMaskedLM(config)

ignore_keys = ["character_bert.pooler.dense.weight",
              "character_bert.pooler.dense.bias",
              "cls.seq_relationship.weight",
              "cls.seq_relationship.bias"]

for key in ignore_keys:
    del orig_model["model"][key]

model.load_state_dict(orig_model["model"], strict=True)

model.half()

model.save_pretrained("./export")
```

However, when I pass a (masked) sequence to the model, it returns always the same predictions. Here's some example code:

```python
from transformers import CharacterBertTokenizer, CharacterBertForMaskedLM
import torch

model_name = "./export"

tokenizer = CharacterBertTokenizer.from_pretrained(model_name)
model = CharacterBertForMaskedLM.from_pretrained(model_name)

sentence = "Heute ist ein [MASK] Tag"
encoding = tokenizer.encode_plus(sentence, return_tensors="pt")

masked_index = 4

predictions = model(input_ids=encoding["input_ids"])[0]
predicted_index = torch.argmax(predictions[0, masked_index]).item()

mlm_vocabulary = [line.strip().split()[-1] for line in open("mlm_vocab.txt", "rt")]

print("Predicted token:", mlm_vocabulary[predicted_index])
```

I'm currently trying to figure out, where the problem could be :)

> How would one acquire the necessary character predictions from the MLM model?

It's not possible. The MLM task is at the word level, where each word has an index. If you want to convert these indices into tokens, you need the MLM vocabulary for a lookup. The character_id stuff is only at the input level. Both are dissociated, which I understand is a bit weird since both are essentially the same thing in BERT (but that's more of a convenience thing than a necessary aspect).

> How would one acquire the necessary character predictions from the MLM model?

Actually, there might be a way but it's not very natural : you could take the MLM ids -> lookup the token in the MLM vocab -> tokenize it with the CharacterBertTokenizer -> get character ids.

If you repeat this too much it may become very slow. But you may be able to cache some things and make some optimizations  :)

Hey @helboukkouri, please let me know if there remains some issues and you don't have time to work on them - happy to unblock you if that's so!

> Hey @helboukkouri, please let me know if there remains some issues and you don't have time to work on them - happy to unblock you if that's so!

Hi @LysandreJik . I think the most important things right now are to fix the `encode_plus` method of the tokenizer then to rebase in order (I guess ) so that the tests can pass again. Then, there is also the code quality which needs to be improved but maybe that's less important. And finally, at some point, there will be a need for methods to handle MLM ids and convert them back into tokens using a MLM vocabulary which I guess will have to be an additional (optional) attribute of the tokenizer class. In this same context, I will also need to add the MLM model checkpoints in the hub and make sure they can load fine (by the way there are medical versions of CharacterBERT - but I'll add those later).

I don't know if you can help on any with any of these remaining topics. If not, I'll handle them but I need to finish working on some other stuff first 😊

Hope the PR is not taking too long!



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Almost ready 🥳! Only a few things left:

- Add MLM checkpoints to the model hub
- Add MLM vocabulary as an optional attribute of the tokenizer class (only needed for pretraining/MLM)
- Add a method for converting the MLM ids into tokens using the MLM vocabulary

> I'm currently trying to figure out, where the problem could be :)

@stefan-it I just found out that the tokenizer was splitting the special tokens (e.g. "[MASK]" -> "[", "mask",  "]"). Maybe this is why you were having your issue (or maybe not, by anyway it was a bug). I didn't realise that before because I tend to handle token sequences and inject mask tokens directly in there instead of tokenizing from a raw string like you did 😊

> The specific use case is literally just to predict masked tokens, which I'm using in the following example right now:
> 
> ```python
> from transformers import CharacterBertTokenizer, CharacterBertForMaskedLM, BertTokenizer
> import torch
> 
> if __name__ == "__main__":
>     tokenizer = CharacterBertTokenizer.from_pretrained("helboukkouri/character-bert")
>     model = CharacterBertForMaskedLM.from_pretrained("helboukkouri/character-bert")
>     
>     tokens = tokenizer.tokenize("[CLS] This is a [MASK] [SEP]")
>     input_tensor = torch.tensor(tokenizer.convert_tokens_to_ids(tokens)).unsqueeze(0)
> 
>     with torch.no_grad():
>         outputs = model(input_tensor)
>         predictions = outputs[0]
> 
>     # How can we interpret the output idx from this?
>     predicted_index = torch.argmax(predictions[0, 4, :]).item()
>     # This fails currently with NotImplementedError
>     predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
> ```
> 
> I think the main issue I have is the [empty vocab file](https://huggingface.co/helboukkouri/character-bert/blob/main/vocab.txt), since I am assuming that you had a specific 100k vocab during your training, right?

Hey @dennlinger 👋🏼, I added support for the MLM vocabulary. You can now do things like:

```python

from transformers import CharacterBertForPreTraining, CharacterBertTokenizer

# NOTE: loading a "pretrained" tokenizer is necessary for loading the MLM vocabulary
tokenizer = CharacterBertTokenizer.from_pretrained('helboukkouri/character-bert')
model = CharacterBertForPreTraining.from_pretrained('helboukkouri/character-bert')

text = 'My name is [MASK]. Nice to meet you!'
encoding = tokenizer.encode_plus(text, return_tensors="pt")
predicted_mlm_id = model(**encoding).prediction_logits[0][4].argmax().item()
predicted_token = tokenizer.convert_mlm_id_to_token(predicted_mlm_id)  # outputs: "chris"
```

@LysandreJik I think that everything is ready. Please let me know if there is anything else that I can do 😊

Fantastic @helboukkouri! I'll take a look, thanks for the ping :)

@LysandreJik just in case you missed my ping, everything should be good now. Please let me know if there is anything else I can do for this PR. 😊

@LysandreJik I did the rebase (correctly I hope) and I removed some imports from the `__init__.py`, namely:

- `CharacterBertLMHeadModel`
- `CharacterBertPreTrainedModel`
- `load_tf_weights_in_character_bert`
- `CharacterCnn`
- `CharacterBertLayer`

But I kept `CharacterBertModel` since I think it can be convenient to be able to import it directly. I can remove it if necessary 😊

Can't wait for the PR to be merged!

Great work @helboukkouri ! After the PR is merged, I can finally upload my German Wikipedia CharacterBERT model 😍

> Great work @helboukkouri ! After the PR is merged, I can finally upload my German Wikipedia CharacterBERT model 😍

Looking forward to it!

@sgugger I made all the changes you suggested, rebased and ran `make style` and `make fix-copies`. Is there anything else I can do ? 😊

As you can see, there are several failures in the CI that need to be addressed:
1. You still have some formatting errors, run `make quality` locally so they are printed and fix them if you can.
2. The documentation can't build because there is no `CharacterBertLMHeadModel` in this PR but you are trying to document it
3. A lot of tests are failing because there is an import error with `from transformers import CharacterCnn` in test_modeling_common

Thanks, I'll fix the things you mentioned. I'll try and look at the CI logs if there are any more errors after that 😊

Hey guys 😊 Sorry for the long period of inactivity. However, I am still unable to dedicate time to finishing this pull request. I was actually wondering wether somebody could take it from here ? Thanks.

Hey @helboukkouri, thanks for letting us know! We'll take a look :)

Hey @LysandreJik , I'd be happy to help on this PR, too. If someone could help me with some of the more ambiguous merge conflicts, I _should_ be able to make progress on the rest of the issues.

Hey @jaminsore, how about I take care of the rebase, and then we can take a look at the failing tests?

@helboukkouri could you invite us both as collaborators to your fork so that we may commit on the PR directly?

> Hey @jaminsore, how about I take care of the rebase, and then we can take a look at the failing tests?
> 
> @helboukkouri could you invite us both as collaborators to your fork so that we may commit on the PR directly?

Done! Thanks a lot for the help 😊

> Hey @jaminsore, how about I take care of the rebase, and then we can take a look at the failing tests?
> 
> @helboukkouri could you invite us both as collaborators to your fork so that we may commit on the PR directly?

@LysandreJik Sounds good to me.

It should be good to go now - let's take a look at the tests now :)

@LysandreJik FYI, I took a new job in December, and I'm not sure when I'll have the to work on this again. 



@LysandreJik I'm pretty sure I've mangled the history of this branch. Locally, I've reset hard to cfdadd27ae3711d57f0a19aba980543eac3cc669, and I'd like to take another crack at it. How would you like me to handle this? A new PR? A force push? Something else?

Hey @jaminsore, if your local branch looks fine, feel free to open a new PR with it directly. GitHub sometimes messes up the history when the git history is actually perfectly fine.

What is important here is that the authors keep their authorship - so if the commits get somehow mixed up, I would squash them all into one single commit and add all users that contributed to this PR as co-authors manually. I'm happy to take care of that if you're unaware of how to do so, once you have opened the new PR!

@LysandreJik Thanks! The issue was that I mixed up "ours" and "theirs" in my rebases so everything got mangled and I was stuck in an endless cycle of large rebases. This also resulted in changes from `master` not making it at all or only partially making it into this branch. After resetting and rebasing (correctly) onto `master` things are sane again :smiley:. 

Once I fix up the documentation, I'll open a new PR. 

That sounds great, thanks a lot @jaminsore!