Add new pre-trained models BERTweet and PhoBERT
Author: datquocnguyen
Pull request: yes
Labels: name: model card

I'd like to add pre-trained [BERTweet](https://github.com/VinAIResearch/BERTweet/) and [PhoBERT](https://github.com/VinAIResearch/PhoBERT/) models to the `transformers` library. 

Users now can use these models directly from `transformers`. E.g: 

    bertweettokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-base")
    bertweetmodel = BertweetModel.from_pretrained("vinai/bertweet-base")

    phoberttokenizer = PhobertTokenizer.from_pretrained("vinai/phobert-large")
    phobertmodel = PhobertModel.from_pretrained("vinai/phobert-large")



[BERTweet: A pre-trained language model for English Tweets](https://github.com/VinAIResearch/BERTweet/)
[PhoBERT: Pre-trained language models for Vietnamese](https://github.com/VinAIResearch/PhoBERT/)



Comments:
> I'd like to add pre-trained [BERTweet](https://github.com/VinAIResearch/BERTweet/) and [PhoBERT](https://github.com/VinAIResearch/PhoBERT/) models to the `transformers` library.
> 
> Users now can use these models directly from `transformers`. E.g:
> 
> ```
> bertweettokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-base")
> bertweetmodel = BertweetModel.from_pretrained("vinai/bertweet-base")
> 
> phoberttokenizer = PhobertTokenizer.from_pretrained("vinai/phobert-large")
> phobertmodel = PhobertModel.from_pretrained("vinai/phobert-large")
> ```
> 
> [BERTweet: A pre-trained language model for English Tweets](https://github.com/VinAIResearch/BERTweet/)
> [PhoBERT: Pre-trained language models for Vietnamese](https://github.com/VinAIResearch/PhoBERT/)

Whether I can get any support from huggingface w.r.t. this pull request @julien-c ? Thanks.

Hello @datquocnguyen ! As you've said, BERTweet and PhoBERT reimplement the RoBERTa model without adding any special behavior. I don't think it's necessary to reimplement them then, is it? Uploading them on the hub should be enough to load them into RoBERTa architectures, right?

Hi @LysandreJik 
They use different tokenizers (i.e. fastBPE), so we cannot load their tokenizers using RoBERTa. 
Please see a loading example using RoBERTa: https://github.com/VinAIResearch/BERTweet#transformers 
An issue related to this is at: #5965   




I hope both BERTweet and PhoBERT could be incorporated into `transformers` in a similar manner to as their counterparts (e.g. CamemBERT and FlauBERT). @LysandreJik Please let me know what I can do for this. Thanks.

Yes, I understand, that makes sense. There shouldn't be any issue in incorporating them into `transformers`.

I've taken a quick look at it, and it looks very cool! Something that we can maybe do better, is regarding the tokenizers:

- They're currently untested, but they're the main contribution of this PR so they definitely should be tested.
- If possible, we would like not to add an additional dependency (in this case FastBPE). It would be great to leverage the already existing library `huggingface/tokenizers`
- On that front, given it's a BPE tokenizer, it should be easy enough to leverage the OpenAI GPT (not GPT-2) tokenizer, which seems very similar. It might even be possible to load the vocab/merge files directly in `OpenAIGPTTokenizer`.

Let me know what you think!

Haven't tried it directly, but as seen with @n1t0, since you're not doing any fancy pre-processing it might be as simple as the following:

```py
class PhobertTokenizerFast(PreTrainedTokenizerFast):
    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["attention_mask"]
    def __init__(self, vocab_file, merges_file, unk_token="<unk>", **kwargs):
        kwargs.setdefault("unk_token", unk_token)
        super().__init__(
            CharBPETokenizer(vocab_file=vocab_file, merges_file=merges_file, unk_token=unk_token, lowercase=False, bert_normalizer=False, split_on_whitespace_only=True),
            **kwargs,
        )
```

Thanks very much @LysandreJik I will revise the code following your comments and inform you as soon as I complete it. 

@datquocnguyen Yeah, these models are cool. Lovin' it. I think we can try to figure out how to convert `fastBPE` formats to our compatible format before adding it directly to our dependency (I believe `XLM` uses `fastBPE`). so would you hold on a little when we try to figure it out? We have to be cautious when adding dependencies! Thanks!
cc @LysandreJik 

Yes. Thanks @JetRunner 

some tokenizer function (decode, convert_ids_to_tokens) hasn't implemented for PhoBertTokenizer right?

@datquocnguyen Thank you for this pull request. I tried the Bertweet model and met a problem that the tokenizer encoded special symbols like "\<pad\>" not as a whole token. Instead, it would split the string into characters like "< p a d >". I fixed the problem by modifying the code at `` as below:
```python
--- a/BERTweet/transformers/tokenization_bertweet.py
+++ b/BERTweet/transformers/tokenization_bertweet.py
@@ -242,9 +242,14 @@ class BertweetTokenizer(PreTrainedTokenizer):
             text = self.normalizeTweet(text)
         return self.bpe.apply([text])[0].split()

-    def convert_tokens_to_ids(self, tokens):
-        """ Converts a list of str tokens into a list of ids using the vocab."""
-        return self.vocab.encode_line(" ".join(tokens), append_eos=False, add_if_not_exist=False).long().tolist()
+    def _convert_token_to_id(self, token):
+        #""" Converts a list of str tokens into a list of ids using the vocab."""
+        #return self.vocab.encode_line(" ".join(tokens), append_eos=False, add_if_not_exist=False).long().tolist()
+        return self.vocab.encode_line(token, append_eos=False, add_if_not_exist=False).long().tolist()[0]
+
+    @property
+    def vocab_size(self) -> int:
+        return len(self.vocab)
```
From my understanding, to encode a sentence, the order of the interfaces called in this case are `PreTrainedTokenizerBase::encode`
->`PreTrainedTokenizer::_encode_plus`
->`PreTrainedTokenizer::convert_tokens_to_ids`
->`PreTrainedTokenizer::_convert_token_to_id_with_added_voc`
->`BertweetTokenizer::_convert_token_to_id` for non-special tokens or `PreTrainedTokenizer::added_tokens_encoder` for special tokens.
So in the class `BertweetTokenizer`, it should implement the interface `_convert_token_to_id` rather than `convert_tokens_to_ids`.

I will have a look soon. Thanks @Miopas.

**I have just tried "BertweetTokenizer" and got this error:**

"ImportError: cannot import name 'BertweetTokenizer' from 'transformers' (/home/apps/anaconda3/lib/python3.7/site-packages/transformers/__init__.py)"

**Is there any solution to it?**

**I have also tried:**

tokenizer2 = BertTokenizer.from_pretrained("vinai/bertweet-base")
trained = tokenizer2.encode("oops!! pelosi & dems admit numbers submitted to cbo are false! someurl #tcot #tlot #sgp #hcr #p2")

and got:
trained = [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]

**Is there any solution to it?**

thks!

Hi @LysandreJik @JetRunner 
It's for your information both PhoBERT and BERTweet are now can be used in the Auto mode and without an external dependency fastBPE. Please help review this pull request (7/8 successful checks). Thanks a lot. 

@justinphan3110 @Miopas @SergioBarretoJr Please update the repository. Both models should work now. Thanks.

[run_tests_torch_and_tf.output.txt](https://github.com/huggingface/transformers/files/5174353/run_tests_torch_and_tf.output.txt)

Hi @LysandreJik @JetRunner @julien-c @sshleifer 
I am wondering whether I can get a support from huggingface to incorporate BERTweet and PhoBERT into the `transformers` master branch ?
There is only a failed test of `FAILED tests/test_modeling_marian.py::TestMarian_en_ROMANCE::test_pipeline - ...` for `run_tests_torch_and_tf` which is not related to BERTweet and PhoBERT, thus out of my control. So my pull request could not pass this test (please see details in the attachment file). Please could you help review my pull request?
Thank you very much.


Dear @LysandreJik, 

Please can you kindly help to add the PhoBERT model as I really want to use it with your great `transformers` tool in a Vietnamese text challenge?  


@datquocnguyen Thanks for your contribution. We discussed internally and given that the modeling part of both BERTweet and PhoBERT is out-of-the-box RoBERTa, we would like to avoid duplicating model files, and instead support them by leveraging https://github.com/huggingface/transformers/pull/6995

i.e. we would only need to add files for the tokenizers (and associated tests)

Potentially we could also help to make those new tokenizers more general/configurable. What do you think?


@julien-c That sounds a nice idea. 
Please inform me when the new configuration type is integrated into the master branch. 
I will then adapt our tokenizers & config files for it. Thanks!

Hi @datquocnguyen, the PR @julien-c linked is now merged!

This should greatly simplify your PR, in that you only need to contribute your tokenizers as well as their tests. Let us know if you can make the change!

Hi @LysandreJik thanks for your information. 
Yes, I will make the change soon (it should be done early next week).

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6129?src=pr&el=h1) Report
> Merging [#6129](https://codecov.io/gh/huggingface/transformers/pull/6129?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b0cbcdb05b39e6c81db049d2b4d7dfc5d823210d?el=desc) will **decrease** coverage by `0.24%`.
> The diff coverage is `71.14%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6129/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6129?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6129      +/-   ##
==========================================
- Coverage   80.32%   80.08%   -0.25%     
==========================================
  Files         168      170       +2     
  Lines       32285    32642     +357     
==========================================
+ Hits        25932    26140     +208     
- Misses       6353     6502     +149     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6129?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bertweet.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydHdlZXQucHk=) | `63.18% <63.18%> (ø)` | |
| [src/transformers/tokenization\_phobert.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcGhvYmVydC5weQ==) | `83.45% <83.45%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.34% <100.00%> (+<0.01%)` | :arrow_up: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `92.06% <100.00%> (+0.26%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90NS5weQ==) | `26.05% <0.00%> (-63.52%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `71.84% <0.00%> (-23.17%)` | :arrow_down: |
| [src/transformers/modeling\_lxmert.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19seG1lcnQucHk=) | `70.01% <0.00%> (-20.75%)` | :arrow_down: |
| [src/transformers/modeling\_transfo\_xl\_utilities.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsX3V0aWxpdGllcy5weQ==) | `52.98% <0.00%> (-13.44%)` | :arrow_down: |
| [src/transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsLnB5) | `67.10% <0.00%> (-12.67%)` | :arrow_down: |
| [src/transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcm9iZXJ0YS5weQ==) | `87.67% <0.00%> (-10.96%)` | :arrow_down: |
| ... and [21 more](https://codecov.io/gh/huggingface/transformers/pull/6129/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6129?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6129?src=pr&el=footer). Last update [b0cbcdb...257b9f1](https://codecov.io/gh/huggingface/transformers/pull/6129?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@datquocnguyen can you also upload your model files on https://huggingface.co/vinai/bertweet-base

I still get this error: 

> ⚠️ Model name 'vinai/bertweet-base' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed 'vinai/bertweet-base' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.


@datquocnguyen I looked a the PR and looking forward to this merge. I have a few suggestions:

1. I find the Phobert and Bertweet models to be quite similar. This makes the tokenizers also similar so we should not need a seperate tokenizer for both. Given that both these tokenizers just load fastBPE tokenizer data format, we can simply call them fastBPETokenizer. 

2. Looking at this other code which also uses fastBPE <sup>[1]</sup> can't we just follow it to convert the fastBPE tokenizer files to the huggingface format. 

   - You can easily convert your `bpe.codes` into `merges.txt` file and then use the Roberta tokenizer. 
   - The format is the same and you only need to drop the 3rd column in your BPE.codes and add a top line for comment. 
   - In your code you are not even using the last column values. 
   - Your `merges.txt` can have the following as the first line `#version: 1`  (look at merges.txt file of Roberta <sup>[2]</sup>)

[1]: https://github.com/huggingface/transformers/blob/b23d3a5ad4aa08decd10671f85be5950767dd052/model_cards/allegro/herbert-klej-cased-v1/README.md
[2]: https://huggingface.co/roberta-base#list-files

Hi @napsternxg The model had been already uploaded to https://huggingface.co/vinai/bertweet-base. For now, you would have to install `transformers` from our development branch (as it has not merged to the master branch of `transformers` yet). Did you try the following commands?

 -  Python version >= 3.6
 - [PyTorch](http://pytorch.org/) version >= 1.4.0
 -  Install `transformers` from our development branch:
    - `git clone https://github.com/datquocnguyen/transformers.git`
    - `cd transformers`
    - `pip install --upgrade .`
 - Install `emoji`: `pip3 install emoji`

Thanks for your suggestions. BertweetTokenizer is specifically designed to work on Tweet data, incorporating a TwitterTokenizer while PhoBERT does not. Note that both our `vocab.txt` and `bpe.codes` are also used in loading our models in `fairseq`. So I would prefer to keep them intact rather than converting them into another format. 

Btw, I should mention that BERTweet is accepted as an EMNLP-2020 demo paper while PhoBERT gets a slot in the Findings of EMNLP-2020 volume. Please help review this pull request so that others might benefit from using them directly from the master branch of `transformers`. Thanks. @LysandreJik @JetRunner @julien-c
All checks have passed and you only need to merge files for the tokenizers and associated tests.

Thanks that makes sense. 
@datquocnguyen I was trying to use it from the models website. 
My suggestion on the bpe.codes file was not to remove it but to generate the merges.txt file from it, which will make it compatible with the huggingface tokenizer. 

@napsternxg Please remove your "transformers" cache folder from `~/.cache/torch` and reinstall `transformers` from our development branch. I am sure that `bertweet`  would work smoothly:

```python
import torch
from transformers import AutoModel, AutoTokenizer

bertweet = AutoModel.from_pretrained("vinai/bertweet-base")
tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base")

# INPUT TWEET IS ALREADY NORMALIZED!
line = "SC has first two presumptive cases of coronavirus , DHEC confirms HTTPURL via @USER :cry:"

input_ids = torch.tensor([tokenizer.encode(line)])

with torch.no_grad():
    features = bertweet(input_ids)  # Models outputs are now tuples
```

@datquocnguyen great work and I am looking forward to seeing the PR gets merged so that I can use the models directly from the huggingface transformers.

Will merge today unless @julien-c, @JetRunner have comments.

LGTM, do not hesitate to make the tokenizers as generic/configurable as possible, but this can be in a subsequent PR

Any news on it? when Phobert available on HuggingFace? 

It's been available since September:

```py
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base")

model = AutoModelForMaskedLM.from_pretrained("vinai/phobert-base")
```

You can see the model card [here](https://huggingface.co/vinai/phobert-base).

> It's been available since September:
> 
> ```python
> from transformers import AutoTokenizer, AutoModelForMaskedLM
> 
> tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base")
> 
> model = AutoModelForMaskedLM.from_pretrained("vinai/phobert-base")
> ```
> 
> You can see the model card [here](https://huggingface.co/vinai/phobert-base).

But i don't see here https://huggingface.co/transformers/pretrained_models.html
How can i integrate with Rasa NLU sir?
Thank you

PhoBERT is based off of the RoBERTa implementation, so you can load it in a `RobertaForMaskedLM` model. The tokenizer is custom, so you should load it through the `PhobertTokenizer`.

I have never used Rasa NLU, so I can't help you much here. Your best option would be to open a thread on our [forum](https://discuss.huggingface.co) with an example of how you do things for other models, so as not to flood this PR.

You can ping me on the thread (@lysandre).