Add Dutch pre-trained BERT model
Author: wietsedv
Pull request: yes

We trained a Dutch cased BERT model at the University of Groningen.

Details are on [Github](https://github.com/wietsedv/bertje/) and [Arxiv](https://arxiv.org/abs/1912.09582).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2281?src=pr&el=h1) Report
> Merging [#2281](https://codecov.io/gh/huggingface/transformers/pull/2281?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ba2378ced560c12f8ee97ca7998fd28b93fcfb47?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2281/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2281?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2281   +/-   ##
=======================================
  Coverage   74.45%   74.45%           
=======================================
  Files          85       85           
  Lines       14610    14610           
=======================================
  Hits        10878    10878           
  Misses       3732     3732
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2281?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2281/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `96.34% <ø> (ø)` | :arrow_up: |
| [src/transformers/configuration\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2281/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2JlcnQucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2281/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `96.26% <ø> (ø)` | :arrow_up: |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2281/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `87.7% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2281?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2281?src=pr&el=footer). Last update [ba2378c...5eb71e6](https://codecov.io/gh/huggingface/transformers/pull/2281?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi @wietsedv! Did you try loading your tokenizer/model directly, cf. https://huggingface.co/wietsedv/bert-base-dutch-cased

i.e. It should work out-of-the-box using:
```python
tokenizer = AutoTokenizer.from_pretrained("wietsedv/bert-base-dutch-cased")

model = AutoModel.from_pretrained("wietsedv/bert-base-dutch-cased")
tf_model = TFAutoModel.from_pretrained("wietsedv/bert-base-dutch-cased")
```

Let us know if it's not the case (we can still merge this PR to have a nice shortcut inside the code but feature-wise it should be equivalent)

Hi! Thanks for your response. I did notice that I can use that snippet and I can confirm it works. I am however not sure whether cased tokenization works correctly that way. Correct me if I am wrong, but it seems that Transformers always lowercases unless that is explicitly disabled. I did disable it in this PR, which makes it work correctly out of the box. But I think lowercasing is enabled by default if people use the snippet above.

Please correct my if I am wrong.

We'll check, thanks for the report.

Just a note: I absolutely love seeing a Dutch version of BERT but this isn't the only BERT model out there. As you mention in your paper, there's also [BERT-NL](http://textdata.nl/). You seem to claim that it performs a lot worse than your version and that it is even outperformed by multilingual BERT. At first glance I don't see any written-down experiments confirming that claim - a comparison between BERTje, BERT-NL, and multilingual BERT on down-stream tasks would've been much more informative. (BERT-NL will be presented at the largest computational linguistics conference on Dutch (Computational Linguistics in the Netherlands; CLIN) at the end of the month, so presumably it does carry some weight.)

All this to say: why does your version deserve to be "the" `bert-base-dutch-cased` model if there is an alternative? Don't get me wrong, I really value your research, but a fair and full comparison is missing.

It is correct that there is another Dutch model and we do full fine-tuning results of this model. These numbers were included in an earlier draft of our paper, but we removed it since they did not add any value. For instance for named entity recognition (conll2002), multilingual BERT achieves about 80% accuracy, our BERT model about 88% and their BERT model just 41%.

More detailed comparison with their model would therefore not add any value since the scores are too low. The authors have not made any claims about model performance yet, so it would have been unfair to be too negative about their model before they have even released any paper.

Someone at Leiden has confirmed that their experiments also showed that they were outperformed by multilingual BERT. Therefore I think that our BERT model is the only _effective_ Dutch BERT model.

PS: The entry barrier for CLIN is not extremely high. The reason we are not presenting at CLIN is that we missed the deadline. 

I think it is in fact very useful information to see such huge difference. This way readers are not confronted with the same question that I posed before: there are two Dutch BERT models - but which one should I use/which one is better? You now clarify it to me, for which I am grateful, but a broader audience won't know. I think the added value is great. However, I do agree that is hard and perhaps unfair to compare to a model that hasn't been published/discussed yet. (Then again, their model is available online so they should be open to criticism.)

The CLIN acceptance rate is indeed high, and the follow-up CLIN Journal is also not bad. Still, what I was aiming for is peer review. If I were to review the BERT-NL (assuming they will submit after conference and assuming I was a reviewer this year), then I would also mention your model and ask for a comparison. To be honest, I put more faith in a peer reviewed journal/conference than arXiv papers.

I really don't want to come off as arrogant and I very much value your work, but I am trying to approach this from a person who is just getting started with this kind of stuff and doesn't follow the trends or what is going on in the field. They might find this model easily available in the Transformers hub, but then they might read in a journal (possibly) about BERT-NL - which then is apparently different from the version in Transformers. On top of that, neither paper (presumably) refers or compares to the other! Those people _must be confused_ by that.

The above wall of text just to say that I have no problem with your model being "the" Dutch BERT model because it seems to clearly be the best one, but that I would very much like to see reference/comparison to the other model in your paper so that it is clear to the community what is going on with these two models. I hope that the authors of BERT-NL do the same. Do you have any plans to submit a paper somewhere?

Thanks for your feedback and clear explanation. We do indeed intend to submit a long paper somewhere. The short paper on arxiv is mainly intended for reference and to demonstrate that the model is effective with some benchmarks. Further evaluation would be included in a longer paper.

The authors of BERT-NL have reported results which can be compared to Bertje, see https://twitter.com/suzan/status/1200361620398125056
Also see the results in this thread https://twitter.com/danieldekok/status/1213378688563253249 and https://twitter.com/danieldekok/status/1213741132863156224
In both cases, Bertje does outperform BERT-NL. On the other hand, the results about Bertje vs multilingual BERT are different, so this needs to be investigated further.

I think @BramVanroy raises some good points about naming convention.

In my opinion the organization name or author name should come after the "bert-base-<language>" skeleton.

So it seems that BERTje is current SOTA now. But: on the next conference maybe another BERT model for Dutch is better... I'm not a fan of the "First come, first served" principle here 😅

/cc @thomwolf , @julien-c 

> On the other hand, the results about Bertje vs multilingual BERT are different, so this needs to be investigated further.

Agreed. Tweeting results of doing "tests" is one thing, but actual thorough investigating and reporting is something else. It has happened to all of us that you quickly wanted to check something and only later realized that you made a silly mistake. (My last one was forgetting the `-` in my learning rate and not noticing it, oh boy what a day.) As I said before I would really like to a see a thorough, reproducible comparison of BERTje, BERT-NL, and multilingual BERT, and I believe that that should be the basis of any new model. Many new models sprout from the community grounds - and that's great! - but without having at least _some_ reference and comparison, it is guess-work trying to figure out which one is best or which one you should use.

> I think @BramVanroy raises some good points about naming convention.
> 
> In my opinion the organization name or author name should come after the "bert-base-" skeleton.
> 
> So it seems that BERTje is current SOTA now. But: on the next conference maybe another BERT model for Dutch is better... I'm not a fan of the "First come, first served" principle here 😅
> 
> /cc @thomwolf , @julien-c

Perhaps it's better to just make the model available through the user and that's all? In this case, only make it available through `wietsedv/bert-base-dutch-cased` and not `bert-base-dutch-cased`? That being said, where do you draw the line of course. Hypothetical question: why does _a Google_ get the rights to make its weights available without a `google/` prefix, acting as a "standard"? I don't know how to answer that question, so ultimately it's up to the HuggingFace people.

I'm also not sure how diverging models would then work. If for instance you bring out a German BERT-derivative that has a slightly different architecture, or e.g. a different tokenizer, how would that then get integrated in Transformers? (For example, IIRC BERTje uses SOP instead of NSP, so that may lead to more structural changing in the available heads than just different weights.)


I inherently completely agree with your points. I think the people at Huggingface are trying to figure out how to do this, but they have not been really consistent. Initially, the "official" models within Transformers were only original models (Google/Facebook) and it is unlikely that there would be competetition for better models with exactly the same architecture in English. But for other monolingual models this is different.

I prefer a curated list with pre-trained general models that has more structure than the long community models list. But existing shortcuts should be renamed to be consistent. German for instance has a regular named german and there is one with the `dbmdz` infix. And Finnish has for some reason the `v1` suffix?

My preference would be to always use `institution/` or `institution-` prefixes in the curated list. In Transformers 2.x, the current shortcuts could be kept for backward compatibility but a structured format should be used in the documentation. I think this may prevent many frustrations if even more non-english models are trained and even more people are wanting to use and trust (!) these models.

200% agree with that. That would be the fairest and probably clearest way of doing this. Curating the list might not be easy, though, unless it is curated by the community (like a wiki)? Perhaps requiring a description, website, paper, any other meta information might help to distinguish models as well, giving the authors a chance to explain, e.g., which data their model was trained on, which hyperparameters were used, and how their model differs from others.

I really like http://nlpprogress.com/ which "tracks" the SOTA across different NLP tasks. It is an open source list and anyone can contribute through github. Some kind of lists like this might be useful, but instead discussing the models. 

You all raise excellent questions, many (most?) of which we don’t have a definitive answer to right now 🤗

Some kind of structured evaluation results (declarative or automated) could be a part of the solution. In addition to nlpprogress, sotabench/paperswithcode is also a good source of inspiration.



On the previous point of being able to load the tokenizer's (remote) config correctly:
- I've added a `tokenizer_config.json` to your user namespace on S3: https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/tokenizer_config.json
- We're fixing the support for those remote tokenizer configs in https://github.com/huggingface/transformers/pull/2535 (you'll see that the unit test uses your model). Feedback welcome.

Merging this as we haven't seen other "better" BERT models for Dutch (coincidentally, [`RobBERT`](https://people.cs.kuleuven.be/~pieter.delobelle/robbert/) from @iPieter looks like a great RoBERTa-like model)

Please see [this discussion on model descriptions/README.md](https://github.com/huggingface/transformers/issues/2520#issuecomment-579009439). If you can upload a README.md with eval results/training methods, that'd be awesome.

Thanks!