ADD ERNIE model
Author: nghuyong
Pull request: yes
Labels: name: model card

ERNIE is a serial model released by Baidu including:
ERNIE1.0: [Ernie: Enhanced representation through knowledge integration](https://arxiv.org/abs/1904.09223)
ERNIE2.0: [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://arxiv.org/abs/1907.12412)
ERNIE-tiny: Use data distillation from ERNIE 2.0     
These models have state-of-the-art performances on NLU tasks, especially on Chinese tasks.
So, ERNIE is a very important model in the transformer model family.

Actually, ERNIE model is the same structure as BERT including the above three ERNIE viersions, so we don't add a new model, and just need to convert the weight. 
> Note that: ERNIE2.0 introduces a task embedding, but the official released version doesn't have this embedding weight, and  all released weights are the same with BERT.

I have successfully converted  ERNIE models into PyTorch version and do serial experiments to check the conversion is the same with the original paddlepaddle implement.
More detail: https://github.com/nghuyong/ERNIE-Pytorch

In this PR, I directly add ERNIE model to BERT model related files.
This PR is link to issue: [#issue5117](https://github.com/huggingface/transformers/issues/5117),[#issue928](https://github.com/huggingface/transformers/issues/928) and [#issue514](https://github.com/huggingface/transformers/issues/514)


Comments:
We shouldn't have to add those because the library should work out of the box with the models at https://huggingface.co/nghuyong

The discoverability of those models on huggingface.co is a different thing and we're open to suggestions to improve it

For instance, you should upload model cards describing the models including metadata etc. See https://huggingface.co/docs for instance

Pinging @JetRunner and @sshleifer for feedback

Thanks for your contribution!

Yep, I think this PR is not very necessary here since ERNIE serves as weights for BERT. We have shifted into a "model hub" model so I can't think of a reason to keep hard-coding urls here (and we think we should even remove the existing ones at some point) @julien-c  

Also, what do you mean by "especially on Chinese"? Aren't these checkpoints in English I think?

We should add some docs and tweet!

You could copy https://github.com/nghuyong/ERNIE-Pytorch/blob/master/Readme.md to each model card.


@JetRunner 
ERNIE1.0 is for Chinese, more performance detail: https://arxiv.org/abs/1904.09223
ERNIE2.0 and ERNIE-tiny are for English.

> @JetRunner 
> 
> ERNIE1.0 is for Chinese, more performance detail: https://arxiv.org/abs/1904.09223
> 
> ERNIE2.0 and ERNIE-tiny are for English.

Yeah, this raises some confusion here. Thanks for your clarification!

@julien-c @JetRunner @sshleifer 
I have withdrew my previous submission, and add model_card in this new commit.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5763?src=pr&el=h1) Report
> Merging [#5763](https://codecov.io/gh/huggingface/transformers/pull/5763?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8ab565a4be5a7fd96b19ef88d474037ef31f27e5&el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5763/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5763?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5763      +/-   ##
==========================================
- Coverage   77.32%   77.24%   -0.09%     
==========================================
  Files         146      146              
  Lines       26047    26047              
==========================================
- Hits        20141    20120      -21     
- Misses       5906     5927      +21     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5763?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5763/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5763/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `97.11% <0.00%> (+0.28%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5763/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5763?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5763?src=pr&el=footer). Last update [8ab565a...0c8a23f](https://codecov.io/gh/huggingface/transformers/pull/5763?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @nghuyong! The model cards look awesome.
We'll take it from here and add some metadata and then merge the model cards. 

@JetRunner  Thanks

Hi @nghuyong ,
I just tried [your script](https://github.com/nghuyong/ERNIE-Pytorch#reproduce-ernie-papers-case), and found that
```
Some weights of BertForMaskedLM were not initialized from the model checkpoint at nghuyong/ernie-1.0 and are newly initialized: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

This error is very common if you upload a model that was loaded by `BertModel`. Could you re-upload these checkpoints by loading them in `BertForMaskedLM` first? That would be something like

```python
tokenizer = BertTokenizer.from_pretrained('./convert')
model = BertForMaskedLM.from_pretrained('./convert')
# instead of `model = BertModel.from_pretrained('./convert')`

model.save_pretrained('./saved')
```
This will allow users to directly use the checkpoints to do mask filling without fine-tuning them. Also, could you help convert the checkpoints to TensorFlow as well? That would be super easy. We have a tutorial here: https://huggingface.co/transformers/model_sharing.html

Thank you!

@JetRunner  OK，I will update soon

@JetRunner have updated now~