[WIP] Add UniLM model
Author: addf400
Pull request: yes

# Typical workflow for including a model

Here an overview of the general workflow: 

- [x] add model/configuration/tokenization classes
- [x] add conversion scripts
- [x] add tests
- [x] finalize

Let's detail what should be done at each step

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the files to replace `XXX` (with various casing) with your model name
- [x] copy-paste or create a simple configuration class for your model in the `configuration_...` file
- [x] copy-paste or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0)
- [x] copy-paste or create a tokenizer class for your model in the `tokenization_...` file

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [x] copy the conversion script (`convert_...`) from the present folder to the main folder.
- [x] edit this script to convert your original checkpoint weights to the current pytorch ones.

# Adding tests:

Here is the workflow for the adding tests:

- [x] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [x] edit the tests files to replace `XXX` (with various casing) with your model name
- [x] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] add import for all the relevant classes in `__init__.py`
- [x] add your configuration in `configuration_auto.py`
- [x] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [x] add your tokenizer in `tokenization_auto.py`
- [x] add your models and tokenizer to `pipeline.py`
- [x] add a link to your conversion script in the main conversion utility (currently in `__main__` but will be moved to the `commands` subfolder in the near future)
- [x] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
- [x] add a mention of your model in the doc: `README.md` and the documentation itself at `docs/source/pretrained_models.rst`.
- [x] upload the pretrained weigths, configurations and vocabulary files.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=h1) Report
> Merging [#2160](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f061606277322a013ec2d96509d3077e865ae875?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `49.69%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2160/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2160      +/-   ##
==========================================
+ Coverage   80.32%   80.37%   +0.04%     
==========================================
  Files         122      127       +5     
  Lines       18342    19000     +658     
==========================================
+ Hits        14734    15272     +538     
- Misses       3608     3728     +120
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `88.91% <15.38%> (-2.55%)` | :arrow_down: |
| [transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fYXV0by5weQ==) | `44.68% <33.33%> (-0.78%)` | :arrow_down: |
| [transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9hdXRvLnB5) | `59.18% <33.33%> (-1.69%)` | :arrow_down: |
| [transformers/modeling\_unilm.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3VuaWxtLnB5) | `36.95% <36.95%> (ø)` | |
| [transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2F1dG8ucHk=) | `38.79% <53.84%> (+1.89%)` | :arrow_up: |
| [transformers/tests/tokenization\_unilm\_test.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL3Rva2VuaXphdGlvbl91bmlsbV90ZXN0LnB5) | `55% <55%> (ø)` | |
| [transformers/configuration\_unilm.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fdW5pbG0ucHk=) | `87.09% <87.09%> (ø)` | |
| [transformers/tests/modeling\_unilm\_test.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3VuaWxtX3Rlc3QucHk=) | `93.75% <93.75%> (ø)` | |
| [transformers/tokenization\_unilm.py](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91bmlsbS5weQ==) | `94.73% <94.73%> (ø)` | |
| ... and [10 more](https://codecov.io/gh/huggingface/transformers/pull/2160/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=footer). Last update [f061606...bbacc86](https://codecov.io/gh/huggingface/transformers/pull/2160?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thank you for the PR! I edited your post to add the guideline for adding a new model; we'll check the boxes as we go. I'll have a look at the code and come back to you quickly!

> # Typical workflow for including a model
> * [ ]  add your models and tokenizer to `pipeline.py`

@rlouf  Sorry, I didn't find the `pipeline.py` file.

@sshleifer Thanks for the comments! We will merge them into the code.  @addf400 

Let's restart with a new pull request @addf400 @donglixp 

Is anyone still working on this? @addf400 @donglixp @JetRunner 
also @thomwolf from #1530

I'm also looking forward to applying the UniLM model via Huggingface Transformers!
@donglixp @JetRunner @thomwolf 

It seems that this pull request has lasted for a year but still not finished? Is someone still working on it? 

Has this PR for UniLM model been added to Huggingface Transformers?
@donglixp @JetRunner @thomwolf @sshleifer

Hey @ontocord , I think it the "minilm" model should work out-of-the-box:

https://github.com/huggingface/transformers/issues/5777

Not sure if you're looking for this model :thinking: 

I haven't tried it yet, but the recent Microsoft papers (on language modeling) are looking really promising!

Thanks @stefan-it. I don't think MiniLM and UniLM are the same thing, altough it all falls under one project. The MS papers are promising!

I'm also looking forward to applying the unilm model via Huggingface Transformers!

2022 year, still not merged the unilm model into the master branch.

I'm still looking forward to applying the unilm model via Huggingface Transformers! 👻👻



I'm still looking forward to applying the unilm model via Huggingface Transformers too!