SqueezeBERT architecture
Author: forresti
Pull request: yes
Labels: name: model card

# Burn-down list:

Here an overview of the general workflow: 

- [x] Add model/configuration/tokenization classes.
- [ ] ~~Add conversion scripts.~~ This model was originally developed in PyTorch.
- [x] Add tests and a @slow integration test.
- [x] Document your model.
- [x] Finalize.

Let's detail what should be done at each step.

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] Copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model
  name.
- [x] Edit the files to replace `XXX` (with various casing) with your model name.
- [x] Copy-paste or create a simple configuration class for your model in the `configuration_...` file.
- [x] Copy-paste or create the code for your model in the `modeling_...` files (PyTorch ~~and TF 2.0~~).
- [x] Copy-paste or create a tokenizer class for your model in the `tokenization_...` file.

### loose ends:
- [ ] support for head_mask, encoder_hidden_states, encoder_attention_mask (Not planning to do for this PR)
- [x] Make sure finetuning works.

## ~~Adding conversion scripts~~

Here is the workflow for the conversion scripts:

- [ ] ~~Copy the conversion script (`convert_...`) from the present folder to the main folder.~~
- [ ] ~~Edit this script to convert your original checkpoint weights to the current pytorch ones.~~

## Adding tests:

Here is the workflow for the adding tests:

- [ ] ~~Copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main
  folder and rename them, replacing `xxx` with your model name.~~
- [ ] ~~Edit the tests files to replace `XXX` (with various casing) with your model name.~~
- [ ] ~~Edit the tests code as needed.~~
- [x] Create tests, using the DistilBERT tests as a starting point
    - [x] `test_modeling_squeezebert.py` (based on `test_modeling_distilbert.py`)
    - [x] `test_tokeization_squeezebert.py` (based on `test_tokenization_distilbert.py`)

## Documenting your model:

Here is the workflow for documentation:

- [x] Make sure all your arguments are properly documented in your configuration and tokenizer.
- [x] Most of the documentation of the models is automatically generated, you just have to make sure that
  `XXX_START_DOCSTRING` contains an introduction to the model you're adding and a link to the original
  article and that `XXX_INPUTS_DOCSTRING` contains all the inputs of your model.
- [x] Create a new page `xxx.rst` in the folder `docs/source/model_doc` and add this file in `docs/source/index.rst`.

Make sure to check you have no sphinx warnings when building the documentation locally and follow our
[documentation guide](https://github.com/huggingface/transformers/tree/master/docs#writing-documentation---specification).

## Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] Add import for all the relevant classes in `__init__.py`.
- [x] Add your configuration in `configuration_auto.py`.
- [x] Add your PyTorch and ~~TF 2.0~~ model respectively in `modeling_auto.py` ~~and `modeling_tf_auto.py`~~.
- [x] Add your tokenizer in `tokenization_auto.py`.
- [ ] ~~Add a link to your conversion script in the main conversion utility (in `commands/convert.py`)~~
- [ ] ~~Edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py`
  file.~~
- [x] Add a mention of your model in...
    - [x] `README.md`
    - [x] `docs/source/index.rst`
    - [x] `docs/source/pretrained_models.rst`.
- [x] Upload the vocabulary files, configurations, and pretrained weights .
    - squeezebert-uncased
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-uncased/vocab.txt
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-uncased/pytorch_model.bin
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-uncased/config.json
    - squeezebert-mnli-headless
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli-headless/vocab.txt
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli-headless/pytorch_model.bin
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli-headless/config.json
    - squeezebert-mnli
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli/vocab.txt
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli/pytorch_model.bin
        - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert-mnli/config.json
- [x] Create model card(s) for your models on huggingface.co (These go in the repo, not in the file upload).

For those last two steps, check the [model sharing documentation](https://huggingface.co/transformers/model_sharing.html).


- [ ] Delete these files (I uploaded these to the wrong directories)
    - https://s3.amazonaws.com/models.huggingface.co/bert/squeezebert/squeezebert/squeezebert-uncased-vocab.txt
    - https://s3.amazonaws.com/models.huggingface.co/squeezebert/squeezebert/squeezebert-uncased-config.json
    - https://s3.amazonaws.com/models.huggingface.co/squeezebert/squeezebert/squeezebert-uncased.bin
    - https://s3.amazonaws.com/models.huggingface.co/squeezebert/squeezebert/squeezebert-mnli.bin
    - https://s3.amazonaws.com/models.huggingface.co/squeezebert/squeezebert/squeezebert-mnli-headless.bin


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7083?src=pr&el=h1) Report
> Merging [#7083](https://codecov.io/gh/huggingface/transformers/pull/7083?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/de4d7b004a24e4bb087eb46d742ea7939bc74644?el=desc) will **increase** coverage by `0.97%`.
> The diff coverage is `97.70%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/7083/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/7083?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #7083      +/-   ##
==========================================
+ Coverage   77.00%   77.98%   +0.97%     
==========================================
  Files         184      184              
  Lines       36734    36216     -518     
==========================================
- Hits        28288    28244      -44     
+ Misses       8446     7972     -474     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/7083?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_squeezebert.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19zcXVlZXplYmVydC5weQ==) | `97.38% <97.38%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.39% <100.00%> (ø)` | |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `96.34% <100.00%> (ø)` | |
| [src/transformers/configuration\_squeezebert.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3NxdWVlemViZXJ0LnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `86.08% <100.00%> (-1.04%)` | :arrow_down: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `92.64% <100.00%> (ø)` | |
| [src/transformers/tokenization\_squeezebert.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fc3F1ZWV6ZWJlcnQucHk=) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9sb25nZm9ybWVyLnB5) | `17.46% <0.00%> (-81.13%)` | :arrow_down: |
| [src/transformers/tokenization\_fsmt.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZnNtdC5weQ==) | `20.34% <0.00%> (-74.90%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `22.03% <0.00%> (-73.03%)` | :arrow_down: |
| ... and [39 more](https://codecov.io/gh/huggingface/transformers/pull/7083/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/7083?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7083?src=pr&el=footer). Last update [de4d7b0...c0521ee](https://codecov.io/gh/huggingface/transformers/pull/7083?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@sgugger @patrickvonplaten - Thanks so much for the feedback! I will make these changes later this week.

@sgugger @patrickvonplaten - I made all the changes that you suggested. Ready to merge?