Funnel transformer
Author: sgugger
Pull request: yes

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #4844 

This PR adds the Funnel Transformer architecture in PyTorch. For now, I have uploaded two of the ten checkpoints for this model, I will convert and upload the other ones while this PR is under review and add them before it's merged.

Due to the fact there are two versions of the Funnel model (one that returns hidden states with a sequence length divided by 4 and one that return hidden states with the same sequence length, that has 2 more layers), I add to make to different Tester in the test file (because the expected number of hidden states / attentions change depending on which model is used). I adapted the script that check all models are tested to account for that.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6908?src=pr&el=h1) Report
> Merging [#6908](https://codecov.io/gh/huggingface/transformers/pull/6908?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0f360d3d1c606d6d79cdf1efa53c3d719249573d?el=desc) will **increase** coverage by `0.71%`.
> The diff coverage is `87.71%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6908/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6908?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6908      +/-   ##
==========================================
+ Coverage   80.23%   80.95%   +0.71%     
==========================================
  Files         161      164       +3     
  Lines       30119    30925     +806     
==========================================
+ Hits        24167    25035     +868     
+ Misses       5952     5890      -62     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6908?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/commands/convert.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy9jb252ZXJ0LnB5) | `26.98% <20.00%> (-0.61%)` | :arrow_down: |
| [src/transformers/modeling\_funnel.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19mdW5uZWwucHk=) | `86.76% <86.76%> (ø)` | |
| [src/transformers/tokenization\_funnel.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZnVubmVsLnB5) | `97.67% <97.67%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.31% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `93.47% <100.00%> (+0.14%)` | :arrow_up: |
| [src/transformers/configuration\_funnel.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2Z1bm5lbC5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `78.97% <100.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `97.87% <100.00%> (+2.22%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `25.13% <0.00%> (-73.83%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `72.25% <0.00%> (-10.00%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/huggingface/transformers/pull/6908/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6908?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6908?src=pr&el=footer). Last update [0f360d3...8c684cc](https://codecov.io/gh/huggingface/transformers/pull/6908?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome! The model seems quite complex so I didn't really understand all the functionality. 
A couple of things from my side:

1) IMO, it's super useful to have hard  coded integration tests in the test file which makes the model a lot easier to maintain (every change can quickly be checked by making sure the model stays mathematically equivalent).

2) I guess a couple of comments and assert statements would be nice to make the code a bit easier to understand

3) Personally, I don't like single letter variables. Search replace commands don't work on such variables and it is very difficult to understand what they mean. 

Thanks for all the comments. I think I replied/addressed all of them except the fast small integration tests, which are going to take a bit more work (starting on this now). Let me know if I missed anything since there are a lot of comments!

All checkpoints uploaded so I updated the incomplete lists. Also added mention of the model in all indexes, the model summary and the big table of pretrained models (sorry about the diff on that file, Funnel Transformer is one character too long and required to add an extra space on every line).

Should be good to merge at the beginning of next week!

@sgugger although you've named the models "`funnel-base`", "`funnel-medium`" so on so forth, the paper talks about all this in a different format, could a docstring be added saying `funnel-base` is `B4-4-4H768` and same for the rest. If someone wants to replicate the papers' results that would be great.

edit: my bad, its there in the comments next to the model name, but still would be better in a docstring too. Sorry!