[WIP] Add Theseus Compression
Author: JetRunner
Pull request: yes
Labels: name: WIP

`transformers.theseus` provides the implementation for BERT-of-Theseus, LayerDrop and Mixout.

Original BERT-of-Theseus authors: @JetRunner @MichaelZhouwang


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5903?src=pr&el=h1) Report
> Merging [#5903](https://codecov.io/gh/huggingface/transformers/pull/5903?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/492bb6aa486856f8243dfeb533ed1b23e996e403?el=desc) will **decrease** coverage by `2.80%`.
> The diff coverage is `83.75%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5903/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5903?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5903      +/-   ##
==========================================
- Coverage   80.12%   77.31%   -2.81%     
==========================================
  Files         169      152      -17     
  Lines       32317    26290    -6027     
==========================================
- Hits        25893    20326    -5567     
+ Misses       6424     5964     -460     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5903?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/theseus/theseus\_list.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90aGVzZXVzL3RoZXNldXNfbGlzdC5weQ==) | `67.64% <67.64%> (ø)` | |
| [src/transformers/theseus/theseus\_module.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90aGVzZXVzL3RoZXNldXNfbW9kdWxlLnB5) | `88.88% <88.88%> (ø)` | |
| [src/transformers/theseus/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90aGVzZXVzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/theseus/layerdrop\_list.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90aGVzZXVzL2xheWVyZHJvcF9saXN0LnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/theseus/mixout\_list.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90aGVzZXVzL21peG91dF9saXN0LnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/theseus/theseus\_errors.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90aGVzZXVzL3RoZXNldXNfZXJyb3JzLnB5) | `100.00% <100.00%> (ø)` | |
| [src/transformers/commands/env.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy9lbnYucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [src/transformers/commands/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy9fX2luaXRfXy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [src/transformers/commands/transformers\_cli.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy90cmFuc2Zvcm1lcnNfY2xpLnB5) | `0.00% <0.00%> (-87.50%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `23.38% <0.00%> (-73.68%)` | :arrow_down: |
| ... and [166 more](https://codecov.io/gh/huggingface/transformers/pull/5903/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5903?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5903?src=pr&el=footer). Last update [492bb6a...20dd34f](https://codecov.io/gh/huggingface/transformers/pull/5903?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


which examples have you tested this with?

> which examples have you tested this with?

run_glue and run_ner but more to come!

Want an early review for this? Would be wonderful! And it can save me some time before I do the docs. @sshleifer 

> run_glue and run_ner but more to come! 

:heart_eyes: can't wait to re-fine-tune my NER models :hugs: 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Well I've been really busy recently but don't close it for me stalebot!

Thanks for reopening it @LysandreJik! I was stuck with some details and I'll probably get it done soon.

Sounds great, looking forward to it!

Will move to an independent package. Closing this.