Make activation functions available from modeling_utils (PyTorch)
Author: BramVanroy
Pull request: yes
Labels: name: wontfix

* This commit replaces references to PyTorch activation functions/modules by a dict of functions that lives in `modeling_utils`. This ensures that all activation functions are available to all modules, praticularly custom functions such as swish and new_gelu.
* In addition, when available (PT1.2) the native PyTorch gelu function will be used - it supports a CPP/CUDA implementation.

**NOTE** that this replaces all `nn.Module`'s by bare functions except for one which was required for testing to be of the type `nn.Module`. If requested, this can be reverted so that only function calls are replaced by ACT2FN functions, and that existing `nn.Module`s are untouched.

**NOTE** that one would thus also expect that _all_ usages of activation functions are taken from `ACT2FN` for consistency's sake.

**NOTE** since the Module counter-part of PyTorch's GeLU [isn't available (yet)](https://github.com/pytorch/pytorch/pull/20665#issuecomment-536359684), it might be worth waiting to implement this pull, and then use Modules and functions in the right places where one would expect, i.e. `Module` when part of architecture, function when processing other kinds of data.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=h1) Report
> Merging [#1371](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ae50ad91ea2fedb64ecd2e7c8e2d0d4778dc03aa?src=pr&el=desc) will **increase** coverage by `0.97%`.
> The diff coverage is `85.71%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1371/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1371      +/-   ##
==========================================
+ Coverage   83.76%   84.74%   +0.97%     
==========================================
  Files          84       84              
  Lines       12596    12559      -37     
==========================================
+ Hits        10551    10643      +92     
+ Misses       2045     1916     -129
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX29wZW5haS5weQ==) | `80.41% <100%> (ø)` | :arrow_up: |
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `72.02% <100%> (+0.77%)` | :arrow_up: |
| [transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2dwdDIucHk=) | `83.88% <100%> (-0.11%)` | :arrow_down: |
| [transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3JvYmVydGEucHk=) | `71.01% <100%> (+5.54%)` | :arrow_up: |
| [transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2Rpc3RpbGJlcnQucHk=) | `95.8% <100%> (-0.03%)` | :arrow_down: |
| [transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RyYW5zZm9feGwucHk=) | `75.16% <100%> (ø)` | :arrow_up: |
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `88.41% <100%> (+0.23%)` | :arrow_up: |
| [transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbS5weQ==) | `88.36% <100%> (-0.07%)` | :arrow_down: |
| [transformers/modeling\_transfo\_xl\_utilities.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RyYW5zZm9feGxfdXRpbGl0aWVzLnB5) | `54.16% <37.5%> (+0.27%)` | :arrow_up: |
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `92.57% <90%> (-0.12%)` | :arrow_down: |
| ... and [7 more](https://codecov.io/gh/huggingface/transformers/pull/1371/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=footer). Last update [ae50ad9...716d783](https://codecov.io/gh/huggingface/transformers/pull/1371?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Unstale. 

Would feel syntactically cleaner if we could do `ACT2FN.gelu()` instead of a dict (also gives some IDE goodness like autocomplete) (I guess through a class or namespace or something), what do you guys think?

> Would feel syntactically cleaner if we could do `ACT2FN.gelu()` instead of a dict (also gives some IDE goodness like autocomplete) (I guess through a class or namespace or something), what do you guys think?

Sounds good but note that this is not something I introduced. The ACT2FN dict already existed, but wasn't used consistently it seemed.

Ah yeah, I see. Would you want to do this change, if you have the time/bandwidth? (+ rebasing on current master so we can merge easily?)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


AFAICT, this has been done by @sshleifer on master. Re-open if necessary!