change layernorm code to pytorch's native layer norm
Author: dhpollack
Pull request: yes

The current code basically recreates pytorch's native [LayerNorm](https://pytorch.org/docs/stable/nn.html#layernorm) code.  The only difference is that the default eps in the pytorch function is 1e-5 instead of 1e-12.  PyTorch's native version is optimized for cudnn so it should be faster than this version.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089?src=pr&el=h1) Report
> Merging [#1089](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/e00b4ff1de0591d5093407b16e665e5c86028f04?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1089      +/-   ##
==========================================
+ Coverage   79.61%   79.66%   +0.04%     
==========================================
  Files          42       42              
  Lines        6898     6898              
==========================================
+ Hits         5492     5495       +3     
+ Misses       1406     1403       -3
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfYmVydC5weQ==) | `87.98% <ø> (ø)` | :arrow_up: |
| [pytorch\_transformers/file\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZmlsZV91dGlscy5weQ==) | `73.94% <0%> (+2.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089?src=pr&el=footer). Last update [e00b4ff...e13465f](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1089?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I think your PR misses the point though? The models need to be 100% accurate reproductions of the Tensorflow code, right down to differences in eps values. Otherwise if you run the activations and get different results, you don't know whether there's a bug. You also can't reason about different results, and whether they matter.

@honnibal but looking at the code, every call of `BertLayerNorm` explicitly sets the eps, thus the actual values used in the BERT models does not change.  Only the default value, but this default value is never used.  Additionally, if APEX is available then you use `FusedLayerNorm`, which uses the [same default eps](https://github.com/NVIDIA/apex/blob/master/apex/normalization/fused_layer_norm.py#L70) of 1e-5 as the pytorch default `LayerNorm`.  So you already have an inconsistency, but you solved this by explicitly setting the eps every time you use the layer.

Oh right! Fair point, sorry.

Yes @dhpollack is right we can switch to PyTorch official LayerNorm.

What made me reimplement the LayerNorm when I was working on Bert last year was actually a typo in PyTorch's doc formula for computing the LayerNorm which indicated, at that time, that the epsilon was added to the square root of the variance instead of being added to the variance it-self. This typo is now corrected in https://github.com/pytorch/pytorch/pull/8545.

Everything is right and we can drop these custom LayerNorms.

Are we sure the names of the parameters are the same though? (`eps` vs. `variance_epsilon`)