Implementation of activations as pytorch modules
Author: eldarkurtic
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/15364

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

A few questions relevant to this PR:
1. Are there any known barriers for usage of `inplace` versions of these activation functions? If you are not aware of any, I can push an update for this, so we could squeeze in some additional performance gains. 
2. Given that currently all implemented activation functions are stateless, it’s safe to initialize them directly in the `ACT2FN`. If at some point activation functions with states are introduced (or something similar to what @stas00 mentioned in https://github.com/huggingface/transformers/issues/15364#issuecomment-1026240845), this approach has to be updated by initializing these activations during the model construction or via `def get_activation` function. This update would be needed even without this PR, when activations were implemented as functions.
3.  Are `quick_gelu` and `gelu_fast` implementations necessary now that the pytorch has a built-in implementation (`nn.functional.gelu`) which should be sufficiently fast?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> I visually checked that nothing was left out - I wonder whether we have all these activation functions tested - have you by chance grepped if all entries in the ACT2FN table appear at least once in the tests?

I've added a few tests that were missing for activations, so now they should all appear at least once.

> (note for future PRs: in general changing the order of the original code chunks and doing changes to it at the same time makes it much more difficult to review, but no need to change anything as I have already verified it - just had to scroll up and down a lot)

Thanks a lot for the tip, will definitely take care about it in the future.

And regarding the inplace and jitted versions, I agree it's better to move this in a new thread. I will try to benchmark these variations so we can have some numbers to help us decide about the best way to proceed. 

The error is unrelated.

Merging now, will pay attention to the slow tests to ensure nothing is changed due to this.

This seems to have broken the `run_translation` script:

```
============================= FAILURES SHORT STACK =============================
______________________ ExamplesTests.test_run_translation ______________________


self = <[AttributeError("'SiLUActivation' object has no attribute '_modules'") raised in repr()] SiLUActivation object at 0x7f5db0d053d0>
name = '_parameters'

    def __getattr__(self, name: str) -> Union[Tensor, 'Module']:
        if '_parameters' in self.__dict__:
            _parameters = self.__dict__['_parameters']
            if name in _parameters:
                return _parameters[name]
        if '_buffers' in self.__dict__:
            _buffers = self.__dict__['_buffers']
            if name in _buffers:
                return _buffers[name]
        if '_modules' in self.__dict__:
            modules = self.__dict__['_modules']
            if name in modules:
                return modules[name]
>       raise AttributeError("'{}' object has no attribute '{}'".format(
            type(self).__name__, name))
E       AttributeError: 'SiLUActivation' object has no attribute '_parameters'

/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:1[17](https://github.com/huggingface/transformers/runs/5240564468?check_suite_focus=true#step:5:17)7: AttributeError
```

This is fixed by #15718 (it's currently breaking the inference API on some models).