Add dropouts to GPT-NeoX
Author: ZHAOTING
Pull request: yes

# What does this PR do?

The current GPT-NeoX modeling code does not contain dropouts as in [the original EleutherAI/gpt-neox code](https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model), possibly because that GPT-NeoX 20B, where the HF gpt-neox implementation was applied for the first time, has disabled all dropouts.

However, EleutherAI/gpt-neox does provide dropouts at several places.
* post-word-embedding dropout, [reference](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/model/word_embeddings.py#L156),
* attention score dropout, [reference](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/model/transformer.py#L453C1-L453C1),
* post-attention dropout, [reference1](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/model/transformer.py#L829-L834), [reference2](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/model/transformer.py#L865-L870), [reference3](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/model/transformer.py#L873-L880),
* post-mlp dropout, [reference1](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/model/transformer.py#L839-L844), [reference2](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/model/transformer.py#L893-L898).

These dropouts can be turned on and help produce better fine-tuning performance.

This PR adds corresponding dropouts to the HF gpt_neox implementation. Following the original EleutherAI code, dropout probabilities are controlled by two config arguments:
* attention_dropout, which controls the probability of the attention score dropout, [reference](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/neox_arguments/neox_args.py#L911),
* hidden_dropout, which controls the probability of remaining dropouts, [reference](https://github.com/EleutherAI/gpt-neox/blob/2534e3d76e320aba095894e7dc2a4b416a1ac8df/megatron/neox_arguments/neox_args.py#L916).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Sorry that I am not sure whom to tag, so I am following the suggestion to tag text model people @ArthurZucker and @younesbelkada.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Glad to make my first PR to transformers!