Add support for Japanese GPT-NeoX-based model by ABEJA, Inc.
Author: SO0529
Pull request: yes

# What does this PR do?
This PR adds a new GPT NeoX Japanese Model and a new Tokenizer. The specific features are,
- Trained for Japanese Dataset with specific preprocess.
- Used [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) to train with Pipe. In addition, we removed bias parameters from the transformer blocks following [PaLM from Google](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html).
- Applied Japanese special sub-word tokenizer to accommodate the distinctive structure of the Japanese. Japanese has a relatively large vocabulary and there is no separation between words. Furthermore, the language is a combination of hiragana, katakana, and kanji, and variants such as "1" and "①" are often used. 

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Thank you in advance to review this PR!
- GPT model and tokenizer: @patrickvonplaten, @LysandreJik
- Documentation: @sgugger


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Impressive PR, @SO0529!

I'd like either @younesbelkada or @ArthurZucker (or both :smile:) to have a look at your PR; they are both well versed in Japanese and have reviewed models in the past.

They're both on leave until the 9th of September, is it okay with you if we review this PR in about 1.5 weeks' time?

Thanks for your understanding!

@LysandreJik 
Thank you for a quick comment. Of course we can wait! We look forward to having you review this PR! 

Hey @SO0529 ! We just came back, gonna take a look at this with @younesbelkada asap 😄 

@ArthurZucker @younesbelkada Thank you in advance for taking your time to review this PR!:smiley:

Thanks for your valuable comments! I correct them one by one:muscle:

Awesome, I will review again once you are done with that 😉 

Sorry I found mistakes, I re-push later when I fix.

@younesbelkada @ArthurZucker 
I think I could fix all your comments like below. I'm glad if you could review this PR again:smile:

> - We might need a bit more documentation about RowParrallelLinear as it is a new module that is not present in the original GPT-NeoX implementation. I also think that the name can be changed as I think that there is no model parallelism involved (we just do a F.linear). Feel free to have a look at the BLOOM implementation to see [how we got rid of the output_bias variable. ](https://github.com/huggingface/transformers/blob/9faa9f9dacf8c818ab2513da3ef92ce66f39515d/src/transformers/models/bloom/modeling_bloom.py#L383)there. But I am not sure yet how to adapt this with your model since the argument skip_bias_add might be important (for BLOOM this argument was set to False on all models).
- I removed both `RowParrallelLinear` and `ColumnParrallelLinear` from this model.

> - A small nit on the Attention block that can be easily fixed with the small change that I proposed, but I feel that the error initially came from using F.linear on the RowParrallelLinear and ColumnParrallelLinear modules (accelerate does not support torch.functionnal functions so that is why it might be related).
- I used `nn.Linear` instead of `F.linear`.

> - If some modules are entirely copied from the original GPT-NeoX implementation, you can just add a # Copied from... statement on the top of the class definition for better tracking 💪
- I put a comment at the top of class definition.

> - Also not sure if we have to keep the bias_dropout_fn as I experienced a very small throughput enhancement that we can neglect for better code readability. Feel free to have a look [https://github.com/huggingface/transformers/blob/9faa9f9dacf8c818ab2513da3ef92ce66f39515d/src/transformers/models/bloom/modeling_bloom.py#L130](https://github.com/huggingface/transformers/pull/here) as we went through the same dilemma when integrating BLOOM from Megatron-DeepSpeed
- I keep `bias_dropout_add` function to use the bias param, but it has been changed from an unnecessarily difficult-to-read writing style to a simple description.

In addition to above, I added a simple `test_generate` function.

Again, thank you for taking your time to review this PR! 

By the way, `build_and_test ` and `ci/circleci: run_tests_torch` looks to be in error with the model `tests/models/pegasus/test_modeling_pegasus.py`, how do I address this error?:worried:

Hey! Awesome work, we will review again today! 
The Pegasus test should not really be related to you, I will have a look 😊

@younesbelkada 
Thank you for quick review!
I modified below 2 items following your comments.
- I put a documentation for `bias_dropout_add`.
- I changed coding style regarding `nn.ModuleList`.

@sgugger 
Thank you for taking your time to review PR!
I think I could modify all your comments. I would like you to look the correction please.

@sgugger 
Thank you for taking your time again and again! I've changed your last comment:fire:
We are looking forward to being merge!