Repository: huggingface/transformers
Pull request: Update model card for gpt neox japanese

# What does this PR do?
Update the model card gpt neox japanese for  https://github.com/huggingface/transformers/issues/36979

## Before submitting

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@stevhliu

Changed files:
filename: docs/source/en/model_doc/gpt_neox_japanese.md
patch: @@ -13,53 +13,109 @@ specific language governing permissions and limitations under the License.
 rendered properly in your Markdown viewer.
 
 -->
+<div style="float: right;">
+    <div class="flex flex-wrap space-x-1">
+           <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColorF=white">
+
+    </div>
+</div>
+
 *This model was released on 2022-07-27 and added to Hugging Face Transformers on 2022-09-14.*
 
 # GPT-NeoX-Japanese
 
-<div class="flex flex-wrap space-x-1">
-<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
-<img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
-</div>
+GPT-NeoX-Japanese, a Japanese language model based on [GPT-NeoX](./gpt_neox).
+Japanese uses three types of characters (hiragana, katakana, kanji) and has a huge vocabulary. This model uses [BPEEncoder V2](https://github.com/tanreinama/Japanese-BPEEncoder_V2), a sub-word tokenizer to handle the different characters.
 
-## Overview
 
-We introduce GPT-NeoX-Japanese, which is an autoregressive language model for Japanese, trained on top of [https://github.com/EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).
-Japanese is a unique language with its large vocabulary and a combination of hiragana, katakana, and kanji writing scripts.
-To address this distinct structure of the Japanese language, we use a [special sub-word tokenizer](https://github.com/tanreinama/Japanese-BPEEncoder_V2). We are very grateful to *tanreinama* for open-sourcing this incredibly helpful tokenizer.
-Following the recommendations from Google's research on [PaLM](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), we have removed bias parameters from transformer blocks, achieving better model performance. Please refer [this article](https://medium.com/ml-abeja/training-a-better-gpt-2-93b157662ae4) in detail.
 
-Development of the model was led by [Shinya Otani](https://github.com/SO0529), [Takayoshi Makabe](https://github.com/spider-man-tm), [Anuj Arora](https://github.com/Anuj040), and [Kyo Hattori](https://github.com/go5paopao) from [ABEJA, Inc.](https://www.abejainc.com/). For more information on this model-building activity, please refer [here (ja)](https://tech-blog.abeja.asia/entry/abeja-gpt-project-202207).
+The model also removes some bias parameters for better performance.
 
-### Usage example
+You can find all the original GPT-NeoX-Japanese checkpoints under the [ABEJA](https://huggingface.co/abeja/models?search=gpt-neo-x) organization.
 
-The `generate()` method can be used to generate text using GPT NeoX Japanese model.
+> [!TIP]
+> This model was contributed by [Shinya Otani](https://github.com/SO0529), [Takayoshi Makabe](https://github.com/spider-man-tm), [Anuj Arora](https://github.com/Anuj040), and [Kyo Hattori](https://github.com/go5paopao) from [ABEJA, Inc.](https://www.abejainc.com/).
+>
+> Click on the GPT-NeoX-Japanese models in the right sidebar for more examples of how to apply GPT-NeoX-Japanese to different language tasks.
 
-```python
->>> from transformers import GPTNeoXJapaneseForCausalLM, GPTNeoXJapaneseTokenizer
+The example below demonstrates how to generate text with [`Pipeline`] or the [`AutoModel`], and from the command line.
 
->>> model = GPTNeoXJapaneseForCausalLM.from_pretrained("abeja/gpt-neox-japanese-2.7b")
->>> tokenizer = GPTNeoXJapaneseTokenizer.from_pretrained("abeja/gpt-neox-japanese-2.7b")
+<hfoptions id="usage">
+<hfoption id="Pipeline">
 
->>> prompt = "人とAIが協調するためには、"
+```py
+import torch
+from transformers import pipeline
+pipeline = pipeline(task="text-generation", 
+                    model="abeja/gpt-neox-japanese-2.7b", torch_dtype=torch.float16, device=0)
+pipeline("人とAIが協調するためには、")
+```
 
->>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+</hfoption>
+<hfoption id="AutoModel">
 
->>> gen_tokens = model.generate(
-...     input_ids,
-...     do_sample=True,
-...     temperature=0.9,
-...     max_length=100,
-... )
->>> gen_text = tokenizer.batch_decode(gen_tokens, skip_special_tokens=True)[0]
+```py
+import torch  
+from transformers import AutoModelForCausalLM, AutoTokenizer  
 
->>> print(gen_text)
-人とAIが協調するためには、AIと人が共存し、AIを正しく理解する必要があります。
+device = "cuda" if torch.cuda.is_available() else "cpu"  
+model = AutoModelForCausalLM.from_pretrained("abeja/gpt-neox-japanese-2.7b", torch_dtype=torch.float16, device_map="auto").to(device)  
+tokenizer = AutoTokenizer.from_pretrained("abeja/gpt-neox-japanese-2.7b")  
+input_ids = tokenizer("人とAIが協調するためには、", return_tensors="pt").input_ids.to(device)  
+outputs = model.generate(input_ids)  
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))  
 ```
 
-## Resources
+</hfoption>
+<hfoption id="transformers CLI">
+```bash
+echo -e "人とAIが協調するためには、" | transformers run --task text-generation --model abeja/gpt-neox-japanese-2.7b --device 0
+```
+
+</hfoption>
+</hfoptions>
+
+Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the [Quantization](../quantization/overview) overview for more available quantization backends.
+
+The example below uses [bitsandbytes](../quantization/bitsandbytes) to only quantize the weights to 4-bits.
+
+```py
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+
+quantization_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype="float16"
+)
+model = AutoModelForCausalLM.from_pretrained(
+    "abeja/gpt-neox-japanese-2.7b",
+    quantization_config=quantization_config,
+    device_map="auto"
+)
+
+tokenizer = AutoTokenizer.from_pretrained("abeja/gpt-neox-japanese-2.7b")
+input_ids = tokenizer.encode("人とAIが協調するためには、", return_tensors="pt").to("cuda")
+output = model.generate(input_ids)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+Use the [AttentionMaskVisualizer](https://github.com/huggingface/transformers/blob/beb9b5b02246b9b7ee81ddf938f93f44cfeaad19/src/transformers/utils/attention_visualizer.py#L139) to better understand what tokens the model can and cannot attend to.
+
+```py
+from transformers.utils.attention_visualizer import AttentionMaskVisualizer
+
+visualizer = AttentionMaskVisualizer("abeja/gpt-neox-japanese-2.7b")
+visualizer("<img>What is shown in this image?")
+```
+
+<div class="flex justify-center">
+    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/gpt_neox_japanese-attn-mask.png"/>
+</div>
 
-- [Causal language modeling task guide](../tasks/language_modeling)
+## Resources
+Refer to the [Training a better GPT model: Learnings from PaLM](https://medium.com/ml-abeja/training-a-better-gpt-2-93b157662ae4) blog post for more details about how ABEJA trained GPT-NeoX-Japanese.
 
 ## GPTNeoXJapaneseConfig