add japanese documentation
Author: rajveer43
Pull request: yes

part of #18413

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #18413 
Fixes #26355

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerz and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @younesbelkada @stevhliu 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26138). All of your documentation changes will be reflected on that endpoint.

@stevhliu I have added documentation for preprocessing.md as well kindly review it and added language code for it as well in the worflows.

also will add for most of the files 

will add for all!

@stevhliu 

in `peft.md`, these things are remaining
-   Link to PEFT docs for further details
-   Trainer  
-   8-bit / 4-bit examples ?


@stevhliu There are still few docs remaining. 

@younesbelkada @stevhliu  review the PR once and merge. there are still few documents left. which can me translated later on! 

> Great job! Can you add the missing files [here](https://github.com/huggingface/transformers/actions/runs/6348047242/job/17261143258?pr=26138#step:11:839) to the `toctree`?

Sure I will. I just saw the Build PR Doc has failed. will add it today!

@stevhliu Kindly approve the workflow!

> Hey, it looks like you're still missing some entries in the `toctree` and or the file name is incorrect. For example, your filename is `glosaary` instead of `glossary`. Finally, can you also follow the same order as it is [here](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml)?

sure!

@stevhliu Can you give me some insight about `toctree.yml`, I mean, How I should name and give title names, a bit of explanation would help me to add contents in toctree.yml for Japanese documentation

I studied the toctree.yml file for English document and I found it contains the documentation for other folder related files. I will be translating those too within this PR!

> I will be translating those too within this PR!

Actually, would it be ok with you to translate those in a separate PR? This one is getting quite big already and it'd be easier to merge this one first. 

The `toctree` order/structure should be the same for both the English and Japanese translations. For example, the Tutorial section should be:

```yml
- sections:
  - local: pipeline_tutorial
    title: Run inference with pipelines
  - local: autoclass_tutorial
    title: Write portable code with AutoClass
  - local: preprocessing
    title: Preprocess data
  - local: training
    title: Fine-tune a pretrained model
...
...
```

You can translate the title names to their equivalent in Japanese, but I was more referring to the local filenames should match the name in the `toctree`.

> > I will be translating those too within this PR!
> 
> Actually, would it be ok with you to translate those in a separate PR? This one is getting quite big already and it'd be easier to merge this one first.
> 
> The `toctree` order/structure should be the same for both the English and Japanese translations. For example, the Tutorial section should be:
> 
> ```yaml
> - sections:
>   - local: pipeline_tutorial
>     title: Run inference with pipelines
>   - local: autoclass_tutorial
>     title: Write portable code with AutoClass
>   - local: preprocessing
>     title: Preprocess data
>   - local: training
>     title: Fine-tune a pretrained model
> ...
> ...
> ```
> 
> You can translate the title names to their equivalent in Japanese, but I was more referring to the local filenames should match the name in the `toctree`.
Yup. I will be taking care of that!

Can you add these files to the `toctree` as well:

```
RuntimeError: The following files are not present in the table of contents:
- data_collator
- custom_model
- transformer_agents
```

@stevhliu Data collator.md is a part of main_classes so will add that in next PR.

I will make sure I do not take this many iteration of reviews in the next PR. And Making Module wise PR

Locally It is not passing I got the following error while building doc locally

```bash
doc-builder build transformers docs/source/ja/ --build_dir ~/tmp/test-build
```
I am running above command and the logs are. do I need to add that files to `_toctree.yml`, Its contradicts to `PR_doc_build` workflow

```bash
Building docs for transformers docs/source/ja/ /home/codespace/tmp/test-build/transformers/main/en
Building the MDX files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:00<00:00, 210.89it/s]
Traceback (most recent call last):
  File "/home/codespace/.python/current/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/doc_builder/commands/build.py", line 102, in build_command
    build_doc(
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/doc_builder/build_doc.py", line 369, in build_doc
    sphinx_refs = check_toc_integrity(doc_folder, output_dir)
  File "/usr/local/python/3.10.8/lib/python3.10/site-packages/doc_builder/build_doc.py", line 427, in check_toc_integrity
    raise RuntimeError(
RuntimeError: The following files are not present in the table of contents:
- debugging
- notebooks
- add_new_pipeline
- sagemaker
- contributing
- tasks/visual_question_answering
- tasks/audio_classification
- tasks/summarization
- tasks/document_question_answering
- tasks/image_classification
- tasks/sequence_classification
- tasks/masked_language_modeling
- tasks/video_classification
- tasks/question_answering
- tasks/semantic_segmentation
- tasks/idefics
- tasks/zero_shot_image_classification
- tasks/language_modeling
- tasks/asr
- tasks/image_captioning
- tasks/monocular_depth_estimation
- tasks/multiple_choice
- tasks/translation
- tasks/text-to-speech
- tasks/object_detection
- tasks/token_classification
- tasks/zero_shot_object_detection
- main_classes/output
- main_classes/configuration
- main_classes/deepspeed
- main_classes/keras_callbacks
- main_classes/onnx
- main_classes/feature_extractor
- main_classes/pipelines
- main_classes/image_processor
- main_classes/trainer
- main_classes/callback
- main_classes/agent
- main_classes/logging
- main_classes/quantization
- main_classes/text_generation
- main_classes/optimizer_schedules
- main_classes/model
- main_classes/tokenizer
- main_classes/data_collator
- main_classes/processors
- internal/file_utils
- internal/modeling_utils
- internal/pipelines_utils
- internal/image_processing_utils
- internal/tokenization_utils
- internal/time_series_utils
- internal/audio_utils
- internal/generation_utils
- internal/trainer_utils
- model_doc/swiftformer
- model_doc/mpnet
- model_doc/auto
- model_doc/cpm
- model_doc/persimmon
- model_doc/mobilevitv2
- model_doc/switch_transformers
- model_doc/roc_bert
- model_doc/flan-ul2
- model_doc/owlvit
- model_doc/barthez
- model_doc/vision-text-dual-encoder
- model_doc/ul2
- model_doc/phobert
- model_doc/xlm-roberta
- model_doc/realm
- model_doc/convnextv2
- model_doc/gptj
- model_doc/git
- model_doc/xlm-prophetnet
- model_doc/xclip
- model_doc/segformer
- model_doc/vilt
- model_doc/pix2struct
- model_doc/clap
- model_doc/speech-encoder-decoder
- model_doc/dpr
- model_doc/canine
- model_doc/herbert
- model_doc/tapas
- model_doc/bigbird_pegasus
- model_doc/deformable_detr
- model_doc/roberta-prelayernorm
- model_doc/marian
- model_doc/pegasus_x
- model_doc/bridgetower
- model_doc/vision-encoder-decoder
- model_doc/bark
- model_doc/encoder-decoder
- model_doc/rembert
- model_doc/oneformer
- model_doc/esm
- model_doc/bert-generation
- model_doc/llama2
- model_doc/dinov2
- model_doc/wav2vec2-conformer
- model_doc/convbert
- model_doc/layoutlmv2
- model_doc/instructblip
- model_doc/trocr
- model_doc/lxmert
- model_doc/vits
- model_doc/vivit
- model_doc/matcha
- model_doc/deta
- model_doc/dinat
- model_doc/cvt
- model_doc/mra
- model_doc/retribert
- model_doc/mobilevit
- model_doc/van
- model_doc/conditional_detr
- model_doc/gpt_neox
- model_doc/squeezebert
- model_doc/data2vec
- model_doc/ernie_m
- model_doc/sam
- model_doc/vit_mae
- model_doc/mbart
- model_doc/opt
- model_doc/informer
- model_doc/visual_bert
- model_doc/mpt
- model_doc/jukebox
- model_doc/convnext
- model_doc/xlnet
- model_doc/gpt_bigcode
- model_doc/code_llama
- model_doc/umt5
- model_doc/transfo-xl
- model_doc/bert-japanese
- model_doc/nystromformer
- model_doc/roformer
- model_doc/fnet
- model_doc/sew-d
- model_doc/musicgen
- model_doc/mobilenet_v1
- model_doc/videomae
- model_doc/rwkv
- model_doc/flan-t5
- model_doc/timesformer
- model_doc/swin
- model_doc/roberta
- model_doc/whisper
- model_doc/camembert
- model_doc/plbart
- model_doc/vit_msn
- model_doc/megatron_gpt2
- model_doc/bort
- model_doc/m2m_100
- model_doc/bloom
- model_doc/codegen
- model_doc/deplot
- model_doc/gpt_neox_japanese
- model_doc/nat
- model_doc/speech_to_text_2
- model_doc/mega
- model_doc/mobilebert
- model_doc/wav2vec2
- model_doc/blip-2
- model_doc/beit
- model_doc/t5v1.1
- model_doc/gpt-sw3
- model_doc/reformer
- model_doc/clip
- model_doc/funnel
- model_doc/resnet
- model_doc/poolformer
- model_doc/regnet
- model_doc/bertweet
- model_doc/fsmt
- model_doc/mask2former
- model_doc/flava
- model_doc/tapex
- model_doc/dialogpt
- model_doc/mvp
- model_doc/longformer
- model_doc/xmod
- model_doc/ernie
- model_doc/idefics
- model_doc/focalnet
- model_doc/rag
- model_doc/pop2piano
- model_doc/layoutlmv3
- model_doc/chinese_clip
- model_doc/xlsr_wav2vec2
- model_doc/trajectory_transformer
- model_doc/layoutxlm
- model_doc/bros
- model_doc/xlm-v
- model_doc/speecht5
- model_doc/unispeech-sat
- model_doc/prophetnet
- model_doc/mgp-str
- model_doc/bart
- model_doc/nezha
- model_doc/table-transformer
- model_doc/upernet
- model_doc/bit
- model_doc/mctct
- model_doc/hubert
- model_doc/graphormer
- model_doc/vit_hybrid
- model_doc/mluke
- model_doc/t5
- model_doc/swin2sr
- model_doc/ctrl
- model_doc/perceiver
- model_doc/led
- model_doc/vitdet
- model_doc/llama
- model_doc/mobilenet_v2
- model_doc/unispeech
- model_doc/dpt
- model_doc/luke
- model_doc/splinter
- model_doc/clipseg
- model_doc/efficientnet
- model_doc/yoso
- model_doc/decision_transformer
- model_doc/donut
- model_doc/nllb
- model_doc/lilt
- model_doc/wav2vec2_phoneme
- model_doc/falcon
- model_doc/blenderbot
- model_doc/bert
- model_doc/gpt_neo
- model_doc/efficientformer
- model_doc/groupvit
- model_doc/imagegpt
- model_doc/nougat
- model_doc/openai-gpt
- model_doc/electra
- model_doc/longt5
- model_doc/megatron-bert
- model_doc/deit
- model_doc/encodec
- model_doc/biogpt
- model_doc/time_series_transformer
- model_doc/mt5
- model_doc/xglm
- model_doc/flaubert
- model_doc/distilbert
- model_doc/pvt
- model_doc/audio-spectrogram-transformer
- model_doc/qdqbert
- model_doc/deberta
- model_doc/wavlm
- model_doc/xlm
- model_doc/glpn
- model_doc/swinv2
- model_doc/gptsan-japanese
- model_doc/blenderbot-small
- model_doc/markuplm
- model_doc/tvlt
- model_doc/speech_to_text
- model_doc/yolos
- model_doc/align
- model_doc/gpt2
- model_doc/xlm-roberta-xl
- model_doc/levit
- model_doc/pegasus
- model_doc/bartpho
- model_doc/autoformer
- model_doc/open-llama
- model_doc/layoutlm
- model_doc/albert
- model_doc/big_bird
- model_doc/detr
- model_doc/ibert
- model_doc/vit
- model_doc/vitmatte
- model_doc/altclip
- model_doc/dit
- model_doc/nllb-moe
- model_doc/maskformer
- model_doc/blip
- model_doc/deberta-v2
- model_doc/mistral
- model_doc/cpmant
- model_doc/sew
- model_doc/mms
- model_doc/xls_r
- model_doc/byt5
Add them to docs/source/ja/_toctree.yml.
```

> Locally It is not passing I got the following error while building doc locally

It's probably because all those files are in the directory, so you can just remove them, and it should be built locally. But it's ok since the CI tests are passing now. There are still some changes (like this [one](https://github.com/huggingface/transformers/pull/26138#discussion_r1349216856)) that need to be addressed.

things I learned:
- Work module wise
- documentation is as much as important as coding.


@stevhliu approve the workflow.

Thanks again for translating and contributing such a significant amount of the docs to Japanese! 🇯🇵🥳 Hope you had fun!

One last thing before we merge, let's revert the changes to the English `md` files. The date should reflect when the doc was written so there's no need to update them to 2023.

> Thanks again for translating and contributing such a significant amount of the docs to Japanese! 🇯🇵🥳 Hope you had fun!
> 
> One last thing before we merge, let's revert the changes to the English `md` files. The date should reflect when the doc was written so there's no need to update them to 2023.

sure

@stevhliu I reverted the changes you told!