[WIP] [doc] performance/scalability revamp
Author: stas00
Pull request: yes

moved from https://github.com/huggingface/transformers/pull/15213 so that we get the doc generator working

XXX: The previous PR has comments/suggestions that need to be integrated here

-----------------

@lvwerra and I are working on a massive performance/scalability docs revamp:

So the rough plan is to make custom plans for each of the combinations `[inference|training] * [1 gpu|many gpus|cpu]` so that it's very easy for the user to follow the instructions that are specific to their needs.

So the proposed doc layout is:

* performance.mdx (main entry point)
* perf_infer.mdx
   - perf_infer_cpu.mdx
   - perf_infer_gpu_many.mdx
   - perf_infer_gpu_one.mdx
* perf_train.mdx
   - perf_train_gpu_many.mdx
   - perf_train_gpu_one.mdx
* scalability.mdx (rename from parallelism.mdx) (XXX: to do)

See the PR's changes for a rough layout of the content.

One big question is this: At the moment everything is pytorch-centric, as we don't have any info on tf/flax. Down the road we will either inject tf/flax-specific instructions into the current docs, or perhaps it'd be better to have dedicated docs for pt/tf/flax. It'd help a lot to decide ahead of time to avoid document renaming and potentially breaking links. If we plan to have these PT-specific perhaps let's embed `_pt` in the filenames?

@lvwerra 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15723). All of your documentation changes will be reflected on that endpoint.

Hi @stas00, first stab at the single GPU section. The text is still WIP but you could have look at the sections and rough content to see if you agree.

Looking at the docs as a whole again here a few thoughts:
- Instead of dividing the whole section into speed and memory related techniques I thought an alternative could be to have a summary table at the beginning or end where the memory and speed impact of all methods is described. Could also help a user navigate the doc better. 
- In the previous PR we discussed where to discuss common concepts (first time they occurred vs. in the performance docs). I am currently leaning towards the former. The main thing that's currently in the `performance.mdx` and I am not sure, yet, where I would put it is the hardware discussion which is quite different to everything else we are doing which makes me think that maybe we can create a dedicated document `hardware.mdx` where we could also expand a bit to other accelerators (TPU/IPUs for example). 
- If we take that road the `performance.mdx` doc is again quite empty and I think we could move the content from `perf_infer.mdx` and `perf_train.mdx` to this document and then have a shared entry point and remove `perf_infer.mdx` and `perf_train.mdx`.

If you are happy with the structure I could start outlining the `performance.mdx` and polishing the single-GPU section while you could for example work on the multi-GPU training content. 

What do you think?

Thank you for taking a stab at redesign, @lvwerra 

It's very clear that we have fundamentally conflicting visions of how the performance documentation should be laid out. So it'd be wasteful to spend energy to try to come up with something that resonates with both of us.

I don't think this project can have 2 masterminds, and since I'm currently busy with BigScience training and we don't want this to fall between the cracks, I propose that if this works for you, please take over and proceed in an unimpeded way that sounds good to you, @lvwerra.

Once you have completed the restructuring and added what you know and you want me to fill in some gaps that I hopefully know to fill in please let me know and I will do so. or alternatively do tell me what sections are missing and I will work on those in free format and you can then adapt them to the structure you end up choosing.



Hi @LysandreJik and @sgugger, 

I revamped the documentation a bit and reduced the scope from all the documents @stas00 lined out to just the `performance.mdx` and the training on single/multi-GPU so we can converge on those before adding the others, which also prevents a monster PR. For some reason the preview of the docs still does not work - if you have any idea, let me know. 

To guide you a bit in the review:
- `performance.mdx` should be the entry point and give an overview
- `perf_train_gpu_single.mdx` shows all the tricks to efficiently train large models on a single GPU (gradient accumulation/checkpointing, optimizers, DeepSpeed Zero, data loaders etc.)
- `perf_train_gpu_single.mdx` shows all the tricks to efficiently train large models on a single GPU (mostly parallelism strategies: data, tensor, pipeline parallelism)
- `perf_hardware.mdx` is where I put some tips and tricks about custom hardware setups

In subsequent PRs we can:
- add more TensorFlow examples
- add TPU training guide
- add inference side
- add guides for train/inference on specialized hardware (Optimum)

This is the plan for the full documentation:

<img width="1177" alt="perf_overview" src="https://user-images.githubusercontent.com/8264887/163152248-1565d6bd-2e2f-4626-897b-5a1d67f6116b.png">

Let me know what you think about style and content of the documentation. There are still a few rough edges but I think it is good for a first review. 

The preview can't work if you don't rebase on master to have the new doc structure (everything nested in an "en" subfolder)

_The documentation is not available anymore as the PR was closed or merged._

Thank you @sgugger it worked!

@lvwerra, I rebased and solved the conflict introduced by https://github.com/huggingface/transformers/pull/16860 - fixing it at its new file the content was moved to.

@stas00 thank you!

@sgugger @LysandreJik would you like to take a look at the PR? I integrated the previous suggestion.

@sgugger are you happy to merge this?

Yes, LGTM :-)