Doc: NLP tasks between NLU and Text Generation
Author: gante
Pull request: yes

# What does this PR do?

As discussed [here](https://github.com/huggingface/transformers/pull/25240#issuecomment-1668029668), this PR splits the NLP task section in two:
- Moves generative tasks to the new text generation task section
- Renames NLP into NLU, which now holds non-generative tasks, for clarity

Related issue: #24575 

Comments:
@sgugger so like this? Or no generative section at all?

(I thought it would make sense to keep a 1:1 correspondence between tasks and the information present in the index, but no strong feelings about it :) )

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25371). All of your documentation changes will be reflected on that endpoint.

> I don't really understand that split especially when you see it written like this in the index. NLU and text generation are the same modality: text.
> 
> Speech has some generative tasks with speech to text and Computer vision will hold generative tasks at some point (like super-resolution) and we will also have multimodal tasks doing generative stuff (text to speech).
> 
> I would leave the index (which is copied from the main README by the way) as it is.

I agree that from the "modality" perspective this is a bit weird. To me the main goal here is to bring "LLM text generation" in the focus and make sure it can easily be found by everyone, simply because it's become too important to just be a subsection in NLP.

I understand the difference between NLU, NLG and NLP as follows:
![0_Nfc79nPWifBPIQT9](https://github.com/huggingface/transformers/assets/23423619/1c66a8e9-f4e2-4fe0-a51d-61d6938374db)

So the idea here was to put all "text-classification/text-understanding" tasks under NLU and use the "text-generation/NLG" section to focus heavily on LLM generation because:

- Everything is about LLMs and we should provide the users more in-detail docs. Having a whole section dedicated to this is important.
- For a bunch of NLG tasks, such as "summarization", "question-answering", and "translation" can now be solved with the same LLM and don't necessarily need a specialized model anymore. There are still a lot of use cases where single models have advantages because of cost, but we should mention/show both the single-model (Marian) and LLM use case with code. We can better explain this by having a whole section + introduction about it.
- I'd argue that **pure** image captioning and **pure** speech recognition are not really associated with "LLM text-generation" and should live in the respective "image" and "speech" sections.
- I'd label multi-modal models actually only as models that can take "understand" multiple modalities such as GPT4 or IDEFICS. <- not the best argument, but to me there is a difference between a model that understands speech and can map it to it's transcription and a model that understands both text and image and can do whatever with it


At the same time we should probably be careful here to keep all the following somewhat in sync:
- https://huggingface.co/tasks
- https://huggingface.co/docs/transformers/tasks/
- https://github.com/huggingface/transformers#online-demos

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.