Refactor model summary
Author: stevhliu
Pull request: yes

This PR refactors the [model summary](https://huggingface.co/docs/transformers/model_summary):

- updated with speech/audio, computer vision, and multimodal models (picked based on the ones with the most doc views, this can be refined to show or hide other models)
- embeds a timeline of when models are released to provide a visual reference
- provides structure and narrative - instead of a list - to discuss the high-level differences between models (users can compare the models, see trends and progression in the larger modelscape)
- removed the [attention section](https://huggingface.co/docs/transformers/model_summary#more-technical-aspects), which will get its own page (and possibly be expanded with more attention types) in the conceptual guide section

Would love to hear what you think about the direction of this doc please! @sgugger @MKhalusova @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I don't think a model summary with sections per modality can work. I'm okay with removing the specific of each model currently in the model summary (make sure they are present in the corresponding model pages however as we don't want to lose anything) but I think we need to have a better structure with h2 sections for modalities than h3 sections for different kinds of models (in NLP encoder/decoder/encoder-decoder, in CV transformer/convnet etc.).

> but I think we need to have a better structure with h2 sections for modalities than h3 sections for different kinds of models

Good point, this’ll work better and allow me to include convnets more naturally!

Great questions @MKhalusova, let me try and clarify (and also refine the purpose of the doc while doing so)! 🙂

> My main issue is that it is not entirely clear to me what audience these documents target and what they aim to achieve.

The audience is a beginner or someone who is coming from a different modality (say like, from NLP to CV), and the goal is to provide a high-level conceptual overview of the different model types available in each modality.

> If I know what model I am interested in, its model doc is much more useful, as it has all of the information.

For sure, the model docs fulfill the role of providing all the nitty-gritty information. But sometimes, this can be too much detail, and you can't really make connections between models or understand why you should use one over the other because you're lacking context. The model summary doc tries to go up a level and give users an introductory overview instead of all the technical details. If they’re interested in learning more, they can follow the links to the specific model doc page.

> If I want to learn about the difference between encoders and decoders, the information is in the course.

The course only has very general information about encoders and decoders. For example, it doesn’t tell you how BERT and DeBERTa are different.

> If I want to compare two different models for the same task, I have to jump up and down in the doc and may learn some differences in how they work internally, but what if I’m interested in other aspects such as benchmarks, size of the model, how recent it is, etc.?

Yeah the structure I have now is not the best! 😅  But I think @sgugger's suggestion will improve this quite a bit, where it’ll be more readable, and related sections will be more localized, so you don’t have to jump around as much. The goal though is not to give users all the technical details about a model (size, performance, etc.).

> So my question is, what are we aiming to achieve with this doc?

It can be difficult to approach Transformers when there are so many X-former variants. This doc hopes to provide users with a beginner-friendly guide to them so they can make connections and be like oh wait, this CV model is just like an NLP model, and it's just the input that's different. I think we also want to give more context about the models in terms of design decisions and constraints (e.g., Swin does _x_, unlike ViT because _y_). In a nutshell, I suppose it's to give users the bigger picture of the Transformer model landscape and give them a mental framework to categorize and think about Transformer models.

> but what if I’m interested in other aspects such as benchmarks
> Are we creating a place where one can compare models on several aspects?

I think we can boost the impact of this doc even more by addressing those issues you raise above. An embedded Space at the top of the doc that lets users discover models based on certain parameters (longer sequences, tasks, memory-efficiency, multilinguality, etc.) would be very useful and guide users toward selecting a model for their use-case. I can look into this as a next step! 🙂

Updated the structure to be:

```
## Computer vision

### Encoder
### ConvNet

## NLP

### Encoder
### Decoder
...
```

If this looks good to everyone, I'll go ahead and fill out the rest of the sections!

Ok I think this is ready for review now, time to call in the experts! The goal of the doc is to provide a high-level overview of the model types in each modality, so users have more context and can start making connections.

@sayakpaul, would you mind reviewing the computer vision and maybe the multimodal sections? @sanchit-gandhi, if you could take a look at the audio section please (I promise it's way shorter this time 😅 )? Thank you both so much! 👏

I kept most of the model summary infos that wasn't redundant with what was already on the model doc pages (for example, the original GPT); these have been added under the **Tips** section. I've also split out the attention mechanisms onto their own page, which I'll expand on later in a separate PR with additional attention types.

Hi @stevhliu !

Last week I was reading a medium page that linked to an older version of the docs before this PR.
And today it took me almost an hour to find the following sentence again (removed by this PR):

> For a gentle introduction check the [annotated transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html).

_links to: http://nlp.seas.harvard.edu/2018/04/03/attention.html_

Do you remember if you moved this **annotated transformer** link to another page in the Hugging Face docs?

<img width="1694" alt="XG0Cw97mWN" src="https://github.com/huggingface/transformers/assets/65138515/322faf1f-ad5a-4a21-afdd-9c7b664f1880">

<img width="1922" alt="Screenshot 2024-03-10 192805" src="https://github.com/huggingface/transformers/assets/65138515/ba668b32-6e0c-4eab-b3d5-f3ebe07658db">

I'm a computer scientist, and out of 100s of articles I've skimmed the past week, this was the most promising link to begin diving into in ernest. :)

I understand if this page wasn't the best fit for it, but on the offchance it was entirely removed from Hugging Face, I think it's a good resource to add back somewhere.

Hey @gamepad-coder, this page isn't from Hugging Face but from Harvard. Clicking the link works fine for me, it doesn't work for you?

https://nlp.seas.harvard.edu/2018/04/03/attention.html

Hi @LysandreJik ~

Apologies, my previous screenshot shows the code before and after, 
but it only shows the **before picture** of the docs.
- (I should add an **after picture** of the docs to better illustrate what my ask is).

Here's a screenshot of
- the docs before this PR on the left.
- the docs after this PR on the right.

**Before PR**
- https://huggingface.co/docs/transformers/v4.26.1/en/model_summary

**After PR**
- https://huggingface.co/docs/transformers/v4.27.2/en/model_summary
- (I think this is the right tag at least, it's the first version without the annotated transformer link, and the rest of the code changes in the file seem to match).

<img width="1920" alt="image" src="https://github.com/huggingface/transformers/assets/65138515/e2c1c558-a3cc-430d-bfeb-aafdb48e267f">

----

Main ask:

The code containing the following was removed from the docs by this PR:<br>

```
For a gentle introduction check the [annotated transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html).
```

and I just want to make sure this link exists somewhere else on Hugging Face.

Although I do have the link now, I want to make sure others can find this resource too.

---

Reason: from what I can tell, Hugging Face is becoming what Open AI was meant to be from the beginning -- a resource to make AI accessible to humanity and non-gated.

While the original **Attention Is All You Need** paper is still linked to after this PR (see the link in the after screenshot: [original Transformer](https://arxiv.org/abs/1706.03762)) (and on current `main` 2024-3-11),
the Annotated Transformer link is a walkthrough of that **Attention Is All You Need** paper annotated by Harvard (see the link in the before screenshot: [annotated transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html)).

To an absolute beginner, I think the Harvard walkthrough is a much better resource -- and to anyone wanting to practice AI with Python, I think this is a much better primer than the original academic paper alone.

Hi, the Annotated Transformer blog post is indeed an excellent and iconic resource! Would you like to open a PR to include it in the docs?

Hi @stevhliu !
Sorry for the delay, I've had a bad work-life balance this quarter :)

Yes I can do that -- do you think this page would still be a good fit for the link?
Or do you think adding it to another section of the docs would be better?

✅ Created a PR here ^

Let me know if you'd like me to update anything.