[core] Large/full refactor of `from_pretrained`
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

## Preview

Large refactor of `from_pretrained`. The most important updates are the following:

- Much more readable and maintainable in the future (hopefully, at least IMO). Simplified a looooot of weird/dead/useless code that accumulated over the years
- faster model downloads (concurrent file download)
- new keyword argument `key_mapping`, allowing direct mapping of the weight names if loading a model from the hub which is compatible to given arch, but was not converted accordingly. For example, the following snippet works nicely:
```py
from transformers import LlamaForCausalLM, AutoTokenizer, LlamaConfig

key_mapping = {
    r"^transformer.wte": r"model.embed_tokens",
    r"^transformer.rotary": r"model.rotary_emb", 
    r"^transformer.ln_f": r"model.norm", 

    r"^transformer.h.(\d+).ln_1": r"model.layers.\1.input_layernorm",
    r"^transformer.h.(\d+).ln_2": r"model.layers.\1.post_attention_layernorm",

    r"^transformer.h.(\d+).mlp.c_fc_0": r"model.layers.\1.mlp.gate_proj",
    r"^transformer.h.(\d+).mlp.c_fc_1": r"model.layers.\1.mlp.up_proj",
    r"^transformer.h.(\d+).mlp.c_proj": r"model.layers.\1.mlp.down_proj",

    r"^transformer.h.(\d+).attn.attention.k_proj": r"model.layers.\1.self_attn.k_proj",
    r"^transformer.h.(\d+).attn.attention.v_proj": r"model.layers.\1.self_attn.v_proj",
    r"^transformer.h.(\d+).attn.attention.q_proj": r"model.layers.\1.self_attn.q_proj",
    r"^transformer.h.(\d+).attn.attention.out_proj": r"model.layers.\1.self_attn.o_proj",
}

model_id = "LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct"

config = LlamaConfig(num_hidden_layers=32,
                     num_key_value_heads=8,
                     intermediate_size=14336,
                     hidden_size=4096,
                     bos_token_id=1,
                     eos_token_id= 361,
                     pad_token_id=0,
                     rope_theta=500000.0,
                     vocab_size=102400
                     )
model = LlamaForCausalLM.from_pretrained(model_id, config=config, key_mapping=key_mapping, torch_dtype="float16", device_map=0)
```
and will allow to simplify model conversions. It will also help when teams want to add their models to the library, but tey are 1:1 compatible with existing archs.

## Some pointers

Files modified apart from `modeling_utils.py` and `hub.py` are due to the removal of legacy functions `get_file_from_repo` and `_load_pretrained_model_low_mem` during the cleanup.

Note that I was **as careful as can be** during the refactor since this is extremely critical code. All tests are passing on the CI, and I tested the most common scenarios, as well as the new features added.  Everything seem to work.

However, I am deeply sorry for the person reviewing @ArthurZucker 😆🥲🤗

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36033). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @Cyrilvallez 

The workflow run: https://github.com/huggingface/transformers/actions/runs/13249941627
The slack report: https://huggingface.slack.com/archives/C06LR9PQA00/p1739225850225799
The new failures: https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/blob/13935d4e1231eb06ab3ce9647a6d3ec92a85b16e/2025-02-10/ci_results_run_models_gpu/new_model_failures.json




All our test suite passes. See [here](https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/blob/495f87e77baa0078f7290adc88fb002b7df4ac52/2025-02-12/ci_results_run_models_gpu/new_model_failures.json) for new failures compared to last full CI run, but I personally checked that they are all unrelated and are due to the diff between the tip of this branch and `main`.
Now rebasing

Hi, @Cyrilvallez! This refactor was long needed. Thank you for streamlining `modeling_utils.py`!

is it possible to avoid creating `state_dict` in RAM altogether and loading model params straight to GPU(s)? sometimes a server has too little RAM or it may be taken by `tmpfs`.
Perhaps single param/buffers items could be sent to GPU right after loading to disk, without waiting for  whole `state_dict` to load?

Also please ensure that quantized models loading/saving is tested in this PR.

> Also please ensure that quantized models loading/saving is tested in this PR.

Hmm, indeed, I need to trigger quantization CI too!

Oh I thought we did already! No worries, I'll ping you on slack @ydshieh, I first have to do a few changes.

@poedator thanks! Concerning the loading, note that state dicts on the hub are usually quite small (never bigger than 10 GB), so loading everything in ram is never an issue. It would otherwise probably be very slow!

> Oh I thought we did already!

My bad. I am (almost always) focus on the CI jobs regarding `tests/models/xxx`, but this time it is indeed important to check the CI jobs regarding `quantization/xxx`

> note that state dicts on the hub are usually quite small (never bigger than 10 GB), so loading everything in ram is never an issue. It would otherwise probably be very slow!

But what about cases like loading models like Llama 70B, which is 140GB in 16bit, or loading 34 B model into 8 GPUs for data parallel, which takes 8 x 34 x 2 = 544GB of RAM? 


State dicts are loaded one after the other, so no need to worry! See e.g. [here](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/tree/main) for Llama 70B: each state dict is at most 5 GB

Merging!