HuggingFace `from_pretrained()` called multiple times when launching with `torchrun` 
Author: chenmoneygithub
Pull request: no
Labels: name: PyTorch FSDP

### System Info

- `transformers` version: 4.38.2
- Platform: Linux-6.5.0-1014-aws-x86_64-with-glibc2.35
- Python version: 3.10.0
- Huggingface_hub version: 0.21.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2

### Who can help?

@yon

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The problem is `from_pretrained()` is called in every process spawned by `torchrun`, which caused the system RAM to OOM. A small reproducible code would be like:

```python
def fsdp_main(args=None):
    model_name = "facebook/opt-350m"

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = OPTForSequenceClassification.from_pretrained(model_name)

if __name__ == '__main__':
    fsdp_main()
```
The launch script is `torchrun --nnodes 1 --nproc_per_node 4 {file_name}`.

I was expecting this to be a common issue, but couldn't find any related docs online. I have tried magical args like `low_cpu_mem_usage=True`, but no luck. Could anyone point to the right doc/example? thanks! 

### Expected behavior

Checkpoints loading should only happen once automatically. 

Comments:
Hi @chenmoneygithub, thanks for raising this issue!

cc @pacman100 

@amyeroberts  Thanks for the reply! To provide a bit more information, I was able to bypass the problem via the following code:  [link](https://github.com/meta-llama/llama-recipes/blob/main/src/llama_recipes/finetuning.py#L116)

```
        if rank == 0:
            model = LlamaForCausalLM.from_pretrained(
                train_config.model_name,
                load_in_8bit=True if train_config.quantization else None,
                device_map="auto" if train_config.quantization else None,
                use_cache=use_cache,
                attn_implementation="sdpa" if train_config.use_fast_kernels else None,
            )
        else:
            llama_config = LlamaConfig.from_pretrained(train_config.model_name)
            llama_config.use_cache = use_cache
            with torch.device("meta"):
                model = LlamaForCausalLM(llama_config)
```

Then when initiailizing FSDP, it is like:

```
        model = FSDP(
            model,
            auto_wrap_policy= my_auto_wrapping_policy if train_config.use_peft else wrapping_policy,
            cpu_offload=CPUOffload(offload_params=True) if fsdp_config.fsdp_cpu_offload else None,
            mixed_precision=mixed_precision_policy if not fsdp_config.pure_bf16 else None,
            sharding_strategy=fsdp_config.sharding_strategy,
            device_mesh=hsdp_device_mesh,
            device_id=device_id,
            limit_all_gathers=True,
            sync_module_states=train_config.low_cpu_fsdp,
            param_init_fn=lambda module: module.to_empty(device=torch.device("cuda"), recurse=False)
            if train_config.low_cpu_fsdp and rank != 0 else None,
        )
```
The important part is `sync_module_states` and `param_init_fn`. 

Would that be possible to integrate this into `from_pretrained()`?  thanks! 



Gentle ping @pacman100 

cc @muellerzr @SunMarc 

Do `cpu_ram_efficient_loading=True` and this will mimic the behavior :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.