4.34 not compatible with accelerate
Author: SingL3
Pull request: no

### System Info

old env:
```
- `transformers` version: 4.33.0
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

new env(all latest version now):
```
- `transformers` version: 4.34.1
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am running a training mission of 34b model using `accelerate` on 2 nodes of A100 80G.
My accelerate config:
```yaml
compute_environment: LOCAL_MACHINE                                                                                             
debug: false                                                                                                                   
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: true
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: 'bf16'
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

launch command:
```
LAUNCHER="accelerate launch \
    --config_file configs/fsdp_config.yaml \
    --main_process_ip $MASTER_ADDR \
    --main_process_port $MASTER_PORT \
    --machine_rank $RANK \
    --num_processes $NUM_PROCESSES \
    --num_machines $NNODES \
```

I can run successfully with the old env listed above, but when I run with the new env(`accelerate` 0.23.0 and 0.24.1), I got this error:
```
[32mpt-mc7v1e5q-worker-0 logs:[0m Traceback (most recent call last):
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/home/xxxxxxxxx/rlhf/LLMFinetune/llmft/commands/run.py", line 17, in <module>
[32mpt-mc7v1e5q-worker-0 logs:[0m     APIS.build(**config.api)
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/home/xxxxxxxxx/rlhf/LLMFinetune/llmft/utils/registry.py", line 64, in build
[32mpt-mc7v1e5q-worker-0 logs:[0m     return obj(**kwargs)
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/home/xxxxxxxxx/rlhf/LLMFinetune/llmft/apis/train_sft.py", line 36, in train_sft
[32mpt-mc7v1e5q-worker-0 logs:[0m     tokenizer, model = build_model(tokenizer, model, model_wrapper)
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/home/xxxxxxxxx/rlhf/LLMFinetune/llmft/models/utils.py", line 69, in build_model
[32mpt-mc7v1e5q-worker-0 logs:[0m     tokenizer, model = MODELS.build(**model, tokenizer=tokenizer)
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/home/xxxxxxxxx/rlhf/LLMFinetune/llmft/utils/registry.py", line 64, in build
[32mpt-mc7v1e5q-worker-0 logs:[0m     return obj(**kwargs)
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/home/xxxxxxxxx/rlhf/LLMFinetune/llmft/models/auto_causal_lm.py", line 16, in auto_causal_lm
[32mpt-mc7v1e5q-worker-0 logs:[0m     model = AutoModelForCausalLM.from_pretrained(model_path, **kwargs)
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/data/conda/envs/megatron_transformers434/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 560, in from_pretrained
[32mpt-mc7v1e5q-worker-0 logs:[0m     return model_class.from_pretrained(
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/data/conda/envs/megatron_transformers434/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3307, in from_pretrained
[32mpt-mc7v1e5q-worker-0 logs:[0m     ) = cls._load_pretrained_model(
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/data/conda/envs/megatron_transformers434/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3695, in _load_pretrained_model
[32mpt-mc7v1e5q-worker-0 logs:[0m     new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/data/conda/envs/megatron_transformers434/lib/python3.9/site-packages/transformers/modeling_utils.py", line 741, in _load_state_dict_into_meta_model
[32mpt-mc7v1e5q-worker-0 logs:[0m     set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
[32mpt-mc7v1e5q-worker-0 logs:[0m   File "/mnt/data/conda/envs/megatron_transformers434/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 317, in set_module_tensor_to_device
[32mpt-mc7v1e5q-worker-0 logs:[0m     new_value = value.to(device)
[32mpt-mc7v1e5q-worker-0 logs:[0m NotImplementedError: Cannot copy out of meta tensor; no data!
```

### Expected behavior

Run successfully with either old or new env.

Comments:
Setting `fsdp_cpu_ram_efficient_loading: false` in `fsdp_config` didnt work.

@SingL3 I confronted similar problem, see [here](https://github.com/huggingface/transformers/issues/27166). is version 4.33.0 ok? how many a100 gpus do you use to train 13b llama? Do you know what's minimal gpus to run 70b?

@fancyerii Yep, 4.33 works. 
It depends on the `max_seq_len` to train 70b. My experience is, with 2048 length, 2 nodes (16 A100 80G) is needed.
I check your issue and the blog you followed, I have done the same and I can run 70b successfully on 2 nodes. Didnt check if one nodes works.

thanks. by the way, could you tell me the speed of 13b model when using a 8 gpus node? how much time it take to do a single batch(if per_device_train_batch_size is 1)?

Didnt run under that config. I have trained using 4 gpus with `per_device_train_batch_size=2`(I think 4 also work) and deepspeed Zero 3. Took 5.5 hours for 5 epochs on dolly 15k.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.