Running on Multiple GPU with DeepSpeed. Error: Model was not initialized with Zero-3 despite being configured for Deepspeed Zero-3. Please re-initialize your model via Model.from_pretrained or Model.from_config after creating your TrainingArguments!
Author: mosama1994
Pull request: no
Labels: name: bug

name: solved

### System Info

## transformers-cli env Output
![WhatsApp Image 2024-08-20 at 9 07 37 AM](https://github.com/user-attachments/assets/9f27b848-13c7-47cf-91bf-ca3f44f2a5d6)
## Accelerate Config
![WhatsApp Image 2024-08-20 at 9 10 22 AM](https://github.com/user-attachments/assets/d843257d-4a90-4874-826b-8fabf5d1c459)



### Who can help?

@muellerzr I get an error when running script using accelerate on 4 GPU.

## The error:
![Error](https://github.com/user-attachments/assets/393c7dc3-41bc-4879-82df-8742cebd8a0a)
## DeepSpeed Config
![Deepspeed Config](https://github.com/user-attachments/assets/c1833466-89b2-4f88-b9b3-f8b8695019e3)
## Training Script
![Training Script Part 1](https://github.com/user-attachments/assets/bda13ca1-d554-490b-a24a-a06fd525c366)
![Training Script Part 2](https://github.com/user-attachments/assets/89f3534d-6d95-485c-8b46-c13b5475c769)
![Training Script Part 3](https://github.com/user-attachments/assets/98835417-4ea8-4b56-8097-d932a5dd588b)
![Training Script Part 4](https://github.com/user-attachments/assets/2f40efba-d0e4-42cb-b0ff-0159cee9c087)
![Training Script Part 5](https://github.com/user-attachments/assets/ea2a3ae1-e655-411f-8f72-0d93a93042ca)

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

## The error:
![Error](https://github.com/user-attachments/assets/393c7dc3-41bc-4879-82df-8742cebd8a0a)
## DeepSpeed Config
![Deepspeed Config](https://github.com/user-attachments/assets/c1833466-89b2-4f88-b9b3-f8b8695019e3)
## Training Script
![Training Script Part 1](https://github.com/user-attachments/assets/bda13ca1-d554-490b-a24a-a06fd525c366)
![Training Script Part 2](https://github.com/user-attachments/assets/89f3534d-6d95-485c-8b46-c13b5475c769)
![Training Script Part 3](https://github.com/user-attachments/assets/98835417-4ea8-4b56-8097-d932a5dd588b)
![Training Script Part 4](https://github.com/user-attachments/assets/2f40efba-d0e4-42cb-b0ff-0159cee9c087)
![Training Script Part 5](https://github.com/user-attachments/assets/ea2a3ae1-e655-411f-8f72-0d93a93042ca)

### Expected behavior

Th error shown below occurs. Although the training arguments are already initialized in script. Am I doing something wrong here?

## Error:
![Error](https://github.com/user-attachments/assets/0acca445-8119-47b1-98f2-0f7cd4b2756e)


Comments:
In general: please submit issues by copy/pasting the code, not screenshots, as it's very hard to follow. This has been fixed in main and will be available in the next patch release.

For now, `pip install git+https://github.com/huggingface/transformers`

In this case though, which you will find upon reading the `TrainingArguments`, you **must** create your model after creating your training arguments when using zero3, which is what this error exactly states. 

Sorry about the screenshots, the code was on my work laptop. In the main function, in the Argument Parser, Training Arguments are being set so that is before initializing the model right?

Looks like yes. You’re in the one main area to why we reverted the PR (since bnb is different)

@muellerzr one more question, this problem got resolved. I was trying to train gemma 2 with 8192 context length with 4 gpu's AWS g6.12xlarge instance total 96 GB GPU. Thing is i think that the parameters are not getting offloaded to CPU. It always gives me OOM issue. The deepspeed config is as follows:

{
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true,
            "ratio": 1.0
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "zero_quantized_weights": true,
        "zero_hpz_partition_size": 4,
        "zero_quantized_gradients": true,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "sub_group_size": 1e3,
        "stage3_max_live_parameters": 1e3,
        "stage3_max_reuse_distance": 1e3,
        "stage3_gather_16bit_weights_on_model_save": true
    },
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "steps_per_print": 2000,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto"
}

Kindly Guide me Regarding this. Thanks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.