Problem initializing Deepspeed with Trainer
Author: lhallee
Pull request: no

### System Info
```python
2023-08-24 17:23:17.908613: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[2023-08-24 17:23:20,478] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/transformers/commands/env.py:100: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-08-24 17:23:29.664543: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
```
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.32.0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@pacman

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Load deepspeed config into json file
2. Pass into TrainingArguments
3. Get error

Here is my code:
```python
import json

deepspeed_config = {
    "fp16": {
        "enabled": True,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },
    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto"
        }
    },
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": True
        },
        "allgather_partitions": True,
        "allgather_bucket_size": 2e8,
        "overlap_comm": True,
        "reduce_scatter": True,
        "reduce_bucket_size": 2e8,
        "contiguous_gradients": True
    },
    "steps_per_print": 100,
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "activation_checkpointing": {
        "partition_activations": True,
        "contiguous_memory_optimization": True
    },
    "wall_clock_breakdown": False
}


config_filename = "deepspeed_config.json"

with open(config_filename, 'w') as f:
    json.dump(deepspeed_config, f)

trainer = Trainer(
    model=model,
    train_dataset=train_data,
    eval_dataset=val_data,
    args=TrainingArguments(
        num_train_epochs=num_epochs,
        evaluation_strategy='steps' if val_set_size > 0 else 'no',
        save_strategy='steps',
        eval_steps=eval_steps if val_set_size > 0 else None,
        save_steps=save_steps,
        output_dir=output_dir,
        save_total_limit=save_total_limit,
        load_best_model_at_end=True if val_set_size > 0 else False,
        deepspeed='./deepspeed_config.json',
    ),
    data_collator=DataCollatorForSeq2Seq(tokenizer, pad_to_multiple_of=8, return_tensors='pt', padding=True),
    callbacks=[print_callback]
)
model.config.use_cache = False
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
model.save_pretrained(output_dir)
```
Here is my error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-55-5c7ff182bcf8>](https://localhost:8080/#) in <cell line: 1>()
      3     train_dataset=train_data,
      4     eval_dataset=val_data,
----> 5     args=TrainingArguments(
      6         num_train_epochs=num_epochs,
      7         evaluation_strategy='steps' if val_set_size > 0 else 'no',

3 frames
[/usr/local/lib/python3.10/dist-packages/transformers/deepspeed.py](https://localhost:8080/#) in __init__(self, config_file_or_dict)
     64         dep_version_check("accelerate")
     65         dep_version_check("deepspeed")
---> 66         super().__init__(config_file_or_dict)
     67 
     68 

TypeError: object.__init__() takes exactly one argument (the instance to initialize)
```
### Expected behavior

Trainer loads and runs.

PS. This is my first ever issue reported, I'm a domain scientist sorry if this isn't the normal way to report things.

Comments:
Hey! Given the error, pretty sure you can just initialize the `TrainingArguments` with:
```python 
TrainingArguments(dict(num_train_epochs=num_epochs,
        evaluation_strategy='steps' if val_set_size > 0 else 'no',
        save_strategy='steps',
        eval_steps=eval_steps if val_set_size > 0 else None,
        save_steps=save_steps,
        output_dir=output_dir,
        save_total_limit=save_total_limit,
        load_best_model_at_end=True if val_set_size > 0 else False,
        deepspeed='./deepspeed_config.json',
))
```
The traceback shows that the issue is with `deepspeed`.

Tips for next time:
- properly format the code
- make the snippet runnable (`TrainingArguments`, `Trainer` not imported etc) 


This is what worked below. Side note, this crashed my session with over 80 gb of ram for llama2-7b. Is that to be expected? Is this due to the pin memory argument?

args = TrainingArguments(num_train_epochs=num_epochs,
        evaluation_strategy='steps' if val_set_size > 0 else 'no',
        save_strategy='steps',
        eval_steps=eval_steps if val_set_size > 0 else None,
        save_steps=save_steps,
        output_dir=output_dir,
        save_total_limit=save_total_limit,
        fp16=True,
        load_best_model_at_end=True if val_set_size > 0 else False,
        deepspeed='./deepspeed_config.json')

trainer = Trainer(
    model=model,
    train_dataset=train_data,
    eval_dataset=val_data,
    args=args,
    data_collator=DataCollatorForSeq2Seq(tokenizer, pad_to_multiple_of=8, return_tensors='pt', padding=True),
    callbacks=[print_callback])

model.config.use_cache = False
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
model.save_pretrained(output_dir)

So you have to initialize TraingingArguments outside of Trainer when using deepspeed? This wasn't the case without it. Works for me though, thanks for the help.

More info for my question above:

This config is OOM for CPU ram on llama-2 7billion on A10040gb with 80gb CPU ram. Curious why this takes up so much space.

deepspeed_config = {
    "fp16": {
        "enabled": True,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },
    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto"
        }
    },
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "cpu",
        },
    },
    "steps_per_print": 100,
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "activation_checkpointing": {
        "partition_activations": True,
        "contiguous_memory_optimization": True
    },
    "wall_clock_breakdown": False
}

Outputs

[2023-08-25 18:18:59,217] [INFO] [comm.py:631:init_distributed] cdb=None
[2023-08-25 18:18:59,218] [INFO] [comm.py:662:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Using /root/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py310_cu118/cpu_adam...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Loading extension module cpu_adam...
Time to load cpu_adam op: 36.26453471183777 seconds
Rank: 0 partition count [1] and sizes[(6738595840, False)] 

cc @pacman100 

I am facing the same issue. Even initialising Training Arguments separately doesn't work.

@lhallee, how are you launching the script?

@tryout3, can't help without any info on versions, minimal example, the config, the command you are running and the hardware on which you are running

@pacman100 not sure what you mean. I am just running the code in a colab notebook.

Hello @lhallee, given that you are using colab notebook, please follow the instructions given here: https://huggingface.co/docs/transformers/main_classes/deepspeed#deployment-in-notebooks.

Next, regarding CPU OOM, given that you are offloading optimizer, for 7B params:
1. FP32 master model weights: 4 Bytes per param * 7B params = 28GB
2. First and second moments for Adam Optimizer: 2 * (4 Bytes per param * 7B params) = 56GB
3. Total Memory = 28GB + 56GB = 84GB > 80GB RAM, hence CPU OOM

Thanks for the clarification! Is there a set of settings you can think of that would accommodate this? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@lhallee Have you solved this problem? I encountered the same problem!

Other than use a machine with more RAM, no.

> @lhallee Have you solved this problem? I encountered the same problem!

In my case, I re-install `transformers==4.31.0` and `accelerate==0.21.0` and `deepspeed==0.9.5`, and problem solved.

I found some codes for 1y ago that was running successfully, but currently all of them failing with this exact error.

Seems something wrong in transformers `class HfDeepSpeedConfig(DeepSpeedConfig)`
```python
/usr/local/lib/python3.10/dist-packages/transformers/integrations/deepspeed.py in __init__(self, config_file_or_dict)
     89 
     90     def __init__(self, config_file_or_dict):
---> 91         super().__init__(config_file_or_dict)
     92         self._dtype = None
     93         self.mismatches = []

/usr/local/lib/python3.10/dist-packages/transformers/integrations/deepspeed.py in __init__(self, config_file_or_dict)
     79         dep_version_check("accelerate")
     80         dep_version_check("deepspeed")
---> 81         super().__init__(config_file_or_dict)
     82 
     83 

TypeError: object.__init__() takes exactly one argument (the instance to initialize)
```

env:
```
transformers:   4.47.1
accelerate:     1.2.1 
deepspeed:      0.16.2
```

@steveepreston I'm don't have a fix, but you could use 0.15.x version of deepspeed. I don't get this error in that version

@cosmic-heart Thank you

I switched to FSDP after getting that error on all examples and notebooks.
I thought it's always a bad practice to back to old versions. 

Can u share the config/code you used for fsdp?

@cosmic-heart Look at this notebook:

[tpuv5e_llama2_pytorch_finetuning_and_serving.ipynb](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/tpuv5e_llama2_pytorch_finetuning_and_serving.ipynb)