Deepspeed optimizer initalization fails (AttributeError: 'DummyOptim' object has no attribute 'step')
Author: jphme
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Linux-5.4.0-156-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.17.1
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0.dev0
- Accelerate config: 	yes
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:4*V100
- Using distributed or parallel set-up in script?: deepspeed

### Who can help?
cc @pacman100 

When trying to start a training with Deepspeed zero 2 and CPU offloading, optimizer initalization fails.

 I get weird errors ("AttributeError: 'DummyOptim' object has no attribute 'step'" for single GPU, "tried to get lr value before scheduler/optimizer started stepping, returning lr=0 "for multi GPU) that I didn't get yesterday with the exact same config (but on a different server and I rebuilt the docker image with transformers from main).

I tried different configs ("standard" config, setting everything to "auto", setting everything to fixed values, Adam + AdamW) but the error persists.

Any idea what could be the reason for this?

EDIT: I am launching with `accelerate launch` . #24640 could be related.

Stacktrace
```
  File "/workspace/axolotl/src/axolotl/train.py", line 120, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1575, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1954, in _inner_training_loop
    self.optimizer.step()
AttributeError: 'DummyOptim' object has no attribute 'step'
```

Example config (as described above happens with different setups):
zero2.json
```
{
    "zero_optimization": {
      "stage": 2,
      "contiguous_gradients": true,
      "offload_optimizer": {
              "device": "cpu",
              "pin_memory": true
      },
      "overlap_comm": true,
      "allgather_partitions": true,
      "allgather_bucket_size": 5e8,
      "reduce_scatter": true,
      "reduce_bucket_size": 5e8
    },
    "bf16": {
      "enabled": "auto"
    },
    "fp16": {
      "enabled": "auto",
      "loss_scale": 0,
      "initial_scale_power": 32,
      "loss_scale_window": 1000,
      "hysteresis": 2,
      "min_loss_scale": 1
    },
    "optimizer": {
      "type": "Adam",
      "params": {
        "lr": 0.00001,
        "betas": [
          0.9,
          0.999
        ],
        "eps": 1e-8,
        "weight_decay": 0.1
      }
    },
    "scheduler": {
      "type": "WarmupDecayLR",
      "params": {
        "warmup_min_lr": "auto",
        "warmup_max_lr": "auto",
        "warmup_num_steps": "auto",
        "total_num_steps": "auto"
      }
    },
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false,
    "flops_profiler": {
      "enabled": false,
      "detailed": false
    }   
}
```

Trainer
```
gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 1
optimizer: paged_adamw_32bit
#adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.00001
```

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

see above

### Expected behavior

Deepspeed optimizer initalizes, as it did yesterday with the same params.

Comments:
so how to solve it


Hello, I did the following and things work as expected:

1. vim ds_config_issue_26148.json
```
{
    "zero_optimization": {
      "stage": 2,
      "contiguous_gradients": true,
      "offload_optimizer": {
              "device": "cpu",
              "pin_memory": true
      },
      "overlap_comm": true,
      "allgather_partitions": true,
      "allgather_bucket_size": 5e8,
      "reduce_scatter": true,
      "reduce_bucket_size": 5e8
    },
    "bf16": {
      "enabled": "auto"
    },
    "fp16": {
      "enabled": "auto",
      "loss_scale": 0,
      "initial_scale_power": 32,
      "loss_scale_window": 1000,
      "hysteresis": 2,
      "min_loss_scale": 1
    },
    "optimizer": {
      "type": "Adam",
      "params": {
        "lr": 0.00001,
        "betas": [
          0.9,
          0.999
        ],
        "eps": 1e-8,
        "weight_decay": 0.1
      }
    },
    "scheduler": {
      "type": "WarmupDecayLR",
      "params": {
        "warmup_min_lr": "auto",
        "warmup_max_lr": "auto",
        "warmup_num_steps": "auto",
        "total_num_steps": "auto"
      }
    },
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false,
    "flops_profiler": {
      "enabled": false,
      "detailed": false
    }   
}
```

2. `accelerate config --config_file config.yaml`:
```
compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
  deepspeed_config_file: ds_config_issue_26148.json
  zero3_init_flag: true
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

3. running below commands:
```
export TASK_NAME=mrpc
export WANDB_DISABLED="true"

 accelerate launch --config_file ds_config.yaml ./examples/pytorch/text-classification/run_glue.py  --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 16 --learning_rate 1e-05 --weight_decay 0.1  --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --bf16
```

output logs:
```
[2023-10-02 08:36:56,659] [INFO] [config.py:957:print_user_config]   json = {
    "zero_optimization": {
        "stage": 2, 
        "contiguous_gradients": true, 
        "offload_optimizer": {
            "device": "cpu", 
            "pin_memory": true
        }, 
        "overlap_comm": true, 
        "allgather_partitions": true, 
        "allgather_bucket_size": 5.000000e+08, 
        "reduce_scatter": true, 
        "reduce_bucket_size": 5.000000e+08
    }, 
    "bf16": {
        "enabled": true
    }, 
    "fp16": {
        "enabled": false, 
        "loss_scale": 0, 
        "initial_scale_power": 32, 
        "loss_scale_window": 1000, 
        "hysteresis": 2, 
        "min_loss_scale": 1
    }, 
    "optimizer": {
        "type": "Adam", 
        "params": {
            "lr": 1e-05, 
            "betas": [0.9, 0.999], 
            "eps": 1e-08, 
            "weight_decay": 0.1
        }
    }, 
    "scheduler": {
        "type": "WarmupDecayLR", 
        "params": {
            "warmup_min_lr": 0, 
            "warmup_max_lr": 1e-05, 
            "warmup_num_steps": 0, 
            "total_num_steps": 345
        }
    }, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 1.0, 
    "train_batch_size": 32, 
    "train_micro_batch_size_per_gpu": 16, 
    "wall_clock_breakdown": false, 
    "flops_profiler": {
        "enabled": false, 
        "detailed": false
    }, 
    "steps_per_print": inf
}
[INFO|trainer.py:1760] 2023-10-02 08:36:56,659 >> ***** Running training *****
[INFO|trainer.py:1761] 2023-10-02 08:36:56,659 >>   Num examples = 3,668
[INFO|trainer.py:1762] 2023-10-02 08:36:56,659 >>   Num Epochs = 3
[INFO|trainer.py:1763] 2023-10-02 08:36:56,659 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1766] 2023-10-02 08:36:56,659 >>   Total train batch size (w. parallel, distributed & accumulation) = 32
[INFO|trainer.py:1767] 2023-10-02 08:36:56,659 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1768] 2023-10-02 08:36:56,659 >>   Total optimization steps = 345
[INFO|trainer.py:1769] 2023-10-02 08:36:56,660 >>   Number of trainable parameters = 108,311,810
  0%|                                                                                                  | 0/345 [00:00<?, ?it/s]/home/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:1886: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
  overflow_gpu = get_accelerator().ByteTensor([overflow])
/home/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:1886: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
  overflow_gpu = get_accelerator().ByteTensor([overflow])
100%|████████████████████████████████████████████████████████████████████████████████████████| 345/345 [01:15<00:00,  4.62it/s][INFO|trainer.py:2017] 2023-10-02 08:38:12,075 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 75.4148, 'train_samples_per_second': 145.913, 'train_steps_per_second': 4.575, 'train_loss': 0.5232365262681159, 'epoch': 3.0}
100%|████████████████████████████████████████████████████████████████████████████████████████| 345/345 [01:15<00:00,  4.57it/s]
[INFO|trainer.py:2939] 2023-10-02 08:38:12,155 >> Saving model checkpoint to /tmp/mrpc/
[INFO|configuration_utils.py:460] 2023-10-02 08:38:12,156 >> Configuration saved in /tmp/mrpc/config.json
[INFO|modeling_utils.py:2114] 2023-10-02 08:38:12,454 >> Model weights saved in /tmp/mrpc/pytorch_model.bin
[INFO|tokenization_utils_base.py:2445] 2023-10-02 08:38:12,455 >> tokenizer config file saved in /tmp/mrpc/tokenizer_config.json
[INFO|tokenization_utils_base.py:2454] 2023-10-02 08:38:12,455 >> Special tokens file saved in /tmp/mrpc/special_tokens_map.json
***** train metrics *****
  epoch                    =        3.0
  train_loss               =     0.5232
  train_runtime            = 0:01:15.41
  train_samples            =       3668
  train_samples_per_second =    145.913
  train_steps_per_second   =      4.575
10/02/2023 08:38:12 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:761] 2023-10-02 08:38:12,491 >> The following columns in the evaluation set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: idx, sentence1, sentence2. If idx, sentence1, sentence2 are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.
[INFO|trainer.py:3213] 2023-10-02 08:38:12,493 >> ***** Running Evaluation *****
[INFO|trainer.py:3215] 2023-10-02 08:38:12,493 >>   Num examples = 408
[INFO|trainer.py:3218] 2023-10-02 08:38:12,493 >>   Batch size = 8
100%|██████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 85.34it/s]
***** eval metrics *****
  epoch                   =        3.0
  eval_accuracy           =     0.7966
  eval_combined_score     =     0.8256
  eval_f1                 =     0.8546
  eval_loss               =     0.4453
  eval_runtime            = 0:00:00.31
  eval_samples            =        408
  eval_samples_per_second =   1281.051
  eval_steps_per_second   =     81.636
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The issue went away with newer library versions, many thanks.

> The issue went away with newer library versions, many thanks.


Have you discovered any answers to the issue you mentioned? It appears I've run into the same problem : "tried to get the lr value before the scheduler/optimizer began stepping, resulting in lr=0."