'DummyOptim' object has no attribute 'step'
Author: karthsu
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.31
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes 4*A100 80GB

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am trying to train a model from the given [script](https://github.com/salesforce/CodeT5/blob/main/CodeT5%2B/instruct_tune_codet5p.py) in a single node multi-GPU setting 
 with DeepSpeed integration and am getting the error as given below
To reproduce one can download the script from [here](https://github.com/salesforce/CodeT5/blob/main/CodeT5%2B) along with the config file and try to run it with the [CodeAlpaca dataset](https://raw.githubusercontent.com/sahil280114/codealpaca/master/data/code_alpaca_20k.json)

```
{'batch_size_per_replica': 1,
 'cache_data': 'cache_data/instructions',
 'data_num': -1,
 'deepspeed': 'deepspeed_config.json',
 'epochs': 3,
 'fp16': False,
 'grad_acc_steps': 16,
 'instruct_data_path': 'code_alpaca_20k.json',
 'load': 'codet5p-16b',
 'local_rank': -1,
 'log_freq': 10,
 'lr': 2e-05,
 'lr_warmup_steps': 30,
 'max_len': 512,
 'save_dir': 'saved_models/instruct_codet5p_16b',
 'save_freq': 500}
  ==> Loaded 20022 samples
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:25<00:00,  5.15s/it]
  ==> Loaded model from codet5p-16b, model size 16493680640
Para before freezing: 16493680640, trainable para: 16494M
Para after freezing: 16493680640, trainable para: 462M
Starting main loop
  0%|                                                                                                                                                                                                                                         | 0/936 [00:00<?, ?it/s]/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '

Traceback (most recent call last):
  File "/root/Custom-LLM/CodeT5/CodeT5+/instruct_tune_codet5p.py", line 212, in <module>
    main(args)
  File "/root/Custom-LLM/CodeT5/CodeT5+/instruct_tune_codet5p.py", line 181, in main
    run_training(args, model, train_data)
  File "/root/Custom-LLM/CodeT5/CodeT5+/instruct_tune_codet5p.py", line 93, in run_training
    trainer.train()
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1881, in _inner_training_loop
    self.optimizer.step()
    ^^^^^^^^^^^^^^^^^^^
AttributeError: 'DummyOptim' object has no attribute 'step'
```

The same script works completely fine in a single-GPU setting. But when I switch to a multi-GPU setup I get this error

### Expected behavior

Expect the training to proceed using Deepspeed as in the case of a single-GPU set up

Comments:
cc @pacman100 

Hello, please provide a minimal reproducible example that we can directly run. Providing links to scripts and dataset doesn't help and is very involved and time taking. 

Also, please provide the accelerate and DeepSpeed versions, the launch command for the minimal example and the minimal example as mentioned above.

Code:
```
"""
Finetune CodeT5+ models on instruction tuning data
You can customize your own training data by following the HF dataset format to cache it to args.cache_data
Author: Yue Wang
Date: June 2023
"""

import os
import pprint
import argparse
import numpy as np
import copy
import torch
from datasets import load_dataset, load_from_disk
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, TrainingArguments, Trainer

PROMPT_DICT = {
    "prompt_input": (
        "Below is an instruction that describes a task, paired with an input that provides further context. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:"
    ),
    "prompt_no_input": (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n{instruction}\n\n### Response:"
    ),
}


def get_model_size(model):
    model_parameters = filter(lambda p: p.requires_grad, model.parameters())
    model_size = sum([np.prod(p.size()) for p in model_parameters])
    return "{}M".format(round(model_size / 1e+6))


def freeze_decoder_except_xattn_codegen(model):
    print(f'Para before freezing: {model.num_parameters()}, trainable para: {get_model_size(model)}')
    for param in model.decoder.parameters():
        param.requires_grad = False

    num_decoder_layers = model.decoder.config.num_layers
    for i in range(num_decoder_layers):
        each_decoder_layer = model.decoder.transformer.h[i]
        if hasattr(each_decoder_layer, 'crossattention'):
            for param in each_decoder_layer.crossattention.parameters():
                param.requires_grad = True
            each_decoder_layer.crossattention.to(torch.float32)

        if hasattr(each_decoder_layer, 'alpha_xattn'):
            each_decoder_layer.alpha_xattn.requires_grad = True
    print(f'Para after freezing: {model.num_parameters()}, trainable para: {get_model_size(model)}')


def run_training(args, model, train_data):
    print(f"Starting main loop")

    training_args = TrainingArguments(
        #report_to='tensorboard',
        output_dir=args.save_dir,
        overwrite_output_dir=False,

        do_train=True,
        save_strategy='epoch',

        num_train_epochs=args.epochs,
        per_device_train_batch_size=args.batch_size_per_replica,
        gradient_accumulation_steps=args.grad_acc_steps,

        learning_rate=args.lr,
        weight_decay=0.0,
        warmup_steps=args.lr_warmup_steps,

        logging_dir=args.save_dir,
        logging_first_step=True,
        logging_steps=args.log_freq,
        save_total_limit=2,

        dataloader_drop_last=True,
        dataloader_num_workers=4,

        local_rank=args.local_rank,
        deepspeed=args.deepspeed,
        fp16=args.fp16,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_data,
    )

    trainer.train()

    if args.local_rank in [0, -1]:
        final_checkpoint_dir = os.path.join(args.save_dir, "final_checkpoint")
        model.save_pretrained(final_checkpoint_dir)
        print(f'  ==> Finish training and save to {final_checkpoint_dir}')


def load_tokenize_data(args):
    # Load and tokenize data
    if os.path.exists(args.cache_data):
        train_data = load_from_disk(args.cache_data)
        print(f'  ==> Loaded {len(train_data)} samples')
        return train_data
    else:
        datasets = load_dataset('json', data_files=args.instruct_data_path)['train']
        tokenizer = AutoTokenizer.from_pretrained(args.load)

        def preprocess_function(examples):
            prompt_input, prompt_no_input = PROMPT_DICT["prompt_input"], PROMPT_DICT["prompt_no_input"]
            source = [prompt_input.format_map({'instruction': instruct, 'input': inp}) if inp != ''
                      else prompt_no_input.format_map({'instruction': instruct})
                      for instruct, inp in zip(examples["instruction"], examples["input"])]
            target = [src + output + tokenizer.eos_token for src, output in zip(source, examples["output"])]

            model_inputs = tokenizer(source, max_length=args.max_len, padding="max_length", truncation=True)
            labels = tokenizer(target, max_length=args.max_len, padding="max_length", truncation=True)
            model_inputs["decoder_input_ids"] = copy.deepcopy(labels["input_ids"])

            # changing labels: convert all tokens in the duplicate prefix prompt and the padding part to -100
            eos_token_id = tokenizer.eos_token_id
            for x, y in zip(model_inputs["input_ids"], labels["input_ids"]):
                label_prefix_len = x.index(eos_token_id) if eos_token_id in x else len(x)
                y[:label_prefix_len] = [-100] * label_prefix_len

                if eos_token_id in y:
                    pad_len = len(y) - y.index(eos_token_id) - 1
                    if pad_len > 0:
                        y[y.index(eos_token_id) + 1:] = [-100] * pad_len

            # shift labels to the right as the decoder input and add decoder start token id
            decoder_start_id = tokenizer.eos_token_id
            for z in model_inputs["decoder_input_ids"]:
                z[1:] = z[:-1]
                z[0] = decoder_start_id

            model_inputs["labels"] = copy.deepcopy(labels["input_ids"])
            model_inputs["decoder_attention_mask"] = labels["attention_mask"]
            return model_inputs

        train_data = datasets.map(
            preprocess_function,
            batched=True,
            remove_columns=datasets.column_names,
            num_proc=64,
            load_from_cache_file=False,
        )

        print(f'  ==> Loaded {len(train_data)} samples')
        train_data.save_to_disk(args.cache_data)
        print(f'  ==> Saved to {args.cache_data}')
        return train_data


def main(args):
    argsdict = vars(args)
    print(pprint.pformat(argsdict))

    # Save command to file
    with open(os.path.join(args.save_dir, "command.txt"), 'w') as f:
        f.write(pprint.pformat(argsdict))

    # Load and tokenize data using the tokenizer from `args.load`. If the data is already cached, load it from there.
    # You can customize this function to load your own data for any Seq2Seq LM tasks.
    train_data = load_tokenize_data(args)

    if args.data_num != -1:
        train_data = train_data.select([i for i in range(args.data_num)])

    # Load model from `args.load`
    model = AutoModelForSeq2SeqLM.from_pretrained(args.load, torch_dtype=torch.float16,
                                                  low_cpu_mem_usage=True, trust_remote_code=True)

    print(f"  ==> Loaded model from {args.load}, model size {model.num_parameters()}")
    #freeze_decoder_except_xattn_codegen(model)

    run_training(args, model, train_data)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="CodeT5+ instruction tuning")
    parser.add_argument('--data-num', default=-1, type=int)
    parser.add_argument('--max-len', default=512, type=int)
    parser.add_argument('--instruct-data-path', default='code_alpaca_20k.json', type=str)
    parser.add_argument('--cache-data', default='cache_data/instructions', type=str)
    parser.add_argument('--load', default='Salesforce/codet5p-16b', type=str)

    # Training
    parser.add_argument('--epochs', default=3, type=int)
    parser.add_argument('--lr', default=2e-5, type=float)
    parser.add_argument('--lr-warmup-steps', default=30, type=int)
    parser.add_argument('--batch-size-per-replica', default=1, type=int)
    parser.add_argument('--grad-acc-steps', default=16, type=int)
    parser.add_argument('--local_rank', default=-1, type=int)
    parser.add_argument('--deepspeed', default=None, type=str)
    parser.add_argument('--fp16', default=False, action='store_true')

    # Logging and stuff
    parser.add_argument('--save-dir', default="saved_models/instruct_codet5p_16b", type=str)
    parser.add_argument('--log-freq', default=10, type=int)
    parser.add_argument('--save-freq', default=500, type=int)

    args = parser.parse_args()

    os.makedirs(args.save_dir, exist_ok=True)

    main(args)
```

command:
```
deepspeed CodeT5+/instruct_tune_codet5p.py --load $MODEL --save-dir $SAVE_DIR --instruct-data-path code_alpaca_2k.json --fp16 --deepspeed ~/transformers/tests/deepspeed/ds_config_zero3.json
```

Output:
![Screenshot 2023-07-04 at 4 44 34 PM](https://github.com/huggingface/transformers/assets/13534540/cb880856-433f-4f2a-b5e1-40b91c845566)

- `Accelerate` version: 0.21.0.dev0
- Platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Numpy version: 1.24.4
- PyTorch version (GPU?): 2.0.1 (True)
- PyTorch XPU available: False
- System RAM: 503.55 GB
- GPU type: NVIDIA A100-SXM4-80GB
- `Accelerate` default config:
	Not found
- 
- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

```bash
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch']
torch version .................... 2.0.1
deepspeed install path ........... ['/home/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.5, unknown, unknown
torch cuda version ............... 11.8
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.8
```

Therefore, unable to reproduce your error.  

I see similar issue. 

I'm also having this issue!

I believe it happens when you include an `"optimizer"` as part of your Deepspeed config. E.g.

```json
"optimizer": {
    "type": "AdamW",
    "params": {
         "lr": "auto",
         "betas": "auto",
         "eps": "auto",
         "weight_decay": "auto",
     },
}
```

Then, `optimizer = DummyOptim(params=model_parameters)`.

https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/deepspeed.py#L282-L288

And eventually `Trainer` tries to call `optimizer.step()`. However, `DummyOptim` doesn't have a `step()` function.

https://github.com/huggingface/accelerate/blob/8514c35192ac9762920f1ab052e5cea4c0e46eeb/src/accelerate/utils/deepspeed.py#L226-L246

I'm not sure what the appropriate fix is: should we not use `DummyOptim` at all or add a `step()` function (that does nothing?) to it?

P.S. the same problem applies to `"scheduler"`.

https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/deepspeed.py#L303-L304

Please take a look, @pacman100 -- thanks!

@apoorvkh , post the `accelerator.prepare` they should be replaced with correct optimizer and scheduler from DeepSpeed and hence should not result in any issues. As shown above, unable to reproduce it. A minimal way to reproduce it would help me deep dive

See https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1653-L1656 wherein it `accelerator.prepare` internally calls `deepspeed.initialize` and replace the dummy objects with appropriate  ones returned by DeepSpeed.

Thanks, that was helpful! Using that information, we found that the cause was because `ACCELERATE_USE_DEEPSPEED=true` was not already being set internally. (If you set it manually, everything works as expected.)

We unfortunately can't very easily share a minimal reproducible example, but will continue to debug soon.

Keeping you posted, the issue I encountered was on our end: for our own reasons, we initialized `TrainingArguments` without `deepspeed = ...`, then set `TrainingArguments.deepspeed = ...` and called `TrainingArguments.__post_init__()` manually. So`__post_init__` is called twice in total for the same dataclass object.

This worked fine in previous versions of `transformers`, but not on the current `main` branch. This does not result in the same behavior related to Accelerate and Deepspeed as creating a TrainingArguments object with all the flags in one go.

Anyway, just sharing in case you want to refine the internal behavior to fix this case. But it's not a problem if you use TrainingArguments in the standard way. I'm not sure if this is related to the what the original creator of this issue had encountered. Thanks!

> Keeping you posted, the issue I encountered was on our end: for our own reasons, we initialized `TrainingArguments` without `deepspeed = ...`, then set `TrainingArguments.deepspeed = ...` and called `TrainingArguments.__post_init__()` manually. So`__post_init__` is called twice in total for the same dataclass object.
> 
> This worked fine in previous versions of `transformers`, but not on the current `main` branch. This does not result in the same behavior related to Accelerate and Deepspeed as creating a TrainingArguments object with all the flags in one go.
> 
> Anyway, just sharing in case you want to refine the internal behavior to fix this case. But it's not a problem if you use TrainingArguments in the standard way. I'm not sure if this is related to the what the original creator of this issue had encountered. Thanks!

what that's mean to "use TrainingArguments in the standard way"?
    ds_plugin = DeepSpeedPlugin(deepspeed_config)
    accelerator = Accelerator(deepspeed_plugin=ds_plugin)
    scheduler, optimizer = get_dummy_scheduler_optimizer(model)
    model, optimizer, train_dataset, scheduler = accelerator.prepare(model, optimizer, train_dataset, scheduler)  


    training_args = TrainingArguments(
        output_dir=os.path.join(model_path, config_obj.model['checkpoints_path']),
        overwrite_output_dir=True,
        num_train_epochs=config_obj.model['epochs'],
        per_device_train_batch_size=config_obj.model['batch_size'],
        logging_strategy="no",
        save_strategy="no",
        deepspeed=deepspeed_config
    )

    # Initialize the Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
    )
    
deepspeed_config = {
                            "optimizer": {
                            "type": "AdamW",
                            "params": {
                                  "lr": "auto",
                                  "weight_decay": "auto",
                            }
                          },
                          "scheduler": {
                            "type": "WarmupLR",
                            "params": {
                                "warmup_min_lr": "auto",
                                "warmup_max_lr": "auto",
                                "warmup_num_steps": "auto",
                                }
                                  },
                                  "zero_optimization": {
                                    "stage": 3,
                                    "offload_optimizer": {
                                        "device": "cpu", 
                                        "pin_memory": True
                                    }, 
                                    "offload_param": {
                                        "device": "cpu", 
                                        "pin_memory": True
                                    },
                                    "contiguous_gradients": True, 
                                  },
                                  "gradient_accumulation_steps": "auto",
                                  "gradient_clipping": "auto",
                                  "train_batch_size": "auto",
                                  "train_micro_batch_size_per_gpu": "auto"
                                }
                                I wonder how to fix that problem...
even with a trick like

from accelerate.utils import DummyOptim, DummyScheduler
class DummyOptimizerWithStep(DummyOptim):
    def __init__(self, optimizer_grouped_parameters, *args, **kwargs):
        super().__init__(params=optimizer_grouped_parameters, *args, **kwargs)
        
    def step(self):
        pass
                  
 I get the same error :(           

I was having this error too, it seemed like the issue was that the `distributed_state` attribute of the TrainingArguments object wasn't being set properly. I overrode it with the following code snippet:

```python
from transformers import TrainingArguments
from accelerate.utils import DistributedType

ds_config = {
      "zero_optimization": {
        "stage": 2, # stage 2 is ideal, want to see if stage 1 avoids error
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": False # changed to False to see if memory usage is better
        },
        "allgather_partitions": True,
        "allgather_bucket_size": 2e8,
        "reduce_scatter": True,
        "reduce_bucket_size": 2e8,
        "overlap_comm": False,
        "contiguous_gradients": False,
      },
      "optimizer": {
          "type": "Adam",
          "params": {
            "lr": 1e-3,
        },
      },
      "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": 1e-3,
            "warmup_max_lr": 1e-3,
            "warmup_num_steps": 100
        }
      },
      "fp16": {
          "enabled": False,
      },
      "train_batch_size": 8,
      "train_micro_batch_sie_per_gpu": 8,
}
args = {
        'group_by_length': True,
        'per_device_train_batch_size': 1,
        'evaluation_strategy': "steps",
        'num_train_epochs': 4,
        'gradient_checkpointing': True,
        'fp16': False,
        'save_steps': 400,
        'eval_steps': 100,
        'logging_steps': 100,
        'learning_rate': 1e-3,
        'warmup_steps': 100,
        'save_total_limit': 2,
        'push_to_hub': True,
    }

training_args = TrainingArguments(deepspeed=ds_config, **args)
# set distributed_state manually
training_args.distributed_state.distributed_type = DistributedType.DEEPSPEED
```

And then I could run `trainer.train()` without any errors.

Happy to provide more code or system info if that would help.

I have the same error.

> I was having this error too, it seemed like the issue was that the distributed_state attribute of the TrainingArguments object wasn't being set properly.

```python
# set distributed_state manually
training_args.distributed_state.distributed_type = DistributedType.DEEPSPEED
```

If I set `DistributedType.DEEPSPEED`, the Train script requires "mpi4py" module.

```
ModuleNotFoundError: No module named 'mpi4py'
```

If it is possible, I don't want to install MPI when we train with single GPU. Are there any workaround to avoid installing MPI when using single GPU?

FWIW, this occurred for me because I was running my code with `python` instead of `deepspeed` but still had `--deepspeed myconfig.json` as one of the parameters.

The 
> training_args.distributed_state.distributed_type = DistributedType.DEEPSPEED

shows me the error 
KeyError: 'train_micro_batch_size_per_gpu'

if it helps anyone else: I was executing `python <myscript.py>`. Instead changed it to: `python -m torch.distributed.launch --nproc_per_node 1 <myscript.py>` which fixed it for me.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

when i train model like: 
```
export CUDA_VISIBLE_DEVICES=1
python run_clm.py ...
```
then:

![WX20231212-153656@2x](https://github.com/huggingface/transformers/assets/11456239/b325b6d5-9482-423e-b14f-80439402c5c9)

fix:
```
master_port=$(shuf -n 1 -i 10000-65535)
deepspeed --include localhost:1 --master_port "${master_port}" run_clm.py \
```



@apoorvkh 
> Keeping you posted, the issue I encountered was on our end: for our own reasons, we initialized `TrainingArguments` without `deepspeed = ...`, then set `TrainingArguments.deepspeed = ...` and called `TrainingArguments.__post_init__()` manually. So`__post_init__` is called twice in total for the same dataclass object.
> 
> This worked fine in previous versions of `transformers`, but not on the current `main` branch. This does not result in the same behavior related to Accelerate and Deepspeed as creating a TrainingArguments object with all the flags in one go.
> 
> Anyway, just sharing in case you want to refine the internal behavior to fix this case. But it's not a problem if you use TrainingArguments in the standard way. I'm not sure if this is related to the what the original creator of this issue had encountered. Thanks!

This method works for me. Thanks!