AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' for multinode training
Author: larrylawl
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-4.18.0-305.25.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: multinode distributed setup
- Deepspeed version: 0.9.5

### Who can help?

@pacman100

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

To reproduce, run the following with your `hostfile` and `deepspeed` config specified. I used the zero3 config [here](https://huggingface.co/docs/transformers/main_classes/deepspeed#zero3-config).

```
deepspeed --hostfile=$PBS_O_WORKDIR/hostfile $ROOT_DIR/transformers/examples/pytorch/language-modeling/run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir ~/scratch/finetuned_models/debug \
    --deepspeed "$ROOT_DIR/FastChat/fastchat/ds_config/ds_config_zero3.json"
```

I ran into the error `AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'`. Here's the logs:

```
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cce/13.0.2/cce/aarch64/lib/libcray-c++-rts.a')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cti/2.15.10/lib/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cce/13.0.2/cce/aarch64/lib')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cce/13.0.2/cce/x86_64/share/nls/En/%N.cat')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/mpich/8.1.15/ofi/@PRGENV@/@PE_MPICH_GENCOMPS@/lib/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('libfabric/1.11.0.4.125'), PosixPath('cray-pals/1.1.6'), PosixPath('craype-x86-rome'), PosixPath('craype-network-ofi'), PosixPath('cce/13.0.2'), PosixPath('cray-dsmml/0.2.2'), PosixPath('perftools-base/22.04.0'), PosixPath('cray-mpich/8.1.15'), PosixPath('craype/2.7.15'), PosixPath('PrgEnv-cray/8.3.3')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cce/13.0.2/cce/aarch64/include/craylibs')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/gcc-cross-aarch64/8.1.0/aarch64')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/fftw/3.3.8.13/@PE_FFTW_DEFAULT_TARGET@/lib/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/libsci/21.08.1.2/@PRGENV@/@PE_LIBSCI_DEFAULT_GENCOMPS@/@PE_LIBSCI_DEFAULT_TARGET@/lib/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/sma/11.5.3.beta/ofi/sma@PE_SMA_DEFAULT_DIR_DEFAULT64@/lib64/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/clmgr/man'), PosixPath('/opt/cray/pe/man')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/modulefiles')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('libexec64/opts')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: CUDA SETUP: Highest compute capability among GPUs detected: 8.0
x1000c1s1b0n0: CUDA SETUP: Detected CUDA version 116
x1000c1s1b0n0: CUDA SETUP: Loading binary /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('() {  eval `/opt/cray/pe/modules/3.2.11.6/bin/modulecmd bash $*`\n}')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: [2023-07-16 11:23:41,585] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
x1000c1s1b0n0: [2023-07-16 11:23:41,586] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
x1000c0s6b0n0: [2023-07-16 11:23:44,841] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
x1000c0s6b0n0: [2023-07-16 11:23:44,841] [INFO] [comm.py:594:init_distributed] cdb=None
x1000c0s6b0n0: [2023-07-16 11:23:44,841] [INFO] [comm.py:625:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
x1000c0s6b0n0: [2023-07-16 11:23:44,842] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
x1000c0s6b0n0: [2023-07-16 11:23:44,842] [INFO] [comm.py:594:init_distributed] cdb=None
x1000c1s1b0n0: [2023-07-16 11:23:46,769] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
x1000c1s1b0n0: [2023-07-16 11:23:46,770] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
x1000c1s1b0n0: [2023-07-16 11:23:46,770] [INFO] [comm.py:594:init_distributed] cdb=None
x1000c1s1b0n0: [2023-07-16 11:23:46,770] [INFO] [comm.py:594:init_distributed] cdb=None
x1000c0s6b0n0: 07/16/2023 11:23:47 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
x1000c0s6b0n0: 07/16/2023 11:23:47 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
x1000c0s6b0n0: _n_gpu=1,
x1000c0s6b0n0: adafactor=False,
x1000c0s6b0n0: adam_beta1=0.9,
x1000c0s6b0n0: adam_beta2=0.999,
x1000c0s6b0n0: adam_epsilon=1e-08,
x1000c0s6b0n0: auto_find_batch_size=False,
x1000c0s6b0n0: bf16=False,
x1000c0s6b0n0: bf16_full_eval=False,
x1000c0s6b0n0: data_seed=None,
x1000c0s6b0n0: dataloader_drop_last=False,
x1000c0s6b0n0: dataloader_num_workers=0,
x1000c0s6b0n0: dataloader_pin_memory=True,
x1000c0s6b0n0: ddp_backend=None,
x1000c0s6b0n0: ddp_broadcast_buffers=None,
x1000c0s6b0n0: ddp_bucket_cap_mb=None,
x1000c0s6b0n0: ddp_find_unused_parameters=None,
x1000c0s6b0n0: ddp_timeout=1800,
x1000c0s6b0n0: debug=[],
x1000c0s6b0n0: deepspeed=/home/users/industry/dso/lannliat/FastChat/fastchat/ds_config/ds_config_zero3.json,
x1000c0s6b0n0: disable_tqdm=False,
x1000c0s6b0n0: do_eval=True,
x1000c0s6b0n0: do_predict=False,
x1000c0s6b0n0: do_train=True,
x1000c0s6b0n0: eval_accumulation_steps=None,
x1000c0s6b0n0: eval_delay=0,
x1000c0s6b0n0: eval_steps=None,
x1000c0s6b0n0: evaluation_strategy=no,
x1000c0s6b0n0: fp16=False,
x1000c0s6b0n0: fp16_backend=auto,
x1000c0s6b0n0: fp16_full_eval=False,
x1000c0s6b0n0: fp16_opt_level=O1,
x1000c0s6b0n0: fsdp=[],
x1000c0s6b0n0: fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},
x1000c0s6b0n0: fsdp_min_num_params=0,
x1000c0s6b0n0: fsdp_transformer_layer_cls_to_wrap=None,
x1000c0s6b0n0: full_determinism=False,
x1000c0s6b0n0: gradient_accumulation_steps=1,
x1000c0s6b0n0: gradient_checkpointing=False,
x1000c0s6b0n0: greater_is_better=None,
x1000c0s6b0n0: group_by_length=False,
x1000c0s6b0n0: half_precision_backend=auto,
x1000c0s6b0n0: hub_model_id=None,
x1000c0s6b0n0: hub_private_repo=False,
x1000c0s6b0n0: hub_strategy=every_save,
x1000c0s6b0n0: hub_token=<HUB_TOKEN>,
x1000c0s6b0n0: ignore_data_skip=False,
x1000c0s6b0n0: include_inputs_for_metrics=False,
x1000c0s6b0n0: jit_mode_eval=False,
x1000c0s6b0n0: label_names=None,
x1000c0s6b0n0: label_smoothing_factor=0.0,
x1000c0s6b0n0: learning_rate=5e-05,
x1000c0s6b0n0: length_column_name=length,
x1000c0s6b0n0: load_best_model_at_end=False,
x1000c0s6b0n0: local_rank=0,
x1000c0s6b0n0: log_level=passive,
x1000c0s6b0n0: log_level_replica=warning,
x1000c0s6b0n0: log_on_each_node=True,
x1000c0s6b0n0: logging_dir=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug/runs/Jul16_11-23-44_x1000c0s6b0n0,
x1000c0s6b0n0: logging_first_step=False,
x1000c0s6b0n0: logging_nan_inf_filter=True,
x1000c0s6b0n0: logging_steps=500,
x1000c0s6b0n0: logging_strategy=steps,
x1000c0s6b0n0: lr_scheduler_type=linear,
x1000c0s6b0n0: max_grad_norm=1.0,
x1000c0s6b0n0: max_steps=-1,
x1000c0s6b0n0: metric_for_best_model=None,
x1000c0s6b0n0: mp_parameters=,
x1000c0s6b0n0: no_cuda=False,
x1000c0s6b0n0: num_train_epochs=3.0,
x1000c0s6b0n0: optim=adamw_hf,
x1000c0s6b0n0: optim_args=None,
x1000c0s6b0n0: output_dir=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug,
x1000c0s6b0n0: overwrite_output_dir=False,
x1000c0s6b0n0: past_index=-1,
x1000c0s6b0n0: per_device_eval_batch_size=8,
x1000c0s6b0n0: per_device_train_batch_size=8,
x1000c0s6b0n0: prediction_loss_only=False,
x1000c0s6b0n0: push_to_hub=False,
x1000c0s6b0n0: push_to_hub_model_id=None,
x1000c0s6b0n0: push_to_hub_organization=None,
x1000c0s6b0n0: push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
x1000c0s6b0n0: ray_scope=last,
x1000c0s6b0n0: remove_unused_columns=True,
x1000c0s6b0n0: report_to=['wandb'],
x1000c0s6b0n0: resume_from_checkpoint=None,
x1000c0s6b0n0: run_name=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug,
x1000c0s6b0n0: save_on_each_node=False,
x1000c0s6b0n0: save_safetensors=False,
x1000c0s6b0n0: save_steps=500,
x1000c0s6b0n0: save_strategy=steps,
x1000c0s6b0n0: save_total_limit=None,
x1000c0s6b0n0: seed=42,
x1000c0s6b0n0: sharded_ddp=[],
x1000c0s6b0n0: skip_memory_metrics=True,
x1000c0s6b0n0: tf32=None,
x1000c0s6b0n0: torch_compile=False,
x1000c0s6b0n0: torch_compile_backend=None,
x1000c0s6b0n0: torch_compile_mode=None,
x1000c0s6b0n0: torchdynamo=None,
x1000c0s6b0n0: tpu_metrics_debug=False,
x1000c0s6b0n0: tpu_num_cores=None,
x1000c0s6b0n0: use_ipex=False,
x1000c0s6b0n0: use_legacy_prediction_loop=False,
x1000c0s6b0n0: use_mps_device=False,
x1000c0s6b0n0: warmup_ratio=0.0,
x1000c0s6b0n0: warmup_steps=0,
x1000c0s6b0n0: weight_decay=0.0,
x1000c0s6b0n0: xpu_backend=None,
x1000c0s6b0n0: )
x1000c0s6b0n0: 07/16/2023 11:23:47 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
x1000c1s1b0n0: 07/16/2023 11:23:47 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
x1000c1s1b0n0: 07/16/2023 11:23:47 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
x1000c1s1b0n0: _n_gpu=1,
x1000c1s1b0n0: adafactor=False,
x1000c1s1b0n0: adam_beta1=0.9,
x1000c1s1b0n0: adam_beta2=0.999,
x1000c1s1b0n0: adam_epsilon=1e-08,
x1000c1s1b0n0: auto_find_batch_size=False,
x1000c1s1b0n0: bf16=False,
x1000c1s1b0n0: bf16_full_eval=False,
x1000c1s1b0n0: data_seed=None,
x1000c1s1b0n0: dataloader_drop_last=False,
x1000c1s1b0n0: dataloader_num_workers=0,
x1000c1s1b0n0: dataloader_pin_memory=True,
x1000c1s1b0n0: ddp_backend=None,
x1000c1s1b0n0: ddp_broadcast_buffers=None,
x1000c1s1b0n0: ddp_bucket_cap_mb=None,
x1000c1s1b0n0: ddp_find_unused_parameters=None,
x1000c1s1b0n0: ddp_timeout=1800,
x1000c1s1b0n0: debug=[],
x1000c1s1b0n0: deepspeed=/home/users/industry/dso/lannliat/FastChat/fastchat/ds_config/ds_config_zero3.json,
x1000c1s1b0n0: disable_tqdm=False,
x1000c1s1b0n0: do_eval=True,
x1000c1s1b0n0: do_predict=False,
x1000c1s1b0n0: do_train=True,
x1000c1s1b0n0: eval_accumulation_steps=None,
x1000c1s1b0n0: eval_delay=0,
x1000c1s1b0n0: eval_steps=None,
x1000c1s1b0n0: evaluation_strategy=no,
x1000c1s1b0n0: fp16=False,
x1000c1s1b0n0: fp16_backend=auto,
x1000c1s1b0n0: fp16_full_eval=False,
x1000c1s1b0n0: fp16_opt_level=O1,
x1000c1s1b0n0: fsdp=[],
x1000c1s1b0n0: fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},
x1000c1s1b0n0: fsdp_min_num_params=0,
x1000c1s1b0n0: fsdp_transformer_layer_cls_to_wrap=None,
x1000c1s1b0n0: full_determinism=False,
x1000c1s1b0n0: gradient_accumulation_steps=1,
x1000c1s1b0n0: gradient_checkpointing=False,
x1000c1s1b0n0: greater_is_better=None,
x1000c1s1b0n0: group_by_length=False,
x1000c1s1b0n0: half_precision_backend=auto,
x1000c1s1b0n0: hub_model_id=None,
x1000c1s1b0n0: hub_private_repo=False,
x1000c1s1b0n0: hub_strategy=every_save,
x1000c1s1b0n0: hub_token=<HUB_TOKEN>,
x1000c1s1b0n0: ignore_data_skip=False,
x1000c1s1b0n0: include_inputs_for_metrics=False,
x1000c1s1b0n0: jit_mode_eval=False,
x1000c1s1b0n0: label_names=None,
x1000c1s1b0n0: label_smoothing_factor=0.0,
x1000c1s1b0n0: learning_rate=5e-05,
x1000c1s1b0n0: length_column_name=length,
x1000c1s1b0n0: load_best_model_at_end=False,
x1000c1s1b0n0: local_rank=0,
x1000c1s1b0n0: log_level=passive,
x1000c1s1b0n0: log_level_replica=warning,
x1000c1s1b0n0: log_on_each_node=True,
x1000c1s1b0n0: logging_dir=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug/runs/Jul16_11-23-46_x1000c1s1b0n0,
x1000c1s1b0n0: logging_first_step=False,
x1000c1s1b0n0: logging_nan_inf_filter=True,
x1000c1s1b0n0: logging_steps=500,
x1000c1s1b0n0: logging_strategy=steps,
x1000c1s1b0n0: lr_scheduler_type=linear,
x1000c1s1b0n0: max_grad_norm=1.0,
x1000c1s1b0n0: max_steps=-1,
x1000c1s1b0n0: metric_for_best_model=None,
x1000c1s1b0n0: mp_parameters=,
x1000c1s1b0n0: no_cuda=False,
x1000c1s1b0n0: num_train_epochs=3.0,
x1000c1s1b0n0: optim=adamw_hf,
x1000c1s1b0n0: optim_args=None,
x1000c1s1b0n0: output_dir=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug,
x1000c1s1b0n0: overwrite_output_dir=False,
x1000c1s1b0n0: past_index=-1,
x1000c1s1b0n0: per_device_eval_batch_size=8,
x1000c1s1b0n0: per_device_train_batch_size=8,
x1000c1s1b0n0: prediction_loss_only=False,
x1000c1s1b0n0: push_to_hub=False,
x1000c1s1b0n0: push_to_hub_model_id=None,
x1000c1s1b0n0: push_to_hub_organization=None,
x1000c1s1b0n0: push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
x1000c1s1b0n0: ray_scope=last,
x1000c1s1b0n0: remove_unused_columns=True,
x1000c1s1b0n0: report_to=['wandb'],
x1000c1s1b0n0: resume_from_checkpoint=None,
x1000c1s1b0n0: run_name=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug,
x1000c1s1b0n0: save_on_each_node=False,
x1000c1s1b0n0: save_safetensors=False,
x1000c1s1b0n0: save_steps=500,
x1000c1s1b0n0: save_strategy=steps,
x1000c1s1b0n0: save_total_limit=None,
x1000c1s1b0n0: seed=42,
x1000c1s1b0n0: sharded_ddp=[],
x1000c1s1b0n0: skip_memory_metrics=True,
x1000c1s1b0n0: tf32=None,
x1000c1s1b0n0: torch_compile=False,
x1000c1s1b0n0: torch_compile_backend=None,
x1000c1s1b0n0: torch_compile_mode=None,
x1000c1s1b0n0: torchdynamo=None,
x1000c1s1b0n0: tpu_metrics_debug=False,
x1000c1s1b0n0: tpu_num_cores=None,
x1000c1s1b0n0: use_ipex=False,
x1000c1s1b0n0: use_legacy_prediction_loop=False,
x1000c1s1b0n0: use_mps_device=False,
x1000c1s1b0n0: warmup_ratio=0.0,
x1000c1s1b0n0: warmup_steps=0,
x1000c1s1b0n0: weight_decay=0.0,
x1000c1s1b0n0: xpu_backend=None,
x1000c1s1b0n0: )
x1000c1s1b0n0: 07/16/2023 11:23:47 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
x1000c1s1b0n0: 07/16/2023 11:23:49 - INFO - datasets.info - Loading Dataset Infos from /home/users/industry/dso/lannliat/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
x1000c1s1b0n0: 07/16/2023 11:23:49 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists.
x1000c1s1b0n0: 07/16/2023 11:23:49 - INFO - datasets.info - Loading Dataset info from /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
x1000c1s1b0n0: 07/16/2023 11:23:49 - WARNING - datasets.builder - Found cached dataset wikitext (/home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
x1000c1s1b0n0: 07/16/2023 11:23:49 - INFO - datasets.info - Loading Dataset info from /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
100%|██████████| 3/3 [00:00<00:00, 272.09it/s]
x1000c0s6b0n0: 07/16/2023 11:23:50 - INFO - datasets.info - Loading Dataset Infos from /home/users/industry/dso/lannliat/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
x1000c0s6b0n0: 07/16/2023 11:23:50 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists.
x1000c0s6b0n0: 07/16/2023 11:23:50 - INFO - datasets.info - Loading Dataset info from /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
x1000c0s6b0n0: 07/16/2023 11:23:50 - WARNING - datasets.builder - Found cached dataset wikitext (/home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
100%|██████████| 3/3 [00:00<00:00, 194.61it/s]
x1000c0s6b0n0: 07/16/2023 11:23:50 - WARNING - datasets.builder - Found cached dataset wikitext (/home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
x1000c0s6b0n0: 07/16/2023 11:23:50 - INFO - datasets.info - Loading Dataset info from /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
100%|██████████| 3/3 [00:00<00:00, 811.12it/s]
x1000c1s1b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:50,641 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c1s1b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:50,641 >> Model config GPT2Config {
x1000c1s1b0n0:   "_name_or_path": "gpt2",
x1000c1s1b0n0:   "activation_function": "gelu_new",
x1000c1s1b0n0:   "architectures": [
x1000c1s1b0n0:     "GPT2LMHeadModel"
x1000c1s1b0n0:   ],
x1000c1s1b0n0:   "attn_pdrop": 0.1,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "embd_pdrop": 0.1,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "initializer_range": 0.02,
x1000c1s1b0n0:   "layer_norm_epsilon": 1e-05,
x1000c1s1b0n0:   "model_type": "gpt2",
x1000c1s1b0n0:   "n_ctx": 1024,
x1000c1s1b0n0:   "n_embd": 768,
x1000c1s1b0n0:   "n_head": 12,
x1000c1s1b0n0:   "n_inner": null,
x1000c1s1b0n0:   "n_layer": 12,
x1000c1s1b0n0:   "n_positions": 1024,
x1000c1s1b0n0:   "reorder_and_upcast_attn": false,
x1000c1s1b0n0:   "resid_pdrop": 0.1,
x1000c1s1b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c1s1b0n0:   "scale_attn_weights": true,
x1000c1s1b0n0:   "summary_activation": null,
x1000c1s1b0n0:   "summary_first_dropout": 0.1,
x1000c1s1b0n0:   "summary_proj_to_labels": true,
x1000c1s1b0n0:   "summary_type": "cls_index",
x1000c1s1b0n0:   "summary_use_proj": true,
x1000c1s1b0n0:   "task_specific_params": {
x1000c1s1b0n0:     "text-generation": {
x1000c1s1b0n0:       "do_sample": true,
x1000c1s1b0n0:       "max_length": 50
x1000c1s1b0n0:     }
x1000c1s1b0n0:   },
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0",
x1000c1s1b0n0:   "use_cache": true,
x1000c1s1b0n0:   "vocab_size": 50257
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c0s6b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:50,647 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c0s6b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:50,648 >> Model config GPT2Config {
x1000c0s6b0n0:   "_name_or_path": "gpt2",
x1000c0s6b0n0:   "activation_function": "gelu_new",
x1000c0s6b0n0:   "architectures": [
x1000c0s6b0n0:     "GPT2LMHeadModel"
x1000c0s6b0n0:   ],
x1000c0s6b0n0:   "attn_pdrop": 0.1,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "embd_pdrop": 0.1,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "initializer_range": 0.02,
x1000c0s6b0n0:   "layer_norm_epsilon": 1e-05,
x1000c0s6b0n0:   "model_type": "gpt2",
x1000c0s6b0n0:   "n_ctx": 1024,
x1000c0s6b0n0:   "n_embd": 768,
x1000c0s6b0n0:   "n_head": 12,
x1000c0s6b0n0:   "n_inner": null,
x1000c0s6b0n0:   "n_layer": 12,
x1000c0s6b0n0:   "n_positions": 1024,
x1000c0s6b0n0:   "reorder_and_upcast_attn": false,
x1000c0s6b0n0:   "resid_pdrop": 0.1,
x1000c0s6b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c0s6b0n0:   "scale_attn_weights": true,
x1000c0s6b0n0:   "summary_activation": null,
x1000c0s6b0n0:   "summary_first_dropout": 0.1,
x1000c0s6b0n0:   "summary_proj_to_labels": true,
x1000c0s6b0n0:   "summary_type": "cls_index",
x1000c0s6b0n0:   "summary_use_proj": true,
x1000c0s6b0n0:   "task_specific_params": {
x1000c0s6b0n0:     "text-generation": {
x1000c0s6b0n0:       "do_sample": true,
x1000c0s6b0n0:       "max_length": 50
x1000c0s6b0n0:     }
x1000c0s6b0n0:   },
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0",
x1000c0s6b0n0:   "use_cache": true,
x1000c0s6b0n0:   "vocab_size": 50257
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c1s1b0n0: 07/16/2023 11:23:50 - WARNING - datasets.builder - Found cached dataset wikitext (/home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
100%|██████████| 3/3 [00:00<00:00, 1171.27it/s]
x1000c0s6b0n0: [INFO|tokenization_auto.py:512] 2023-07-16 11:23:50,890 >> Could not locate the tokenizer configuration file, will try to use the model config instead.
x1000c0s6b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:51,134 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c0s6b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:51,135 >> Model config GPT2Config {
x1000c0s6b0n0:   "_name_or_path": "gpt2",
x1000c0s6b0n0:   "activation_function": "gelu_new",
x1000c0s6b0n0:   "architectures": [
x1000c0s6b0n0:     "GPT2LMHeadModel"
x1000c0s6b0n0:   ],
x1000c0s6b0n0:   "attn_pdrop": 0.1,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "embd_pdrop": 0.1,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "initializer_range": 0.02,
x1000c0s6b0n0:   "layer_norm_epsilon": 1e-05,
x1000c0s6b0n0:   "model_type": "gpt2",
x1000c0s6b0n0:   "n_ctx": 1024,
x1000c0s6b0n0:   "n_embd": 768,
x1000c0s6b0n0:   "n_head": 12,
x1000c0s6b0n0:   "n_inner": null,
x1000c0s6b0n0:   "n_layer": 12,
x1000c0s6b0n0:   "n_positions": 1024,
x1000c0s6b0n0:   "reorder_and_upcast_attn": false,
x1000c0s6b0n0:   "resid_pdrop": 0.1,
x1000c0s6b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c0s6b0n0:   "scale_attn_weights": true,
x1000c0s6b0n0:   "summary_activation": null,
x1000c0s6b0n0:   "summary_first_dropout": 0.1,
x1000c0s6b0n0:   "summary_proj_to_labels": true,
x1000c0s6b0n0:   "summary_type": "cls_index",
x1000c0s6b0n0:   "summary_use_proj": true,
x1000c0s6b0n0:   "task_specific_params": {
x1000c0s6b0n0:     "text-generation": {
x1000c0s6b0n0:       "do_sample": true,
x1000c0s6b0n0:       "max_length": 50
x1000c0s6b0n0:     }
x1000c0s6b0n0:   },
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0",
x1000c0s6b0n0:   "use_cache": true,
x1000c0s6b0n0:   "vocab_size": 50257
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c1s1b0n0: [INFO|tokenization_auto.py:512] 2023-07-16 11:23:51,338 >> Could not locate the tokenizer configuration file, will try to use the model config instead.
x1000c1s1b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:51,581 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c1s1b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:51,582 >> Model config GPT2Config {
x1000c1s1b0n0:   "_name_or_path": "gpt2",
x1000c1s1b0n0:   "activation_function": "gelu_new",
x1000c1s1b0n0:   "architectures": [
x1000c1s1b0n0:     "GPT2LMHeadModel"
x1000c1s1b0n0:   ],
x1000c1s1b0n0:   "attn_pdrop": 0.1,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "embd_pdrop": 0.1,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "initializer_range": 0.02,
x1000c1s1b0n0:   "layer_norm_epsilon": 1e-05,
x1000c1s1b0n0:   "model_type": "gpt2",
x1000c1s1b0n0:   "n_ctx": 1024,
x1000c1s1b0n0:   "n_embd": 768,
x1000c1s1b0n0:   "n_head": 12,
x1000c1s1b0n0:   "n_inner": null,
x1000c1s1b0n0:   "n_layer": 12,
x1000c1s1b0n0:   "n_positions": 1024,
x1000c1s1b0n0:   "reorder_and_upcast_attn": false,
x1000c1s1b0n0:   "resid_pdrop": 0.1,
x1000c1s1b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c1s1b0n0:   "scale_attn_weights": true,
x1000c1s1b0n0:   "summary_activation": null,
x1000c1s1b0n0:   "summary_first_dropout": 0.1,
x1000c1s1b0n0:   "summary_proj_to_labels": true,
x1000c1s1b0n0:   "summary_type": "cls_index",
x1000c1s1b0n0:   "summary_use_proj": true,
x1000c1s1b0n0:   "task_specific_params": {
x1000c1s1b0n0:     "text-generation": {
x1000c1s1b0n0:       "do_sample": true,
x1000c1s1b0n0:       "max_length": 50
x1000c1s1b0n0:     }
x1000c1s1b0n0:   },
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0",
x1000c1s1b0n0:   "use_cache": true,
x1000c1s1b0n0:   "vocab_size": 50257
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file vocab.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/vocab.json
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file merges.txt from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/merges.txt
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file tokenizer.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/tokenizer.json
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file added_tokens.json from cache at None
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file special_tokens_map.json from cache at None
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file tokenizer_config.json from cache at None
x1000c0s6b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:52,139 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c0s6b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:52,140 >> Model config GPT2Config {
x1000c0s6b0n0:   "_name_or_path": "gpt2",
x1000c0s6b0n0:   "activation_function": "gelu_new",
x1000c0s6b0n0:   "architectures": [
x1000c0s6b0n0:     "GPT2LMHeadModel"
x1000c0s6b0n0:   ],
x1000c0s6b0n0:   "attn_pdrop": 0.1,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "embd_pdrop": 0.1,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "initializer_range": 0.02,
x1000c0s6b0n0:   "layer_norm_epsilon": 1e-05,
x1000c0s6b0n0:   "model_type": "gpt2",
x1000c0s6b0n0:   "n_ctx": 1024,
x1000c0s6b0n0:   "n_embd": 768,
x1000c0s6b0n0:   "n_head": 12,
x1000c0s6b0n0:   "n_inner": null,
x1000c0s6b0n0:   "n_layer": 12,
x1000c0s6b0n0:   "n_positions": 1024,
x1000c0s6b0n0:   "reorder_and_upcast_attn": false,
x1000c0s6b0n0:   "resid_pdrop": 0.1,
x1000c0s6b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c0s6b0n0:   "scale_attn_weights": true,
x1000c0s6b0n0:   "summary_activation": null,
x1000c0s6b0n0:   "summary_first_dropout": 0.1,
x1000c0s6b0n0:   "summary_proj_to_labels": true,
x1000c0s6b0n0:   "summary_type": "cls_index",
x1000c0s6b0n0:   "summary_use_proj": true,
x1000c0s6b0n0:   "task_specific_params": {
x1000c0s6b0n0:     "text-generation": {
x1000c0s6b0n0:       "do_sample": true,
x1000c0s6b0n0:       "max_length": 50
x1000c0s6b0n0:     }
x1000c0s6b0n0:   },
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0",
x1000c0s6b0n0:   "use_cache": true,
x1000c0s6b0n0:   "vocab_size": 50257
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c0s6b0n0: [INFO|modeling_utils.py:2603] 2023-07-16 11:23:52,218 >> loading weights file model.safetensors from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/model.safetensors
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file vocab.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/vocab.json
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file merges.txt from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/merges.txt
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file tokenizer.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/tokenizer.json
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file added_tokens.json from cache at None
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file special_tokens_map.json from cache at None
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file tokenizer_config.json from cache at None
x1000c1s1b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:52,573 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c1s1b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:52,574 >> Model config GPT2Config {
x1000c1s1b0n0:   "_name_or_path": "gpt2",
x1000c1s1b0n0:   "activation_function": "gelu_new",
x1000c1s1b0n0:   "architectures": [
x1000c1s1b0n0:     "GPT2LMHeadModel"
x1000c1s1b0n0:   ],
x1000c1s1b0n0:   "attn_pdrop": 0.1,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "embd_pdrop": 0.1,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "initializer_range": 0.02,
x1000c1s1b0n0:   "layer_norm_epsilon": 1e-05,
x1000c1s1b0n0:   "model_type": "gpt2",
x1000c1s1b0n0:   "n_ctx": 1024,
x1000c1s1b0n0:   "n_embd": 768,
x1000c1s1b0n0:   "n_head": 12,
x1000c1s1b0n0:   "n_inner": null,
x1000c1s1b0n0:   "n_layer": 12,
x1000c1s1b0n0:   "n_positions": 1024,
x1000c1s1b0n0:   "reorder_and_upcast_attn": false,
x1000c1s1b0n0:   "resid_pdrop": 0.1,
x1000c1s1b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c1s1b0n0:   "scale_attn_weights": true,
x1000c1s1b0n0:   "summary_activation": null,
x1000c1s1b0n0:   "summary_first_dropout": 0.1,
x1000c1s1b0n0:   "summary_proj_to_labels": true,
x1000c1s1b0n0:   "summary_type": "cls_index",
x1000c1s1b0n0:   "summary_use_proj": true,
x1000c1s1b0n0:   "task_specific_params": {
x1000c1s1b0n0:     "text-generation": {
x1000c1s1b0n0:       "do_sample": true,
x1000c1s1b0n0:       "max_length": 50
x1000c1s1b0n0:     }
x1000c1s1b0n0:   },
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0",
x1000c1s1b0n0:   "use_cache": true,
x1000c1s1b0n0:   "vocab_size": 50257
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c1s1b0n0: [INFO|modeling_utils.py:2603] 2023-07-16 11:23:52,649 >> loading weights file model.safetensors from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/model.safetensors
x1000c1s1b0n0: [INFO|modeling_utils.py:2694] 2023-07-16 11:23:52,754 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model
x1000c0s6b0n0: [INFO|modeling_utils.py:2694] 2023-07-16 11:23:52,754 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model
x1000c1s1b0n0: [INFO|configuration_utils.py:599] 2023-07-16 11:23:52,757 >> Generate config GenerationConfig {
x1000c1s1b0n0:   "_from_model_config": true,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0"
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c0s6b0n0: [INFO|configuration_utils.py:599] 2023-07-16 11:23:52,759 >> Generate config GenerationConfig {
x1000c0s6b0n0:   "_from_model_config": true,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0"
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c0s6b0n0: [2023-07-16 11:23:58,743] [INFO] [partition_parameters.py:453:__exit__] finished initializing model with 0.16B parameters
x1000c1s1b0n0: [INFO|modeling_utils.py:3329] 2023-07-16 11:23:59,764 >> All model checkpoint weights were used when initializing GPT2LMHeadModel.
x1000c1s1b0n0: 
x1000c0s6b0n0: [INFO|modeling_utils.py:3329] 2023-07-16 11:23:59,764 >> All model checkpoint weights were used when initializing GPT2LMHeadModel.
x1000c1s1b0n0: [INFO|modeling_utils.py:3337] 2023-07-16 11:23:59,764 >> All the weights of GPT2LMHeadModel were initialized from the model checkpoint at gpt2.
x1000c0s6b0n0: 
x1000c1s1b0n0: If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.
x1000c0s6b0n0: [INFO|modeling_utils.py:3337] 2023-07-16 11:23:59,764 >> All the weights of GPT2LMHeadModel were initialized from the model checkpoint at gpt2.
x1000c0s6b0n0: If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.
x1000c0s6b0n0: [INFO|configuration_utils.py:561] 2023-07-16 11:24:00,009 >> loading configuration file generation_config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/generation_config.json
x1000c0s6b0n0: [INFO|configuration_utils.py:599] 2023-07-16 11:24:00,009 >> Generate config GenerationConfig {
x1000c0s6b0n0:   "_from_model_config": true,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0"
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c1s1b0n0: [INFO|configuration_utils.py:561] 2023-07-16 11:24:00,009 >> loading configuration file generation_config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/generation_config.json
x1000c1s1b0n0: [INFO|configuration_utils.py:599] 2023-07-16 11:24:00,010 >> Generate config GenerationConfig {
x1000c1s1b0n0:   "_from_model_config": true,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0"
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-44ec8a7bce9ef049.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-721cda7e77511ffc.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-44ec8a7bce9ef049.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-721cda7e77511ffc.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-49f826bee8b8a100.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-49f826bee8b8a100.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c87119ecc69384c8.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c87119ecc69384c8.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-44ec8a7bce9ef049.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-44ec8a7bce9ef049.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-721cda7e77511ffc.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-49f826bee8b8a100.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-721cda7e77511ffc.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-49f826bee8b8a100.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-acd38b65189dc44f.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-acd38b65189dc44f.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c1cf316f13c4acf5.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c1cf316f13c4acf5.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c87119ecc69384c8.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c87119ecc69384c8.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-acd38b65189dc44f.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-acd38b65189dc44f.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c1cf316f13c4acf5.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c1cf316f13c4acf5.arrow
x1000c0s6b0n0: [2023-07-16 11:24:01,734] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.5, git-hash=unknown, git-branch=unknown
x1000c0s6b0n0: [2023-07-16 11:24:02,277] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
x1000c0s6b0n0:  [WARNING]  cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
x1000c0s6b0n0: Using /home/users/industry/dso/lannliat/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
x1000c1s1b0n0:  [WARNING]  cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
x1000c1s1b0n0: Using /home/users/industry/dso/lannliat/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
x1000c1s1b0n0:  [WARNING]  cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
x1000c1s1b0n0: Using /home/users/industry/dso/lannliat/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
x1000c1s1b0n0: Traceback (most recent call last):
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 634, in <module>
x1000c1s1b0n0:     main()
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 582, in main
x1000c1s1b0n0:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1539, in train
x1000c1s1b0n0:     return inner_training_loop(
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1659, in _inner_training_loop
x1000c1s1b0n0:     model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1198, in prepare
x1000c1s1b0n0:     result = self._prepare_deepspeed(*args)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1537, in _prepare_deepspeed
x1000c1s1b0n0:     engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/__init__.py", line 165, in initialize
x1000c1s1b0n0:     engine = DeepSpeedEngine(args=args,
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 309, in __init__
x1000c1s1b0n0:     self._configure_optimizer(optimizer, model_parameters)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1173, in _configure_optimizer
x1000c1s1b0n0:     basic_optimizer = self._configure_basic_optimizer(model_parameters)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1229, in _configure_basic_optimizer
x1000c1s1b0n0:     optimizer = DeepSpeedCPUAdam(model_parameters,
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
x1000c1s1b0n0:     self.ds_opt_adam = CPUAdamBuilder().load()
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 454, in load
x1000c1s1b0n0:     return self.jit_load(verbose)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 497, in jit_load
x1000c1s1b0n0:     op_module = load(name=self.name,
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
x1000c1s1b0n0:     return _jit_compile(
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
x1000c1s1b0n0:     _write_ninja_file_and_build_library(
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1592, in _write_ninja_file_and_build_library
x1000c1s1b0n0:     verify_ninja_availability()
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1648, in verify_ninja_availability
x1000c1s1b0n0:     raise RuntimeError("Ninja is required to load C++ extensions")
x1000c1s1b0n0: RuntimeError: Ninja is required to load C++ extensions
x1000c0s6b0n0: Traceback (most recent call last):
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 634, in <module>
x1000c0s6b0n0:     main()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 582, in main
x1000c0s6b0n0:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1539, in train
x1000c0s6b0n0:     return inner_training_loop(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1659, in _inner_training_loop
x1000c0s6b0n0:     model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1198, in prepare
x1000c0s6b0n0:     result = self._prepare_deepspeed(*args)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1537, in _prepare_deepspeed
x1000c0s6b0n0:     engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/__init__.py", line 165, in initialize
x1000c0s6b0n0:     engine = DeepSpeedEngine(args=args,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 309, in __init__
x1000c0s6b0n0:     self._configure_optimizer(optimizer, model_parameters)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1173, in _configure_optimizer
x1000c0s6b0n0:     basic_optimizer = self._configure_basic_optimizer(model_parameters)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1229, in _configure_basic_optimizer
x1000c0s6b0n0:     optimizer = DeepSpeedCPUAdam(model_parameters,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
x1000c0s6b0n0:     self.ds_opt_adam = CPUAdamBuilder().load()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 454, in load
x1000c0s6b0n0:     return self.jit_load(verbose)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 497, in jit_load
x1000c0s6b0n0:     op_module = load(name=self.name,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
x1000c0s6b0n0:     return _jit_compile(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
x1000c0s6b0n0:     _write_ninja_file_and_build_library(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1592, in _write_ninja_file_and_build_library
x1000c0s6b0n0:     verify_ninja_availability()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1648, in verify_ninja_availability
x1000c0s6b0n0:     raise RuntimeError("Ninja is required to load C++ extensions")
x1000c0s6b0n0: RuntimeError: Ninja is required to load C++ extensions
x1000c1s1b0n0: Loading extension module cpu_adam...
x1000c1s1b0n0: Time to load cpu_adam op: 0.502467155456543 seconds
x1000c1s1b0n0: Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f42085dca60>
x1000c1s1b0n0: Traceback (most recent call last):
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
x1000c1s1b0n0:     self.ds_opt_adam.destroy_adam(self.opt_id)
x1000c1s1b0n0: AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
x1000c0s6b0n0:  [WARNING]  cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
x1000c0s6b0n0: Using /home/users/industry/dso/lannliat/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
x1000c0s6b0n0: Traceback (most recent call last):
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 634, in <module>
x1000c0s6b0n0:     main()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 582, in main
x1000c0s6b0n0:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1539, in train
x1000c0s6b0n0:     return inner_training_loop(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1659, in _inner_training_loop
x1000c0s6b0n0:     model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1198, in prepare
x1000c0s6b0n0:     result = self._prepare_deepspeed(*args)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1537, in _prepare_deepspeed
x1000c0s6b0n0:     engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/__init__.py", line 165, in initialize
x1000c0s6b0n0:     engine = DeepSpeedEngine(args=args,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 309, in __init__
x1000c0s6b0n0:     self._configure_optimizer(optimizer, model_parameters)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1173, in _configure_optimizer
x1000c0s6b0n0:     basic_optimizer = self._configure_basic_optimizer(model_parameters)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1229, in _configure_basic_optimizer
x1000c0s6b0n0:     optimizer = DeepSpeedCPUAdam(model_parameters,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
x1000c0s6b0n0:     self.ds_opt_adam = CPUAdamBuilder().load()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 454, in load
x1000c0s6b0n0:     return self.jit_load(verbose)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 497, in jit_load
x1000c0s6b0n0:     op_module = load(name=self.name,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
x1000c0s6b0n0:     return _jit_compile(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
x1000c0s6b0n0:     _write_ninja_file_and_build_library(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1592, in _write_ninja_file_and_build_library
x1000c0s6b0n0:     verify_ninja_availability()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1648, in verify_ninja_availability
x1000c0s6b0n0:     raise RuntimeError("Ninja is required to load C++ extensions")
x1000c0s6b0n0: RuntimeError: Ninja is required to load C++ extensions
x1000c0s6b0n0: Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7eff0daaca60>
x1000c0s6b0n0: Traceback (most recent call last):
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
x1000c0s6b0n0:     self.ds_opt_adam.destroy_adam(self.opt_id)
x1000c0s6b0n0: AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
x1000c0s6b0n0: Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f0145cb8a60>
x1000c0s6b0n0: Traceback (most recent call last):
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
x1000c0s6b0n0:     self.ds_opt_adam.destroy_adam(self.opt_id)
x1000c0s6b0n0: AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
x1000c1s1b0n0: [2023-07-16 11:24:04,470] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2431494
x1000c0s6b0n0: [2023-07-16 11:24:04,784] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 665498
x1000c0s6b0n0: [2023-07-16 11:24:04,804] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 665499
```

When I tried single node training, the training procedes smoothly:
```
deepspeed --num_gpus=2 $ROOT_DIR/transformers/examples/pytorch/language-modeling/run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir ~/scratch/finetuned_models/debug \
    --deepspeed "$ROOT_DIR/FastChat/fastchat/ds_config/ds_config_zero3.json"
```

### Expected behavior

Expected training to proceed smoothly as in single node case.

Comments:
For the error log

```bash
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1229, in _configure_basic_optimizer
x1000c0s6b0n0:     optimizer = DeepSpeedCPUAdam(model_parameters,
```

I believe it's best to open an issue on [DeepSpeed GitHub issues](https://github.com/microsoft/DeepSpeed/issues) page for this.

It's likely a deepspeed version issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.