Repository: huggingface/accelerate

Issue:
`main_process_ip` not working
### System Info

```Shell
- `Accelerate` version: 0.25.0
- Platform: Linux-5.15.0-86-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Numpy version: 1.26.4
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 110.32 GB
- GPU type: NVIDIA L4
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: DEEPSPEED
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 2
        - main_process_ip: 192.168.10.161
        - main_process_port: 1040
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {'deepspeed_hostfile': '/root/hostfile', 'deepspeed_multinode_launcher': 'pdsh', 'gradient_accumulation_steps': 1, 'gradient_clipping': 1.0, 'offload_optimizer_device': 'none', 'offload_param_device': 'cpu', 'zero3_init_flag': True, 'zero3_save_16bit_model': True, 'zero_stage': 3}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

Hi, I'm trying multi-node training with accelerate+deepspeed.
During the broadcast before the start of the training, an error occurred regarding the network interface.
I set 192.168.10.161 in the main_process_ip of the accelerate config, but runner.py targets 192.168.121.122. It is master node's first network interface and that can't communicate between nodes.
Is there a way to set the master_addr used by runner.py in my environment?
I would greatly appreciate your assistance.

![issue](https://github.com/huggingface/accelerate/assets/39879672/409237d6-bfba-4f5d-95b2-5628d7589244)


### Expected behavior

runner.py targets 192.168.10.161


Discussion:
Hi @asdfry, could you try to debug and check that you have the right args (`master_addr`) when using deepspeed ? Try to print `args` and `current_env` in this [function](https://github.com/huggingface/accelerate/blob/97d2168e5953fe7373a06c69c02c5a00a84d5344/src/accelerate/commands/launch.py#L724). cc @pacman100 
As you suggested, I added a print to the launch.py code of the master node and checked the logs.
Is the `main_process_ip` set in the accelerate config unrelated to the `master_addr` used in deepspeed?
Then how to set `master_addr` when using accelerate+deepspeed?

![deepspeed_launcher](https://github.com/huggingface/accelerate/assets/39879672/0a7687ee-b267-4812-beb8-962b9ad600db)

```
root@vnode1:~# accelerate launch train.py -b 1 -m Llama-2-7b-chat-hf -o sgd -t
/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
=== CURRENT_ENV: {'NV_LIBCUBLAS_VERSION': '12.3.4.1-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'NV_NVML_DEV_VERSION': '12.3.101-1', 'NV_LIBNCCL_DEV_PACKAGE': 'libnccl-dev=2.19.3-1+cuda12.3', 'TRANSFORMERS_OFFLINE': '1', 'NV_LIBNCCL_DEV_PACKAGE_VERSION': '2.19.3-1', 'HOSTNAME': 'vnode1.pnode9.idc1.ten1010.io', 'HF_DATASETS_OFFLINE': '1', 'NVIDIA_REQUIRE_CUDA': 'cuda>=12.3 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536', 'NV_LIBCUBLAS_DEV_PACKAGE': 'libcublas-dev-12-3=12.3.4.1-1', 'NV_NVTX_VERSION': '12.3.101-1', 'nsys_image': 'NsightSystems-linux-cli-public-2023.4.1.97-3355750', 'NV_CUDA_CUDART_DEV_VERSION': '12.3.101-1', 'NV_LIBCUSPARSE_VERSION': '12.2.0.103-1', 'NV_LIBNPP_VERSION': '12.2.3.2-1', 'TORCH_CUDA_ARCH_LIST': '9.0', 'NCCL_VERSION': '2.19.3-1', 'PWD': '/root', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NV_NVPROF_DEV_PACKAGE': 'cuda-nvprof-12-3=12.3.101-1', 'NV_LIBNPP_PACKAGE': 'libnpp-12-3=12.2.3.2-1', 'NCCL_DEBUG': 'INFO', 'NV_LIBNCCL_DEV_PACKAGE_NAME': 'libnccl-dev', 'HPCX_HOME': '/hpcx-v2.17.1-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64', 'DS_BUILD_CPU_ADAM': '1', 'NV_LIBCUBLAS_DEV_VERSION': '12.3.4.1-1', 'NVIDIA_PRODUCT_NAME': 'CUDA', 'NV_LIBCUBLAS_DEV_PACKAGE_NAME': 'libcublas-dev-12-3', 'NV_CUDA_CUDART_VERSION': '12.3.101-1', 'HOME': '/root', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'mlnx_image': 'MLNX_OFED_LINUX-23.10-1.1.9.0-ubuntu22.04-x86_64', 'CUDA_VERSION': '12.3.1', 'NV_LIBCUBLAS_PACKAGE': 'libcublas-12-3=12.3.4.1-1', 'NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE': 'cuda-nsight-compute-12-3=12.3.1-1', 'NV_LIBNPP_DEV_PACKAGE': 'libnpp-dev-12-3=12.2.3.2-1', 'hpcx_image': 'hpcx-v2.17.1-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64', 'NV_LIBCUBLAS_PACKAGE_NAME': 'libcublas-12-3', 'NCCL_DEBUG_FILE': 'mnt/output/nccl-debug.log', 'NV_LIBNPP_DEV_VERSION': '12.2.3.2-1', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'TERM': 'xterm', 'NV_LIBCUSPARSE_DEV_VERSION': '12.2.0.103-1', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'LIBRARY_PATH': '/usr/local/cuda/lib64/stubs', 'NCCL_TOPO_DUMP_FILE': 'mnt/output/nccl-topo.xml', 'SHLVL': '1', 'NV_CUDA_LIB_VERSION': '12.3.1-1', 'NVARCH': 'x86_64', 'NV_CUDA_COMPAT_PACKAGE': 'cuda-compat-12-3', 'NV_LIBNCCL_PACKAGE': 'libnccl2=2.19.3-1+cuda12.3', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64', 'NV_CUDA_NSIGHT_COMPUTE_VERSION': '12.3.1-1', 'NV_NVPROF_VERSION': '12.3.101-1', 'PATH': '/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'NV_LIBNCCL_PACKAGE_NAME': 'libnccl2', 'NV_LIBNCCL_PACKAGE_VERSION': '2.19.3-1', 'DEBIAN_FRONTEND': 'noninteractive', '_': '/usr/local/bin/accelerate', 'LC_CTYPE': 'C.UTF-8', 'PYTHONPATH': '/root', 'ACCELERATE_MIXED_PRECISION': 'bf16', 'ACCELERATE_CONFIG_DS_FIELDS': 'deepspeed_hostfile,deepspeed_multinode_launcher,gradient_accumulation_steps,gradient_clipping,offload_optimizer_device,offload_param_device,zero3_init_flag,zero3_save_16bit_model,zero_stage,mixed_precision', 'ACCELERATE_USE_DEEPSPEED': 'true', 'ACCELERATE_DEEPSPEED_ZERO_STAGE': '3', 'ACCELERATE_GRADIENT_ACCUMULATION_STEPS': '1', 'ACCELERATE_GRADIENT_CLIPPING': '1.0', 'ACCELERATE_DEEPSPEED_OFFLOAD_OPTIMIZER_DEVICE': 'none', 'ACCELERATE_DEEPSPEED_OFFLOAD_PARAM_DEVICE': 'cpu', 'ACCELERATE_DEEPSPEED_ZERO3_INIT': 'true', 'ACCELERATE_DEEPSPEED_ZERO3_SAVE_16BIT_MODEL': 'true'} ===
=== ARGS: Namespace(config_file=None, quiet=False, cpu=False, multi_gpu=False, tpu=False, ipex=False, mixed_precision='bf16', num_processes=8, num_machines=2, num_cpu_threads_per_process=1, dynamo_backend='no', dynamo_mode='default', dynamo_use_fullgraph=False, dynamo_use_dynamic=False, use_deepspeed=True, use_fsdp=False, use_megatron_lm=False, use_xpu=False, gpu_ids='all', same_network=False, machine_rank=0, main_process_ip='192.168.10.161', main_process_port=1040, tee='0', role='default', rdzv_backend='static', rdzv_conf='', max_restarts=0, monitor_interval=5, module=False, no_python=False, tpu_use_cluster=False, tpu_use_sudo=False, vm=None, env=None, main_training_function='main', downcast_bf16=False, deepspeed_config_file=None, zero_stage=3, offload_optimizer_device='none', offload_param_device='cpu', offload_optimizer_nvme_path=None, offload_param_nvme_path=None, gradient_accumulation_steps=1, gradient_clipping=1.0, zero3_init_flag=True, zero3_save_16bit_model=True, deepspeed_hostfile='/root/hostfile', deepspeed_exclusion_filter=None, deepspeed_inclusion_filter=None, deepspeed_multinode_launcher='pdsh', fsdp_offload_params='false', fsdp_min_num_params=100000000.0, fsdp_sharding_strategy=1, fsdp_auto_wrap_policy=None, fsdp_transformer_layer_cls_to_wrap=None, fsdp_backward_prefetch_policy=None, fsdp_state_dict_type=None, fsdp_forward_prefetch='false', fsdp_use_orig_params='true', fsdp_cpu_ram_efficient_loading='true', fsdp_sync_module_states='true', megatron_lm_tp_degree=1, megatron_lm_pp_degree=1, megatron_lm_num_micro_batches=None, megatron_lm_sequence_parallelism=None, megatron_lm_recompute_activations=None, megatron_lm_use_distributed_optimizer=None, megatron_lm_gradient_clipping=1.0, aws_access_key_id=None, aws_secret_access_key=None, debug=False, training_script='train.py', training_script_args=['-b', '1', '-m', 'Llama-2-7b-chat-hf', '-o', 'sgd', '-t'], func=<function launch_command at 0x7f6312c53ac0>, use_cpu=False, tpu_name=None, tpu_zone=None, command_file=None, commands=None, tpu_vm=None, tpu_env=[], deepspeed_fields_from_accelerate_config='deepspeed_hostfile,deepspeed_multinode_launcher,gradient_accumulation_steps,gradient_clipping,offload_optimizer_device,offload_param_device,zero3_init_flag,zero3_save_16bit_model,zero_stage,mixed_precision') ===
=== IF ===
[2024-02-24 05:30:32,028] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
[2024-02-24 05:30:33,264] [INFO] [runner.py:463:main] Using IP address of 192.168.121.122 for node trainer-1
[2024-02-24 05:30:33,264] [INFO] [runner.py:559:main] deepspeed_env file = .deepspeed_env
[2024-02-24 05:30:33,264] [INFO] [runner.py:559:main] deepspeed_env file = .deepspeed_env
[2024-02-24 05:30:33,264] [INFO] [multinode_runner.py:72:get_cmd] Running on the following workers: trainer-1,trainer-2
[2024-02-24 05:30:33,265] [INFO] [runner.py:570:main] cmd = pdsh -S -f 1024 -w trainer-1,trainer-2 export NCCL_VERSION=2.19.3-1; export NCCL_DEBUG=INFO; export NCCL_DEBUG_FILE=mnt/output/nccl-debug.log; export NCCL_TOPO_DUMP_FILE=mnt/output/nccl-topo.xml; export PYTHONPATH=/root; export NV_LIBCUBLAS_VERSION=12.3.4.1-1; export NVIDIA_VISIBLE_DEVICES=all; export NV_NVML_DEV_VERSION=12.3.101-1; export NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.19.3-1+cuda12.3; export TRANSFORMERS_OFFLINE=1; export NV_LIBNCCL_DEV_PACKAGE_VERSION=2.19.3-1; export HOSTNAME=vnode1.pnode9.idc1.ten1010.io; export HF_DATASETS_OFFLINE=1; export NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-3=12.3.4.1-1; export NV_NVTX_VERSION=12.3.101-1; export nsys_image=NsightSystems-linux-cli-public-2023.4.1.97-3355750; export NV_CUDA_CUDART_DEV_VERSION=12.3.101-1; export NV_LIBCUSPARSE_VERSION=12.2.0.103-1; export NV_LIBNPP_VERSION=12.2.3.2-1; export TORCH_CUDA_ARCH_LIST=9.0; export PWD=/root; export NVIDIA_DRIVER_CAPABILITIES=compute,utility; export NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-3=12.3.101-1; export NV_LIBNPP_PACKAGE=libnpp-12-3=12.2.3.2-1; export NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev; export HPCX_HOME=/hpcx-v2.17.1-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64; export DS_BUILD_CPU_ADAM=1; export NV_LIBCUBLAS_DEV_VERSION=12.3.4.1-1; export NVIDIA_PRODUCT_NAME=CUDA; export NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-3; export NV_CUDA_CUDART_VERSION=12.3.101-1; export HOME=/root; export mlnx_image=MLNX_OFED_LINUX-23.10-1.1.9.0-ubuntu22.04-x86_64; export CUDA_VERSION=12.3.1; export NV_LIBCUBLAS_PACKAGE=libcublas-12-3=12.3.4.1-1; export NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-3=12.3.1-1; export NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-3=12.2.3.2-1; export hpcx_image=hpcx-v2.17.1-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64; export NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-3; export NV_LIBNPP_DEV_VERSION=12.2.3.2-1; export TERM=xterm; export NV_LIBCUSPARSE_DEV_VERSION=12.2.0.103-1; export LIBRARY_PATH=/usr/local/cuda/lib64/stubs; export SHLVL=1; export NV_CUDA_LIB_VERSION=12.3.1-1; export NVARCH=x86_64; export NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-3; export NV_LIBNCCL_PACKAGE=libnccl2=2.19.3-1+cuda12.3; export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64; export NV_CUDA_NSIGHT_COMPUTE_VERSION=12.3.1-1; export NV_NVPROF_VERSION=12.3.101-1; export PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin; export NV_LIBNCCL_PACKAGE_NAME=libnccl2; export NV_LIBNCCL_PACKAGE_VERSION=2.19.3-1; export DEBIAN_FRONTEND=noninteractive; export _=/usr/local/bin/accelerate; export LC_CTYPE=C.UTF-8; export ACCELERATE_MIXED_PRECISION=bf16; export ACCELERATE_CONFIG_DS_FIELDS=deepspeed_hostfile,deepspeed_multinode_launcher,gradient_accumulation_steps,gradient_clipping,offload_optimizer_device,offload_param_device,zero3_init_flag,zero3_save_16bit_model,zero_stage,mixed_precision; export ACCELERATE_USE_DEEPSPEED=true; export ACCELERATE_DEEPSPEED_ZERO_STAGE=3; export ACCELERATE_GRADIENT_ACCUMULATION_STEPS=1; export ACCELERATE_GRADIENT_CLIPPING=1.0; export ACCELERATE_DEEPSPEED_OFFLOAD_OPTIMIZER_DEVICE=none; export ACCELERATE_DEEPSPEED_OFFLOAD_PARAM_DEVICE=cpu; export ACCELERATE_DEEPSPEED_ZERO3_INIT=true; export ACCELERATE_DEEPSPEED_ZERO3_SAVE_16BIT_MODEL=true;  cd /root; /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJ0cmFpbmVyLTEiOiBbMCwgMSwgMiwgM10sICJ0cmFpbmVyLTIiOiBbMCwgMSwgMiwgM119 --node_rank=%n --master_addr=192.168.121.122 --master_port=1040 --no_local_rank train.py -b '1' -m 'Llama-2-7b-chat-hf' -o 'sgd' -t
trainer-1: [2024-02-24 05:30:34,392] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.19.3-1+cuda12.3
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_DEV_PACKAGE_VERSION=2.19.3-1
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NCCL_VERSION=2.19.3-1
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NCCL_DEBUG=INFO
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NCCL_DEBUG_FILE=mnt/output/nccl-debug.log
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NCCL_TOPO_DUMP_FILE=mnt/output/nccl-topo.xml
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_PACKAGE=libnccl2=2.19.3-1+cuda12.3
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_PACKAGE_NAME=libnccl2
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:138:main] 0 NV_LIBNCCL_PACKAGE_VERSION=2.19.3-1
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:145:main] WORLD INFO DICT: {'trainer-1': [0, 1, 2, 3], 'trainer-2': [0, 1, 2, 3]}
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:151:main] nnodes=2, num_local_procs=4, node_rank=0
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'trainer-1': [0, 1, 2, 3], 'trainer-2': [4, 5, 6, 7]})
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:163:main] dist_world_size=8
trainer-1: [2024-02-24 05:30:35,350] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
trainer-2: [2024-02-24 05:30:36,631] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: [2024-02-24 05:30:37,485] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-1: [2024-02-24 05:30:37,506] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-1: [2024-02-24 05:30:37,548] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-1: [2024-02-24 05:30:37,551] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.19.3-1+cuda12.3
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NV_LIBNCCL_DEV_PACKAGE_VERSION=2.19.3-1
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NCCL_VERSION=2.19.3-1
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NCCL_DEBUG=INFO
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NCCL_DEBUG_FILE=mnt/output/nccl-debug.log
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NCCL_TOPO_DUMP_FILE=mnt/output/nccl-topo.xml
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NV_LIBNCCL_PACKAGE=libnccl2=2.19.3-1+cuda12.3
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NV_LIBNCCL_PACKAGE_NAME=libnccl2
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:138:main] 1 NV_LIBNCCL_PACKAGE_VERSION=2.19.3-1
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:145:main] WORLD INFO DICT: {'trainer-1': [0, 1, 2, 3], 'trainer-2': [0, 1, 2, 3]}
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:151:main] nnodes=2, num_local_procs=4, node_rank=1
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'trainer-1': [0, 1, 2, 3], 'trainer-2': [4, 5, 6, 7]})
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:163:main] dist_world_size=8
trainer-2: [2024-02-24 05:30:37,588] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: [2024-02-24 05:30:37,693] [INFO] [comm.py:637:init_distributed] cdb=None
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: [2024-02-24 05:30:37,725] [INFO] [comm.py:637:init_distributed] cdb=None
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-1:   _torch_pytree._register_pytree_node(
trainer-1: [2024-02-24 05:30:37,797] [INFO] [comm.py:637:init_distributed] cdb=None
trainer-1: [2024-02-24 05:30:37,797] [INFO] [comm.py:637:init_distributed] cdb=None
trainer-1: [2024-02-24 05:30:37,797] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: [2024-02-24 05:30:39,718] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-2: [2024-02-24 05:30:39,718] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-2: [2024-02-24 05:30:39,785] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-2: [2024-02-24 05:30:39,785] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: [2024-02-24 05:30:39,886] [INFO] [comm.py:637:init_distributed] cdb=None
trainer-2: [2024-02-24 05:30:39,887] [INFO] [comm.py:637:init_distributed] cdb=None
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
trainer-2:   _torch_pytree._register_pytree_node(
trainer-2: [2024-02-24 05:30:39,952] [INFO] [comm.py:637:init_distributed] cdb=None
trainer-2: [2024-02-24 05:30:39,952] [INFO] [comm.py:637:init_distributed] cdb=None
trainer-2: [W socket.cpp:697] [c10d] The client socket has failed to connect to [::ffff:192.168.121.122]:1040 (errno: 113 - No route to host).
trainer-2: [W socket.cpp:697] [c10d] The client socket has failed to connect to [::ffff:192.168.121.122]:1040 (errno: 113 - No route to host).
trainer-2: [W socket.cpp:697] [c10d] The client socket has failed to connect to [::ffff:192.168.121.122]:1040 (errno: 113 - No route to host).
trainer-2: [W socket.cpp:697] [c10d] The client socket has failed to connect to [::ffff:192.168.121.122]:1040 (errno: 113 - No route to host).
trainer-2: [W socket.cpp:697] [c10d] The client socket has failed to connect to 192.168.121.122:1040 (errno: 113 - No route to host).
trainer-2: [E socket.cpp:753] [c10d] The client socket has failed to connect to any network address of (192.168.121.122, 1040).
trainer-2: [W socket.cpp:697] [c10d] The client socket has failed to connect to 192.168.121.122:1040 (errno: 113 - No route to host).
trainer-2: [E socket.cpp:753] [c10d] The client socket has failed to connect to any network address of (192.168.121.122, 1040).
trainer-2: [W socket.cpp:697] [c10d] The client socket has failed to connect to 192.168.121.122:1040 (errno: 113 - No route to host).
trainer-2: [E socket.cpp:753] [c10d] The client socket has failed to connect to any network address of (192.168.121.122, 1040).
trainer-2: [W socket.cpp:697] [c10d] The client socket has failed to connect to 192.168.121.122:1040 (errno: 113 - No route to host).
trainer-2: [E socket.cpp:753] [c10d] The client socket has failed to connect to any network address of (192.168.121.122, 1040).
trainer-2: Traceback (most recent call last):
trainer-2: Traceback (most recent call last):
trainer-2:   File "/root/train.py", line 44, in <module>
trainer-2:   File "/root/train.py", line 44, in <module>
trainer-2:         accelerator = Accelerator()accelerator = Accelerator()
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 371, in __init__
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 371, in __init__
trainer-2:         self.state = AcceleratorState(self.state = AcceleratorState(
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/state.py", line 758, in __init__
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/state.py", line 758, in __init__
trainer-2: Traceback (most recent call last):
trainer-2:   File "/root/train.py", line 44, in <module>
trainer-2: Traceback (most recent call last):
trainer-2:   File "/root/train.py", line 44, in <module>
trainer-2:             PartialState(cpu, **kwargs)accelerator = Accelerator()PartialState(cpu, **kwargs)
trainer-2: 
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/state.py", line 183, in __init__
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/state.py", line 183, in __init__
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 371, in __init__
trainer-2:     accelerator = Accelerator()
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 371, in __init__
trainer-2:     dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)    
trainer-2: dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)  File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/comm.py", line 670, in init_distributed
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/comm.py", line 670, in init_distributed
trainer-2:     self.state = AcceleratorState(
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/state.py", line 758, in __init__
trainer-2:     self.state = AcceleratorState(
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/state.py", line 758, in __init__
trainer-2:         cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py", line 120, in __init__
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py", line 120, in __init__
trainer-2:     PartialState(cpu, **kwargs)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/state.py", line 183, in __init__
trainer-2:     self.init_process_group(backend, timeout, init_method, rank, world_size)    
trainer-2: self.init_process_group(backend, timeout, init_method, rank, world_size)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py", line 146, in init_process_group
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py", line 146, in init_process_group
trainer-2:     PartialState(cpu, **kwargs)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/accelerate/state.py", line 183, in __init__
trainer-2:     torch.distributed.init_process_group(backend,    
trainer-2:     dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 86, in wrapper
trainer-2: torch.distributed.init_process_group(backend,
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/comm.py", line 670, in init_distributed
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 86, in wrapper
trainer-2:     dist.init_distributed(dist_backend=self.backend, auto_mpi_discovery=False, **kwargs)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/comm.py", line 670, in init_distributed
trainer-2:     func_return = func(*args, **kwargs)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1177, in init_process_group
trainer-2:     func_return = func(*args, **kwargs)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1177, in init_process_group
trainer-2:         cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py", line 120, in __init__
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py", line 120, in __init__
trainer-2:         store, rank, world_size = next(rendezvous_iterator)store, rank, world_size = next(rendezvous_iterator)
trainer-2: 
trainer-2:       File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 246, in _env_rendezvous_handler
trainer-2:       File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 246, in _env_rendezvous_handler
trainer-2: self.init_process_group(backend, timeout, init_method, rank, world_size)self.init_process_group(backend, timeout, init_method, rank, world_size)
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py", line 146, in init_process_group
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py", line 146, in init_process_group
trainer-2:         torch.distributed.init_process_group(backend,torch.distributed.init_process_group(backend,
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 86, in wrapper
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 86, in wrapper
trainer-2:         store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout, use_libuv)store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout, use_libuv)
trainer-2: 
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 174, in _create_c10d_store
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 174, in _create_c10d_store
trainer-2:     func_return = func(*args, **kwargs)    
trainer-2: func_return = func(*args, **kwargs)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1177, in init_process_group
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1177, in init_process_group
trainer-2:     return TCPStore(    
trainer-2: return TCPStore(
trainer-2: torch.distributedtorch.distributed..DistNetworkErrorDistNetworkError: The client socket has failed to connect to any network address of (192.168.121.122, 1040). The client socket has failed to connect to 192.168.121.122:1040 (errno: 113 - No route to host).: 
trainer-2: The client socket has failed to connect to any network address of (192.168.121.122, 1040). The client socket has failed to connect to 192.168.121.122:1040 (errno: 113 - No route to host).
trainer-2:         store, rank, world_size = next(rendezvous_iterator)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 246, in _env_rendezvous_handler
trainer-2: store, rank, world_size = next(rendezvous_iterator)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 246, in _env_rendezvous_handler
trainer-2:     store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout, use_libuv)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 174, in _create_c10d_store
trainer-2:     store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout, use_libuv)
trainer-2:   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/rendezvous.py", line 174, in _create_c10d_store
trainer-2:     return TCPStore(
trainer-2: torch.distributed.DistNetworkError: The client socket has failed to connect to any network address of (192.168.121.122, 1040). The client socket has failed to connect to 192.168.121.122:1040 (errno: 113 - No route to host).    
trainer-2: return TCPStore(
trainer-2: torch.distributed.DistNetworkError: The client socket has failed to connect to any network address of (192.168.121.122, 1040). The client socket has failed to connect to 192.168.121.122:1040 (errno: 113 - No route to host).
trainer-2: [2024-02-24 05:30:46,599] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 19616
trainer-2: [2024-02-24 05:30:46,601] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 19617
trainer-2: [2024-02-24 05:30:46,602] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 19618
trainer-2: [2024-02-24 05:30:46,616] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 19619
trainer-2: [2024-02-24 05:30:46,643] [ERROR] [launch.py:321:sigkill_handler] ['/usr/bin/python3', '-u', 'train.py', '-b', '1', '-m', 'Llama-2-7b-chat-hf', '-o', 'sgd', '-t'] exits with return code = 1
pdsh@vnode1: trainer-2: ssh exited with exit code 1
```
Hi @asdfry, from the traceback, it looks like `master_addr` is not being defined in args. This is probably due to `same_network = False` (see its value in ARGS ) and the following code [here](https://github.com/huggingface/accelerate/blob/4aa71049c3dce0331caab428b3d9879912c09082/src/accelerate/utils/launch.py#L274). As you can see in the code, we use `main_process_ip` to set `master_addr` that is used in deepspeed. However, I see that in your config,  that `same_network` is set to `True`.  Could you try to debug what changed its value ? 

Resolving patch:
diff --git a/src/accelerate/utils/launch.py b/src/accelerate/utils/launch.py
--- a/src/accelerate/utils/launch.py
+++ b/src/accelerate/utils/launch.py
@@ -258,6 +258,8 @@ def prepare_deepspeed_cmd_env(args: argparse.Namespace) -> Tuple[List[str], Dict
             )
         else:
             cmd.extend(["--num_gpus", str(args.num_processes // args.num_machines)])
+        if main_process_ip:
+            cmd.extend(["--master_addr", str(main_process_ip)])
         cmd.extend(["--master_port", str(main_process_port)])
         if args.module and args.no_python:
             raise ValueError("--module and --no_python cannot be used together")


Test patch: