Torchrun vllm error：TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host

yishuizhao: <p>i have two server with GPU，i want to use tourchrun a vllm on two gpu server。<br>
torchrun --nnodes=2 --nproc_per_node=8 --node_rank=0 --rdzv-backend=c10d --rdzv-endpoint=192.168.0.13:29400 start_vllm.py --config /root/config.json</p>
<p>stop here，it’s like hung<br>
(VllmWorkerProcess pid=1968773) INFO 03-02 19:51:52 cuda.py:160] Using Triton MLA backend.<br>
(VllmWorkerProcess pid=1968801) INFO 03-02 19:51:52 cuda.py:160] Using Triton MLA backend.<br>
(VllmWorkerProcess pid=1968800) INFO 03-02 19:51:52 cuda.py:160] Using Triton MLA backend.<br>
(VllmWorkerProcess pid=1968791) (VllmWorkerProcess pid=1968799) INFO 03-02 19:51:52 cuda.py:160] Using Triton MLA backend.<br>
INFO 03-02 19:51:52 cuda.py:160] Using Triton MLA backend.<br>
(VllmWorkerProcess pid=1968783) (VllmWorkerProcess pid=1968790) WARNING 03-02 19:51:52 triton_decode_attention.py:44] The following error message ‘operation scheduled before its operands’ can be ignored.<br>
WARNING 03-02 19:51:52 triton_decode_attention.py:44] The following error message ‘operation scheduled before its operands’ can be ignored.<br>
WARNING 03-02 19:51:52 triton_decode_attention.py:44] The following error message ‘operation scheduled before its operands’ can be ignored.<br>
(VllmWorkerProcess pid=1968773) WARNING 03-02 19:51:52 triton_decode_attention.py:44] The following error message ‘operation scheduled before its operands’ can be ignored.<br>
(VllmWorkerProcess pid=1968801) WARNING 03-02 19:51:52 triton_decode_attention.py:44] The following error message ‘operation scheduled before its operands’ can be ignored.<br>
(VllmWorkerProcess pid=1968800) WARNING 03-02 19:51:52 triton_decode_attention.py:44] The following error message ‘operation scheduled before its operands’ can be ignored.<br>
(VllmWorkerProcess pid=1968799) (VllmWorkerProcess pid=1968791) WARNING 03-02 19:51:52 triton_decode_attention.py:44] The following error message ‘operation scheduled before its operands’ can be ignored.<br>
WARNING 03-02 19:51:52 triton_decode_attention.py:44] The following error message ‘operation scheduled before its operands’ can be ignored.</p>
<p>then wait 10m，error eg.<br>
[E302 20:01:08.812651775 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (192.168.0.14, 54273).<br>
[W302 20:01:08.813334394 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host 192.168.0.14:54273 - retrying (try=0, timeout=600000ms, delay=66616ms): The client socket has timed out after 600000ms while trying to connect to (192.168.0.14, 54273).<br>
Exception raised from throwTimeoutError at …/torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7cc9f6c446 in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span>  + 0x15e04c6 (0x7f7cb514d4c6 in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0x6029d95 (0x7f7cb9b96d95 in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x6029f36 (0x7f7cb9b96f36 in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span>  + 0x602a3a4 (0x7f7cb9b973a4 in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#5:</span>  + 0x5fe8016 (0x7f7cb9b55016 in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#6:</span> c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&amp;) + 0x20c (0x7f7cb9b57f7c in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#7:</span>  + 0xd9acdd (0x7f7cc953acdd in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#8:</span>  + 0x4cb474 (0x7f7cc8c6b474 in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#9:</span>  + 0x15c31e (0x55e128ee731e in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#10:</span> _PyObject_MakeTpCall + 0x25b (0x55e128edde4b in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#11:</span>  + 0x16a9f0 (0x55e128ef59f0 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#12:</span>  + 0x166de7 (0x55e128ef1de7 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#13:</span>  + 0x1531fb (0x55e128ede1fb in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#14:</span>  + 0x4c9ccb (0x7f7cc8c69ccb in /root/deepseek-env/lib/python3.10/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#15:</span> _PyObject_MakeTpCall + 0x25b (0x55e128edde4b in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#16:</span> _PyEval_EvalFrameDefault + 0x6542 (0x55e128ed64f2 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#17:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#18:</span> _PyEval_EvalFrameDefault + 0x6c5 (0x55e128ed0675 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#19:</span>  + 0x2030a5 (0x55e128f8e0a5 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#20:</span>  + 0x15cdc9 (0x55e128ee7dc9 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#21:</span> _PyEval_EvalFrameDefault + 0x6c5 (0x55e128ed0675 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#22:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#23:</span> PyObject_Call + 0x122 (0x55e128ef6262 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#24:</span> _PyEval_EvalFrameDefault + 0x289f (0x55e128ed284f in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#25:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#26:</span> PyObject_Call + 0x122 (0x55e128ef6262 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#27:</span> _PyEval_EvalFrameDefault + 0x289f (0x55e128ed284f in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#28:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#29:</span> _PyEval_EvalFrameDefault + 0x19b6 (0x55e128ed1966 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#30:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#31:</span> _PyEval_EvalFrameDefault + 0x6c5 (0x55e128ed0675 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#32:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#33:</span> _PyEval_EvalFrameDefault + 0x6c5 (0x55e128ed0675 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#34:</span>  + 0x16a821 (0x55e128ef5821 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#35:</span> _PyEval_EvalFrameDefault + 0x289f (0x55e128ed284f in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#36:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#37:</span> _PyEval_EvalFrameDefault + 0x6c5 (0x55e128ed0675 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#38:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#39:</span> PyObject_Call + 0x122 (0x55e128ef6262 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#40:</span> _PyEval_EvalFrameDefault + 0x289f (0x55e128ed284f in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#41:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#42:</span> _PyEval_EvalFrameDefault + 0x8b6 (0x55e128ed0866 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#43:</span>  + 0x16a5c1 (0x55e128ef55c1 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#44:</span> _PyEval_EvalFrameDefault + 0x19b6 (0x55e128ed1966 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#45:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#46:</span> _PyEval_EvalFrameDefault + 0x8b6 (0x55e128ed0866 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#47:</span> _PyObject_FastCallDictTstate + 0xc4 (0x55e128edcfd4 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#48:</span>  + 0x1667c4 (0x55e128ef17c4 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#49:</span> _PyObject_MakeTpCall + 0x1fc (0x55e128edddec in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#50:</span> _PyEval_EvalFrameDefault + 0x6542 (0x55e128ed64f2 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#51:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#52:</span> _PyEval_EvalFrameDefault + 0x61a2 (0x55e128ed6152 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#53:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#54:</span> _PyEval_EvalFrameDefault + 0x8b6 (0x55e128ed0866 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#55:</span> _PyObject_FastCallDictTstate + 0xc4 (0x55e128edcfd4 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#56:</span>  + 0x1667c4 (0x55e128ef17c4 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#57:</span> _PyObject_MakeTpCall + 0x1fc (0x55e128edddec in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#58:</span> _PyEval_EvalFrameDefault + 0x6542 (0x55e128ed64f2 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#59:</span> _PyFunction_Vectorcall + 0x7c (0x55e128ee7b6c in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#60:</span> _PyEval_EvalFrameDefault + 0x8b6 (0x55e128ed0866 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#61:</span>  + 0x16a5c1 (0x55e128ef55c1 in /root/deepseek-env/bin/python3)<br>
frame <span class="hashtag-raw">#62:</span> PyObject_Call + 0x122 (0x55e128ef6262 in /root/deepseek-env/bin/python3)</p>
<p>Recurring error，about over 50 times，then</p>
<h1><a name="p-466523-traceback-most-recent-call-last-file-rootstart_vllmpy-line-54-in-module-main-file-rootstart_vllmpy-line-30-in-main-llm-llm-file-rootdeepseek-envlibpython310site-packagesvllmutilspy-line-1022-in-inner-return-fnargs-kwargs-file-rootdeepseek-envlibpython310site-packagesvllmentrypointsllmpy-line-242-in-__init__-selfllm_engine-selfengine_classfrom_engine_args-file-rootdeepseek-envlibpython310site-packagesvllmenginellm_enginepy-line-489-in-from_engine_args-engine-cls-file-rootdeepseek-envlibpython310site-packagesvllmenginellm_enginepy-line-273-in-__init__-selfmodel_executor-executor_classvllm_configvllm_config-file-rootdeepseek-envlibpython310site-packagesvllmexecutorexecutor_basepy-line-271-in-__init__-super__init__args-kwargs-file-rootdeepseek-envlibpython310site-packagesvllmexecutorexecutor_basepy-line-52-in-__init__-self_init_executor-file-rootdeepseek-envlibpython310site-packagesvllmexecutormp_distributed_executorpy-line-124-in-_init_executor-self_run_workersinit_device-file-rootdeepseek-envlibpython310site-packagesvllmexecutormp_distributed_executorpy-line-185-in-_run_workers-driver_worker_output-run_methodselfdriver_worker-sent_method-file-rootdeepseek-envlibpython310site-packagesvllmutilspy-line-2196-in-run_method-return-funcargs-kwargs-file-rootdeepseek-envlibpython310site-packagesvllmworkerworkerpy-line-166-in-init_device-init_worker_distributed_environmentselfvllm_config-selfrank-file-rootdeepseek-envlibpython310site-packagesvllmworkerworkerpy-line-504-in-init_worker_distributed_environment-init_distributed_environmentparallel_configworld_size-rank-file-rootdeepseek-envlibpython310site-packagesvllmdistributedparallel_statepy-line-819-in-init_distributed_environment-torchdistributedinit_process_group-file-rootdeepseek-envlibpython310site-packagestorchdistributedc10d_loggerpy-line-83-in-wrapper-return-funcargs-kwargs-file-rootdeepseek-envlibpython310site-packagestorchdistributedc10d_loggerpy-line-97-in-wrapper-func_return-funcargs-kwargs-file-rootdeepseek-envlibpython310site-packagestorchdistributeddistributed_c10dpy-line-1520-in-init_process_group-store-rank-world_size-nextrendezvous_iterator-file-rootdeepseek-envlibpython310site-packagestorchdistributedrendezvouspy-line-221-in-_tcp_rendezvous_handler-store-_create_c10d_store-file-rootdeepseek-envlibpython310site-packagestorchdistributedrendezvouspy-line-185-in-_create_c10d_store-tcp_store-tcpstorehostname-port-world_size-false-timeout-torchdistributeddistnetworkerror-the-client-socket-has-timed-out-after-600000ms-while-trying-to-connect-to-192168014-54779-error-03-02-201130-multiproc_worker_utilspy124-worker-vllmworkerprocess-pid-2019556-died-exit-code-15-info-03-02-201130-multiproc_worker_utilspy128-killing-local-vllm-worker-processes-w0302-201131454276-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019182-closing-signal-sigterm-w0302-201131455976-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019183-closing-signal-sigterm-w0302-201131457252-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019184-closing-signal-sigterm-w0302-201131458984-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019185-closing-signal-sigterm-w0302-201131460362-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019186-closing-signal-sigterm-w0302-201131461746-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019187-closing-signal-sigterm-w0302-201131463161-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019188-closing-signal-sigterm-e0302-201137014529-2019108-torchdistributedelasticmultiprocessingapipy869-failed-exitcode-1-local_rank-0-pid-2019181-of-binary-rootdeepseek-envbinpython3-traceback-most-recent-call-last-file-rootdeepseek-envbintorchrun-line-8-in-module-sysexitmain-file-rootdeepseek-envlibpython310site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-355-in-wrapper-return-fargs-kwargs-file-rootdeepseek-envlibpython310site-packagestorchdistributedrunpy-line-919-in-main-runargs-file-rootdeepseek-envlibpython310site-packagestorchdistributedrunpy-line-910-in-run-elastic_launch-file-rootdeepseek-envlibpython310site-packagestorchdistributedlauncherapipy-line-138-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-rootdeepseek-envlibpython310site-packagestorchdistributedlauncherapipy-line-269-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#p-466523-traceback-most-recent-call-last-file-rootstart_vllmpy-line-54-in-module-main-file-rootstart_vllmpy-line-30-in-main-llm-llm-file-rootdeepseek-envlibpython310site-packagesvllmutilspy-line-1022-in-inner-return-fnargs-kwargs-file-rootdeepseek-envlibpython310site-packagesvllmentrypointsllmpy-line-242-in-__init__-selfllm_engine-selfengine_classfrom_engine_args-file-rootdeepseek-envlibpython310site-packagesvllmenginellm_enginepy-line-489-in-from_engine_args-engine-cls-file-rootdeepseek-envlibpython310site-packagesvllmenginellm_enginepy-line-273-in-__init__-selfmodel_executor-executor_classvllm_configvllm_config-file-rootdeepseek-envlibpython310site-packagesvllmexecutorexecutor_basepy-line-271-in-__init__-super__init__args-kwargs-file-rootdeepseek-envlibpython310site-packagesvllmexecutorexecutor_basepy-line-52-in-__init__-self_init_executor-file-rootdeepseek-envlibpython310site-packagesvllmexecutormp_distributed_executorpy-line-124-in-_init_executor-self_run_workersinit_device-file-rootdeepseek-envlibpython310site-packagesvllmexecutormp_distributed_executorpy-line-185-in-_run_workers-driver_worker_output-run_methodselfdriver_worker-sent_method-file-rootdeepseek-envlibpython310site-packagesvllmutilspy-line-2196-in-run_method-return-funcargs-kwargs-file-rootdeepseek-envlibpython310site-packagesvllmworkerworkerpy-line-166-in-init_device-init_worker_distributed_environmentselfvllm_config-selfrank-file-rootdeepseek-envlibpython310site-packagesvllmworkerworkerpy-line-504-in-init_worker_distributed_environment-init_distributed_environmentparallel_configworld_size-rank-file-rootdeepseek-envlibpython310site-packagesvllmdistributedparallel_statepy-line-819-in-init_distributed_environment-torchdistributedinit_process_group-file-rootdeepseek-envlibpython310site-packagestorchdistributedc10d_loggerpy-line-83-in-wrapper-return-funcargs-kwargs-file-rootdeepseek-envlibpython310site-packagestorchdistributedc10d_loggerpy-line-97-in-wrapper-func_return-funcargs-kwargs-file-rootdeepseek-envlibpython310site-packagestorchdistributeddistributed_c10dpy-line-1520-in-init_process_group-store-rank-world_size-nextrendezvous_iterator-file-rootdeepseek-envlibpython310site-packagestorchdistributedrendezvouspy-line-221-in-_tcp_rendezvous_handler-store-_create_c10d_store-file-rootdeepseek-envlibpython310site-packagestorchdistributedrendezvouspy-line-185-in-_create_c10d_store-tcp_store-tcpstorehostname-port-world_size-false-timeout-torchdistributeddistnetworkerror-the-client-socket-has-timed-out-after-600000ms-while-trying-to-connect-to-192168014-54779-error-03-02-201130-multiproc_worker_utilspy124-worker-vllmworkerprocess-pid-2019556-died-exit-code-15-info-03-02-201130-multiproc_worker_utilspy128-killing-local-vllm-worker-processes-w0302-201131454276-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019182-closing-signal-sigterm-w0302-201131455976-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019183-closing-signal-sigterm-w0302-201131457252-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019184-closing-signal-sigterm-w0302-201131458984-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019185-closing-signal-sigterm-w0302-201131460362-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019186-closing-signal-sigterm-w0302-201131461746-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019187-closing-signal-sigterm-w0302-201131463161-2019108-torchdistributedelasticmultiprocessingapipy897-sending-process-2019188-closing-signal-sigterm-e0302-201137014529-2019108-torchdistributedelasticmultiprocessingapipy869-failed-exitcode-1-local_rank-0-pid-2019181-of-binary-rootdeepseek-envbinpython3-traceback-most-recent-call-last-file-rootdeepseek-envbintorchrun-line-8-in-module-sysexitmain-file-rootdeepseek-envlibpython310site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-355-in-wrapper-return-fargs-kwargs-file-rootdeepseek-envlibpython310site-packagestorchdistributedrunpy-line-919-in-main-runargs-file-rootdeepseek-envlibpython310site-packagestorchdistributedrunpy-line-910-in-run-elastic_launch-file-rootdeepseek-envlibpython310site-packagestorchdistributedlauncherapipy-line-138-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-rootdeepseek-envlibpython310site-packagestorchdistributedlauncherapipy-line-269-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>Traceback (most recent call last):<br>
File “/root/start_vllm.py”, line 54, in <br>
main()<br>
File “/root/start_vllm.py”, line 30, in main<br>
llm = LLM(<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/utils.py”, line 1022, in inner<br>
return fn(*args, **kwargs)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/entrypoints/llm.py”, line 242, in <strong>init</strong><br>
self.llm_engine = self.engine_class.from_engine_args(<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/engine/llm_engine.py”, line 489, in from_engine_args<br>
engine = cls(<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/engine/llm_engine.py”, line 273, in <strong>init</strong><br>
self.model_executor = executor_class(vllm_config=vllm_config, )<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/executor/executor_base.py”, line 271, in <strong>init</strong><br>
super().<strong>init</strong>(*args, **kwargs)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/executor/executor_base.py”, line 52, in <strong>init</strong><br>
self._init_executor()<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py”, line 124, in _init_executor<br>
self._run_workers(“init_device”)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py”, line 185, in _run_workers<br>
driver_worker_output = run_method(self.driver_worker, sent_method,<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/utils.py”, line 2196, in run_method<br>
return func(*args, **kwargs)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/worker/worker.py”, line 166, in init_device<br>
init_worker_distributed_environment(self.vllm_config, self.rank,<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/worker/worker.py”, line 504, in init_worker_distributed_environment<br>
init_distributed_environment(parallel_config.world_size, rank,<br>
File “/root/deepseek-env/lib/python3.10/site-packages/vllm/distributed/parallel_state.py”, line 819, in init_distributed_environment<br>
torch.distributed.init_process_group(<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/c10d_logger.py”, line 83, in wrapper<br>
return func(*args, **kwargs)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/c10d_logger.py”, line 97, in wrapper<br>
func_return = func(*args, **kwargs)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py”, line 1520, in init_process_group<br>
store, rank, world_size = next(rendezvous_iterator)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/rendezvous.py”, line 221, in _tcp_rendezvous_handler<br>
store = _create_c10d_store(<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/rendezvous.py”, line 185, in _create_c10d_store<br>
tcp_store = TCPStore(hostname, port, world_size, False, timeout)<br>
torch.distributed.DistNetworkError: The client socket has timed out after 600000ms while trying to connect to (192.168.0.14, 54779).<br>
ERROR 03-02 20:11:30 multiproc_worker_utils.py:124] Worker VllmWorkerProcess pid 2019556 died, exit code: -15<br>
INFO 03-02 20:11:30 multiproc_worker_utils.py:128] Killing local vLLM worker processes<br>
W0302 20:11:31.454276 2019108 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2019182 closing signal SIGTERM<br>
W0302 20:11:31.455976 2019108 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2019183 closing signal SIGTERM<br>
W0302 20:11:31.457252 2019108 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2019184 closing signal SIGTERM<br>
W0302 20:11:31.458984 2019108 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2019185 closing signal SIGTERM<br>
W0302 20:11:31.460362 2019108 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2019186 closing signal SIGTERM<br>
W0302 20:11:31.461746 2019108 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2019187 closing signal SIGTERM<br>
W0302 20:11:31.463161 2019108 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 2019188 closing signal SIGTERM<br>
E0302 20:11:37.014529 2019108 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 2019181) of binary: /root/deepseek-env/bin/python3<br>
Traceback (most recent call last):<br>
File “/root/deepseek-env/bin/torchrun”, line 8, in <br>
sys.exit(main())<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 355, in wrapper<br>
return f(*args, **kwargs)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/run.py”, line 919, in main<br>
run(args)<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/run.py”, line 910, in run<br>
elastic_launch(<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/launcher/api.py”, line 138, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/root/deepseek-env/lib/python3.10/site-packages/torch/distributed/launcher/api.py”, line 269, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2><a name="p-466523-start_vllmpy-failed-2" class="anchor" href="#p-466523-start_vllmpy-failed-2"></a>start_vllm.py FAILED</h2>
<h2><a name="p-466523-failures-no_other_failures-3" class="anchor" href="#p-466523-failures-no_other_failures-3"></a>Failures:<br>
&lt;NO_OTHER_FAILURES&gt;</h2>
<h1><a name="p-466523-root-cause-first-observed-failure-0-time-2025-03-02_201131-host-h20-2-rank-8-local_rank-0-exitcode-1-pid-2019181-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4" class="anchor" href="#p-466523-root-cause-first-observed-failure-0-time-2025-03-02_201131-host-h20-2-rank-8-local_rank-0-exitcode-1-pid-2019181-error_file-na-traceback-to-enable-traceback-see-httpspytorchorgdocsstableelasticerrorshtml-4"></a>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2025-03-02_20:11:31<br>
host      : h20-2<br>
rank      : 8 (local_rank: 0)<br>
exitcode  : 1 (pid: 2019181)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.6 documentation</a></h1>
<p>can someone help，tkx！</p>

zhanwenchen: <p>I encountered a similar issue. What’s your vllm/ray version?</p>

yishuizhao: <p>vllm 0.7.3<br>
ray 2.40.0</p>

bcml: <p>I have same issue, did you find a fix?<br>
vllm 0.8.2<br>
ray 2.43.0</p>