Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:675 (most recent call first):

username: gaopeifeng
cooked: <p>WARNING 08-20 16:57:38 [config.py:1363] Default sampling parameters have been overridden by the model’s Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with <code>--generation-config vllm</code>.<br>
INFO 08-20 16:57:38 [serving_chat.py:118] Using default chat sampling params from model: {‘repetition_penalty’: 1.05, ‘temperature’: 0.7, ‘top_k’: 20, ‘top_p’: 0.8}<br>
INFO 08-20 16:57:38 [serving_completion.py:66] Using default completion sampling params from model: {‘repetition_penalty’: 1.05, ‘temperature’: 0.7, ‘top_k’: 20, ‘top_p’: 0.8}<br>
INFO 08-20 16:57:38 [api_server.py:1349] Starting vLLM API server 0 on <a href="http://0.0.0.0:8006" rel="noopener nofollow ugc">http://0.0.0.0:8006</a><br>
INFO 08-20 16:57:38 [launcher.py:29] Available routes are:<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /openapi.json, Methods: GET, HEAD<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /docs, Methods: GET, HEAD<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: GET, HEAD<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /redoc, Methods: GET, HEAD<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /health, Methods: GET<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /load, Methods: GET<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /ping, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /ping, Methods: GET<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /tokenize, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /detokenize, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /v1/models, Methods: GET<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /version, Methods: GET<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /v1/chat/completions, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /v1/completions, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /v1/embeddings, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /pooling, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /classify, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /score, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /v1/score, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /rerank, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /v1/rerank, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /v2/rerank, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /invocations, Methods: POST<br>
INFO 08-20 16:57:38 [launcher.py:37] Route: /metrics, Methods: GET<br>
INFO:     Started server process [10201]<br>
INFO:     Waiting for application startup.<br>
INFO:     Application startup complete.<br>
INFO 08-20 16:58:54 [chat_utils.py:420] Detected the chat template content format to be ‘string’. You can set <code>--chat-template-content-format</code> to override this.<br>
INFO 08-20 16:58:54 [logger.py:43] Received request chatcmpl-ba8950d2c453476285ef8fac5facef96: prompt: ‘&lt;|im_start|&gt;system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.&lt;|im_end|&gt;\n&lt;|im_start|&gt;user\n你好啊&lt;|im_end|&gt;\n&lt;|im_start|&gt;assistant\n’, params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=<span class="chcklst-box fa fa-square-o fa-fw"></span>, stop_token_ids=<span class="chcklst-box fa fa-square-o fa-fw"></span>, bad_words=<span class="chcklst-box fa fa-square-o fa-fw"></span>, include_stop_str_in_output=False, ignore_eos=False, max_tokens=32737, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.<br>
INFO 08-20 16:58:54 [async_llm.py:271] Added request chatcmpl-ba8950d2c453476285ef8fac5facef96.<br>
INFO:     10.13.51.205:0 - “POST /v1/chat/completions HTTP/1.0” 200 OK<br>
INFO 08-20 16:58:58 [loggers.py:118] Engine 000: Avg prompt throughput: 3.1 tokens/s, Avg generation throughput: 0.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%<br>
INFO 08-20 16:58:59 [logger.py:43] Received request chatcmpl-7a1ee86b3d47400da888856221697c76: prompt: ‘&lt;|im_start|&gt;system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.&lt;|im_end|&gt;\n&lt;|im_start|&gt;user\n你好啊&lt;|im_end|&gt;\n&lt;|im_start|&gt;assistant\n你好！有什么可以帮助你的吗？&lt;|im_end|&gt;\n&lt;|im_start|&gt;user\n你是谁&lt;|im_end|&gt;\n&lt;|im_start|&gt;assistant\n’, params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=<span class="chcklst-box fa fa-square-o fa-fw"></span>, stop_token_ids=<span class="chcklst-box fa fa-square-o fa-fw"></span>, bad_words=<span class="chcklst-box fa fa-square-o fa-fw"></span>, include_stop_str_in_output=False, ignore_eos=False, max_tokens=32718, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.<br>
INFO 08-20 16:58:59 [async_llm.py:271] Added request chatcmpl-7a1ee86b3d47400da888856221697c76.<br>
INFO:     10.13.51.205:0 - “POST /v1/chat/completions HTTP/1.0” 200 OK<br>
INFO 08-20 16:59:08 [loggers.py:118] Engine 000: Avg prompt throughput: 5.0 tokens/s, Avg generation throughput: 6.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%<br>
INFO 08-20 16:59:18 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%<br>
INFO 09-16 09:09:59 [logger.py:43] Received request chatcmpl- SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=<span class="chcklst-box fa fa-square-o fa-fw"></span>, stop_token_ids=<span class="chcklst-box fa fa-square-o fa-fw"></span>, bad_words=<span class="chcklst-box fa fa-square-o fa-fw"></span>, include_stop_str_in_output=False, ignore_eos=False, max_tokens=30750, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.<br>
INFO 09-16 09:09:59 [async_llm.py:271] Added request chatcmpl-4e7f382fadcf4b7a88778ee277b48f05.<br>
INFO:     10.13.51.205:0 - “POST /v1/chat/completions HTTP/1.0” 200 OK<br>
[rank0]:[E916 09:09:59.303108842 ProcessGroupNCCL.cpp:1896] [PG ID 2 PG GUID 3 Rank 0] Process group watchdog thread terminated with exception: CUDA error: unknown error<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;) + 0x98 (0x7fb2332975e8 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;) + 0xe0 (0x7fb23322c4a2 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#2:</span> c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7fb233375422 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#3:</span> c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fb2345a2476 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#4:</span> c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7fb2345b2710 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#5:</span> c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7fb2345b42a2 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#6:</span> c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fb2345b5ead in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#7:</span>  + 0xdbbf4 (0x7fb224579bf4 in /data/miniconda3/envs/vllm_0.9.1/bin/../lib/libstdc++.so.6)<br>
frame <span class="hashtag-raw">#8:</span>  + 0x8fed (0x7fb29bbd8fed in /lib64/libpthread.so.0)<br>
frame <span class="hashtag-raw">#9:</span> clone + 0x3f (0x7fb29b97f18f in /lib64/libc.so.6)</p>
<p>ERROR 09-16 09:10:04 [multiproc_executor.py:140] Worker proc VllmWorker-0 died unexpectedly, shutting down executor.<br>
[rank1]:[W916 09:10:04.997585247 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=80, addr=[localhost]:34610, remote=[localhost]:51295): Connection reset by peer<br>
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:675 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;) + 0x98 (0x7efd8ae505e8 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span>  + 0x5ba8afe (0x7efdcef73afe in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0x5baaecf (0x7efdcef75ecf in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x5bab74a (0x7efdcef7674a in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span> c10d::TCPStore::check(std::vector&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::allocator&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; const&amp;) + 0x2a9 (0x7efdcef701a9 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#5:</span> c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7efd8c1699a9 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0xdbbf4 (0x7efd7c132bf4 in /data/miniconda3/envs/vllm_0.9.1/bin/../lib/libstdc++.so.6)<br>
frame <span class="hashtag-raw">#7:</span>  + 0x8fed (0x7efdf3791fed in /lib64/libpthread.so.0)<br>
frame <span class="hashtag-raw">#8:</span> clone + 0x3f (0x7efdf353818f in /lib64/libc.so.6)</p>
<p>[rank1]:[W916 09:10:04.023253087 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 1] Failed to check the “should dump” flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Connection reset by peer<br>
[rank1]:[W916 09:10:05.023369978 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=80, addr=[localhost]:34610, remote=[localhost]:51295): Broken pipe<br>
Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;) + 0x98 (0x7efd8ae505e8 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span>  + 0x5ba8afe (0x7efdcef73afe in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0x5baa358 (0x7efdcef75358 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x5babb3e (0x7efdcef76b3e in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span> c10d::TCPStore::check(std::vector&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::allocator&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; const&amp;) + 0x298 (0x7efdcef70198 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#5:</span> c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7efd8c1699a9 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0xdbbf4 (0x7efd7c132bf4 in /data/miniconda3/envs/vllm_0.9.1/bin/../lib/libstdc++.so.6)<br>
frame <span class="hashtag-raw">#7:</span>  + 0x8fed (0x7efdf3791fed in /lib64/libpthread.so.0)<br>
frame <span class="hashtag-raw">#8:</span> clone + 0x3f (0x7efdf353818f in /lib64/libc.so.6)</p>
<p>[rank1]:[W916 09:10:05.026869604 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 1] Failed to check the “should dump” flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe<br>
[rank1]:[W916 09:10:06.026969474 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=80, addr=[localhost]:34610, remote=[localhost]:51295): Broken pipe<br>
Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;) + 0x98 (0x7efd8ae505e8 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span>  + 0x5ba8afe (0x7efdcef73afe in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0x5baa358 (0x7efdcef75358 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x5babb3e (0x7efdcef76b3e in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span> c10d::TCPStore::check(std::vector&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::allocator&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; const&amp;) + 0x298 (0x7efdcef70198 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#5:</span> c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7efd8c1699a9 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0xdbbf4 (0x7efd7c132bf4 in /data/miniconda3/envs/vllm_0.9.1/bin/../lib/libstdc++.so.6)<br>
frame <span class="hashtag-raw">#7:</span>  + 0x8fed (0x7efdf3791fed in /lib64/libpthread.so.0)<br>
frame <span class="hashtag-raw">#8:</span> clone + 0x3f (0x7efdf353818f in /lib64/libc.so.6)</p>
<p>[rank1]:[W916 09:10:06.030563545 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 1] Failed to check the “should dump” flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe<br>
[rank1]:[W916 09:10:07.030705811 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=80, addr=[localhost]:34610, remote=[localhost]:51295): Broken pipe<br>
Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:653 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;) + 0x98 (0x7efd8ae505e8 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span>  + 0x5ba8afe (0x7efdcef73afe in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0x5baa358 (0x7efdcef75358 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x5babb3e (0x7efdcef76b3e in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span> c10d::TCPStore::check(std::vector&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;, std::allocator&lt;std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; &gt; &gt; const&amp;) + 0x298 (0x7efdcef70198 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#5:</span> c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7efd8c1699a9 in /data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#6:</span>  + 0xdbbf4 (0x7efd7c132bf4 in /data/miniconda3/envs/vllm_0.9.1/bin/../lib/libstdc++.so.6)<br>
frame <span class="hashtag-raw">#7:</span>  + 0x8fed (0x7efdf3791fed in /lib64/libpthread.so.0)<br>
frame <span class="hashtag-raw">#8:</span> clone + 0x3f (0x7efdf353818f in /lib64/libc.so.6)</p>
<p>[rank1]:[W916 09:10:07.034255233 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 1] Failed to check the “should dump” flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe<br>
INFO 09-16 09:10:59 [async_llm.py:432] Aborted request chatcmpl-4e7f382fadcf4b7a88778ee277b48f05.<br>
INFO 09-16 09:10:59 [async_llm.py:340] Request chatcmpl-4e7f382fadcf4b7a88778ee277b48f05 aborted.<br>
INFO 09-16 09:11:06 [logger.py:43] Received request chatcmpl-SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=<span class="chcklst-box fa fa-square-o fa-fw"></span>, stop_token_ids=<span class="chcklst-box fa fa-square-o fa-fw"></span>, bad_words=<span class="chcklst-box fa fa-square-o fa-fw"></span>, include_stop_str_in_output=False, ignore_eos=False, max_tokens=30750, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.<br>
INFO 09-16 09:11:06 [async_llm.py:271] Added request chatcmpl-a51f12a52ba242ca9d922a358a3b4830.<br>
INFO:     10.13.51.205:0 - “POST /v1/chat/completions HTTP/1.0” 200 OK<br>
INFO 09-16 09:12:07 [async_llm.py:432] Aborted request chatcmpl-a51f12a52ba242ca9d922a358a3b4830.<br>
INFO 09-16 09:12:07 [async_llm.py:340] Request chatcmpl-a51f12a52ba242ca9d922a358a3b4830 aborted.<br>
ERROR 09-16 09:14:59 [dump_input.py:69] Dumping input data<br>
ERROR 09-16 09:14:59 [dump_input.py:71] V1 LLM engine (v0.9.1) with config: model=‘/data/xkmodel/Qwen/Qwen2___5-72B-Instruct’, speculative_config=None, tokenizer=‘/data/xkmodel/Qwen/Qwen2___5-72B-Instruct’, skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend=‘auto’, disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=‘’), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen2.5-72B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={“level”:0,“debug_dump_path”:“”,“cache_dir”:“”,“backend”:“”,“custom_ops”:<span class="chcklst-box fa fa-square-o fa-fw"></span>,“splitting_ops”:<span class="chcklst-box fa fa-square-o fa-fw"></span>,“use_inductor”:true,“compile_sizes”:<span class="chcklst-box fa fa-square-o fa-fw"></span>,“inductor_compile_config”:{“enable_auto_functionalized_v2”:false},“inductor_passes”:{},“use_cudagraph”:true,“cudagraph_num_of_warmups”:0,“cudagraph_capture_sizes”:<span class="chcklst-box fa fa-square-o fa-fw"></span>,“cudagraph_copy_inputs”:false,“full_cuda_graph”:false,“max_capture_size”:0,“local_cache_dir”:null},<br>
ERROR 09-16 09:14:59 [dump_input.py:79] Dumping scheduler output for model execution:<br>
ERROR 09-16 09:14:59 [dump_input.py:80] SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-4e7f382fadcf4b7a88778ee277b48f05,prompt_token_ids_len=2018,mm_inputs=<span class="chcklst-box fa fa-square-o fa-fw"></span>,mm_hashes=<span class="chcklst-box fa fa-square-o fa-fw"></span>,mm_positions=<span class="chcklst-box fa fa-square-o fa-fw"></span>,sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, seed=None, stop=<span class="chcklst-box fa fa-square-o fa-fw"></span>, stop_token_ids=[151643], bad_words=<span class="chcklst-box fa fa-square-o fa-fw"></span>, include_stop_str_in_output=False, ignore_eos=False, max_tokens=30750, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=([11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137],),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=<span class="chcklst-box fa fa-square-o fa-fw"></span>, num_scheduled_tokens={chatcmpl-4e7f382fadcf4b7a88778ee277b48f05: 2018}, total_num_scheduled_tokens=2018, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[127], finished_req_ids=<span class="chcklst-box fa fa-square-o fa-fw"></span>, free_encoder_input_ids=<span class="chcklst-box fa fa-square-o fa-fw"></span>, structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)<br>
ERROR 09-16 09:14:59 [dump_input.py:82] SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, gpu_cache_usage=0.05313408053134083, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None)<br>
ERROR 09-16 09:14:59 [core.py:517] EngineCore encountered a fatal error.<br>
ERROR 09-16 09:14:59 [core.py:517] Traceback (most recent call last):<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 220, in collective_rpc<br>
ERROR 09-16 09:14:59 [core.py:517]     result = get_response(w, dequeue_timeout)<br>
ERROR 09-16 09:14:59 [core.py:517]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 203, in get_response<br>
ERROR 09-16 09:14:59 [core.py:517]     status, result = w.worker_response_mq.dequeue(<br>
ERROR 09-16 09:14:59 [core.py:517]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/distributed/device_communicators/shm_broadcast.py”, line 507, in dequeue<br>
ERROR 09-16 09:14:59 [core.py:517]     with self.acquire_read(timeout, cancel) as buf:<br>
ERROR 09-16 09:14:59 [core.py:517]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/contextlib.py”, line 137, in <strong>enter</strong><br>
ERROR 09-16 09:14:59 [core.py:517]     return next(self.gen)<br>
ERROR 09-16 09:14:59 [core.py:517]            ^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/distributed/device_communicators/shm_broadcast.py”, line 469, in acquire_read<br>
ERROR 09-16 09:14:59 [core.py:517]     raise TimeoutError<br>
ERROR 09-16 09:14:59 [core.py:517] TimeoutError<br>
ERROR 09-16 09:14:59 [core.py:517]<br>
ERROR 09-16 09:14:59 [core.py:517] The above exception was the direct cause of the following exception:<br>
ERROR 09-16 09:14:59 [core.py:517]<br>
ERROR 09-16 09:14:59 [core.py:517] Traceback (most recent call last):<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 508, in run_engine_core<br>
ERROR 09-16 09:14:59 [core.py:517]     engine_core.run_busy_loop()<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 535, in run_busy_loop<br>
ERROR 09-16 09:14:59 [core.py:517]     self._process_engine_step()<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 560, in _process_engine_step<br>
ERROR 09-16 09:14:59 [core.py:517]     outputs, model_executed = self.step_fn()<br>
ERROR 09-16 09:14:59 [core.py:517]                               ^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 231, in step<br>
ERROR 09-16 09:14:59 [core.py:517]     model_output = self.execute_model(scheduler_output)<br>
ERROR 09-16 09:14:59 [core.py:517]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 217, in execute_model<br>
ERROR 09-16 09:14:59 [core.py:517]     raise err<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 211, in execute_model<br>
ERROR 09-16 09:14:59 [core.py:517]     return self.model_executor.execute_model(scheduler_output)<br>
ERROR 09-16 09:14:59 [core.py:517]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 163, in execute_model<br>
ERROR 09-16 09:14:59 [core.py:517]     (output, ) = self.collective_rpc(“execute_model”,<br>
ERROR 09-16 09:14:59 [core.py:517]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [core.py:517]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 226, in collective_rpc<br>
ERROR 09-16 09:14:59 [core.py:517]     raise TimeoutError(f"RPC call to {method} timed out.") from e<br>
ERROR 09-16 09:14:59 [core.py:517] TimeoutError: RPC call to execute_model timed out.<br>
Process EngineCore_0:<br>
Traceback (most recent call last):<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 220, in collective_rpc<br>
result = get_response(w, dequeue_timeout)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 203, in get_response<br>
status, result = w.worker_response_mq.dequeue(<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/distributed/device_communicators/shm_broadcast.py”, line 507, in dequeue<br>
with self.acquire_read(timeout, cancel) as buf:<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/contextlib.py”, line 137, in <strong>enter</strong><br>
return next(self.gen)<br>
^^^^^^^^^^^^^^<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/distributed/device_communicators/shm_broadcast.py”, line 469, in acquire_read<br>
raise TimeoutError<br>
TimeoutError</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/multiprocessing/process.py”, line 314, in _bootstrap<br>
self.run()<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/multiprocessing/process.py”, line 108, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 519, in run_engine_core<br>
raise e<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 508, in run_engine_core<br>
engine_core.run_busy_loop()<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 535, in run_busy_loop<br>
self._process_engine_step()<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 560, in _process_engine_step<br>
outputs, model_executed = self.step_fn()<br>
^^^^^^^^^^^^^^<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 231, in step<br>
model_output = self.execute_model(scheduler_output)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 217, in execute_model<br>
raise err<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core.py”, line 211, in execute_model<br>
return self.model_executor.execute_model(scheduler_output)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 163, in execute_model<br>
(output, ) = self.collective_rpc(“execute_model”,<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py”, line 226, in collective_rpc<br>
raise TimeoutError(f"RPC call to {method} timed out.") from e<br>
TimeoutError: RPC call to execute_model timed out.<br>
ERROR 09-16 09:14:59 [async_llm.py:420] AsyncLLM output_handler failed.<br>
ERROR 09-16 09:14:59 [async_llm.py:420] Traceback (most recent call last):<br>
ERROR 09-16 09:14:59 [async_llm.py:420]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py”, line 379, in output_handler<br>
ERROR 09-16 09:14:59 [async_llm.py:420]     outputs = await engine_core.get_output_async()<br>
ERROR 09-16 09:14:59 [async_llm.py:420]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
ERROR 09-16 09:14:59 [async_llm.py:420]   File “/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/site-packages/vllm/v1/engine/core_client.py”, line 790, in get_output_async<br>
ERROR 09-16 09:14:59 [async_llm.py:420]     raise self._format_exception(outputs) from None<br>
ERROR 09-16 09:14:59 [async_llm.py:420] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.<br>
INFO:     Shutting down<br>
INFO:     Waiting for application shutdown.<br>
INFO:     Application shutdown complete.<br>
INFO:     Finished server process [10201]<br>
/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown<br>
warnings.warn('resource_tracker: There appear to be %d ’<br>
/data/miniconda3/envs/vllm_0.9.1/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown<br>
warnings.warn('resource_tracker: There appear to be %d ’</p>