Repository: vllm-project/vllm
Pull request #15977: [V1] DP scale-out (2/N): Decouple engine process management and comms
Author: login: njhill
Labels: name: frontend
name: ready
description: ONLY add when PR is ready to merge/full CI is needed
name: v1
name: needs-tests
description: Tests needed for this PR
Created: 2025-04-03T01:18:54Z | Merged: 2025-05-13T17:48:21Z

This decouples the management of engine processes from the IPC, and adds support for a mix of local and/or remote engines (where remote are running on a different node).

When there are any remote engines, tcp transport is used for the zmq sockets, otherwise ipc (domain-socket based) is used.

Engines are bootstrapped with the input queue address and use this to perform a handshake with the front-end running in the head node, which provides other necessary configuration details:
```
  Front-End                    Engine Core (N)
      |                              |
 (1)  | <-------- HELLO ------------ |
 (2)  | ---- config / conn info ---> |  (address of output queue and data parallel torch process group)
      |                              | 
      |                  [ engine init - load model ]
      |                              |
 (3)  | <-------- READY ------------ |
```

There is a new `--headless` option for `vllm serve` to run on secondary nodes, which launches one or more engines (data parallel ranks) without the front-end / API server).

#### Examples

This will run DP=4 with DP ranks 0 and 1 on the head node and ranks 2 and 3 on the second node:
```
# Node 1  (with ip address 10.99.48.128)
vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 2 \
                  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345
# Node 2
vllm serve $MODEL --headless --data-parallel-size 4 --data-parallel-size-local 2 --data-parallel-start-rank 2 \
                  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345
```

This will run DP=4 with only the API server on the first node and all engines on the second node:
```
# Node 1  (with ip address 10.99.48.128)
vllm serve $MODEL --data-parallel-size 4 --data-parallel-size-local 0 \
                  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345
# Node 2
vllm serve $MODEL --headless --data-parallel-size 4 --data-parallel-size-local 4 \
                  --data-parallel-address 10.99.48.128 --data-parallel-rpc-port 13345
```

It's assumed that local engine ranks (if any) will always be lower than remote engine ranks. Note that it's not actually necessary to specify the (global) dp size on the secondary nodes since this is obtained during the handshake. It would be straightforward to extend this to any other config which must be consistent across the ranks - so that you only need to specify it in the head node cli args.

#### TODO (this PR):
- [x] Feedback
- [x] Some code simplification
- [x] Fix offline DP compatiblity
- [ ] CI tests

#### Next PR:
- API server scale-out: https://github.com/vllm-project/vllm/pull/17546


Diff:
diff --git a/tests/async_engine/test_async_llm_engine.py b/tests/async_engine/test_async_llm_engine.py
index 48e2e31e5db8..b6f44871497c 100644
--- a/tests/async_engine/test_async_llm_engine.py
+++ b/tests/async_engine/test_async_llm_engine.py
@@ -41,7 +41,7 @@ def __init__(self):
         self.abort_request_calls = 0
         self.request_id = None
         # Ugly, remove dependency when possible
-        self.parallel_config = ParallelConfig(1, 1, False)
+        self.parallel_config = ParallelConfig()
         self.model_config = MockModelConfig()
 
     async def step_async(self, virtual_engine):
diff --git a/tests/v1/engine/test_engine_core_client.py b/tests/v1/engine/test_engine_core_client.py
index fd8d1fd7ff48..452fe1e37e2c 100644
--- a/tests/v1/engine/test_engine_core_client.py
+++ b/tests/v1/engine/test_engine_core_client.py
@@ -18,9 +18,10 @@
 from vllm.usage.usage_lib import UsageContext
 from vllm.v1.engine import EngineCoreRequest
 from vllm.v1.engine.core import EngineCore
-from vllm.v1.engine.core_client import (AsyncMPClient, CoreEngine,
-                                        EngineCoreClient, SyncMPClient)
+from vllm.v1.engine.core_client import (AsyncMPClient, EngineCoreClient,
+                                        SyncMPClient)
 from vllm.v1.executor.abstract import Executor
+from vllm.v1.utils import CoreEngineProcManager
 
 from ...distributed.conftest import MockSubscriber
 from ...utils import create_new_process_for_each_test
@@ -348,13 +349,13 @@ def test_startup_failure(monkeypatch: pytest.MonkeyPatch):
 
         # Monkey-patch to extract core process pid while it's starting.
         core_proc_pid = [None]
-        ce_ctor = CoreEngine.__init__
+        cepm_ctor = CoreEngineProcManager.__init__
 
-        def patched_ce_ctor(self, *args, **kwargs):
-            ce_ctor(self, *args, **kwargs)
-            core_proc_pid[0] = self.proc_handle.proc.pid
+        def patched_cepm_ctor(self: CoreEngineProcManager, *args, **kwargs):
+            cepm_ctor(self, *args, **kwargs)
+            core_proc_pid[0] = self.processes[0].pid
 
-        m.setattr(CoreEngine, "__init__", patched_ce_ctor)
+        m.setattr(CoreEngineProcManager, "__init__", patched_cepm_ctor)
 
         t = time.time()
         engine_args = EngineArgs(model=MODEL_NAME)
diff --git a/vllm/config.py b/vllm/config.py
index dd0791537b96..d8eabfb2e4f0 100644
--- a/vllm/config.py
+++ b/vllm/config.py
@@ -1668,25 +1668,17 @@ class ParallelConfig:
     data_parallel_size: int = 1
     """Number of data parallel groups. MoE layers will be sharded according to
     the product of the tensor parallel size and data parallel size."""
+    data_parallel_size_local: int = 1
+    """Number of local data parallel groups."""
     data_parallel_rank: int = 0
     """Rank of the data parallel group."""
-    _data_parallel_rank_local: Optional[int] = field(default=None, init=False)
-    """Private field to store the local rank of the data parallel group."""
-
-    @property
-    def data_parallel_rank_local(self) -> int:
-        """Local rank of the data parallel group, defaults to global rank."""
-        if self._data_parallel_rank_local is None:
-            return self.data_parallel_rank
-        return self._data_parallel_rank_local
-
-    @data_parallel_rank_local.setter
-    def data_parallel_rank_local(self, value: int) -> None:
-        """Set the local rank of the data parallel group."""
-        self._data_parallel_rank_local = value
-
+    data_parallel_rank_local: Optional[int] = None
+    """Local rank of the data parallel group,
+    set only in SPMD mode."""
     data_parallel_master_ip: str = "127.0.0.1"
     """IP of the data parallel master."""
+    data_parallel_rpc_port: int = 29550
+    """Port for data parallel messaging."""
     data_parallel_master_port: int = 29500
     """Port of the data parallel master."""
     enable_expert_parallel: bool = False
@@ -1734,13 +1726,16 @@ class is dynamically inherited by the worker class. This is used to inject
 
     world_size: int = field(init=False)
     """world_size is TPxPP, it affects the number of workers we create."""
-    world_size_across_dp: int = field(init=False)
-    """world_size_across_dp is TPxPPxDP, it is the size of the world
-    including data parallelism."""
 
     rank: int = 0
     """Global rank in distributed setup."""
 
+    @property
+    def world_size_across_dp(self) -> int:
+        """world_size_across_dp is TPxPPxDP, it is the size of the world
+        including data parallelism."""
+        return self.world_size * self.data_parallel_size
+
     def get_next_dp_init_port(self) -> int:
         """
         We might need to initialize process groups in multiple
@@ -1800,10 +1795,14 @@ def __post_init__(self) -> None:
         self.world_size = self.pipeline_parallel_size * \
             self.tensor_parallel_size
 
-        if self.data_parallel_size > 1:
+        if self.data_parallel_size_local > self.data_parallel_size:
+            raise ValueError(
+                f"data_parallel_size_local ({self.data_parallel_size_local}) "
+                f"must be <= data_parallel_size ({self.data_parallel_size})")
+
+        if self.data_parallel_size > 1 or self.data_parallel_size_local == 0:
             # Data parallel was specified in the engine args.
             self.data_parallel_master_port = get_open_port()
-            # TODO multi-node
         else:
             # Otherwise fall back to env vars (e.g. for offline SPMD case).
             self.data_parallel_size = envs.VLLM_DP_SIZE
@@ -1812,8 +1811,6 @@ def __post_init__(self) -> None:
             self.data_parallel_master_ip = envs.VLLM_DP_MASTER_IP
             self.data_parallel_master_port = envs.VLLM_DP_MASTER_PORT
 
-        self.world_size_across_dp = self.world_size * self.data_parallel_size
-
         if self.distributed_executor_backend == "external_launcher":
             import os
             os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"
diff --git a/vllm/distributed/utils.py b/vllm/distributed/utils.py
index e4d4008cd0a6..a8f292c6e31f 100644
--- a/vllm/distributed/utils.py
+++ b/vllm/distributed/utils.py
@@ -22,6 +22,7 @@
 
 import vllm.envs as envs
 from vllm.logger import init_logger
+from vllm.utils import get_tcp_uri
 
 logger = init_logger(__name__)
 
@@ -303,7 +304,7 @@ def stateless_init_torch_distributed_process_group(
     always formed with process 1, 2, ..., 8, and the additional communication
     channel is formed with process 9 and 10.
     """
-    init_method = f"tcp://{host}:{port}"
+    init_method = get_tcp_uri(host, port)
     backend = Backend(backend)  # it is basically string
     timeout = _get_default_timeout(backend)
 
diff --git a/vllm/engine/arg_utils.py b/vllm/engine/arg_utils.py
index bba05c4c3e1b..240142a1c5d1 100644
--- a/vllm/engine/arg_utils.py
+++ b/vllm/engine/arg_utils.py
@@ -283,6 +283,9 @@ class EngineArgs:
     pipeline_parallel_size: int = ParallelConfig.pipeline_parallel_size
     tensor_parallel_size: int = ParallelConfig.tensor_parallel_size
     data_parallel_size: int = ParallelConfig.data_parallel_size
+    data_parallel_size_local: Optional[int] = None
+    data_parallel_address: Optional[str] = None
+    data_parallel_rpc_port: Optional[int] = None
     enable_expert_parallel: bool = ParallelConfig.enable_expert_parallel
     max_parallel_loading_workers: Optional[
         int] = ParallelConfig.max_parallel_loading_workers
@@ -596,6 +599,21 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                                     **parallel_kwargs["tensor_parallel_size"])
         parallel_group.add_argument("--data-parallel-size", "-dp",
                                     **parallel_kwargs["data_parallel_size"])
+        parallel_group.add_argument('--data-parallel-size-local',
+                                    '-dpl',
+                                    type=int,
+                                    help='Number of data parallel replicas '
+                                    'to run on this node.')
+        parallel_group.add_argument('--data-parallel-address',
+                                    '-dpa',
+                                    type=str,
+                                    help='Address of data parallel cluster '
+                                    'head-node.')
+        parallel_group.add_argument('--data-parallel-rpc-port',
+                                    '-dpp',
+                                    type=int,
+                                    help='Port for data parallel RPC '
+                                    'communication.')
         parallel_group.add_argument(
             "--enable-expert-parallel",
             **parallel_kwargs["enable_expert_parallel"])
@@ -1019,10 +1037,30 @@ def create_engine_config(
             # but we should not do this here.
             placement_group = ray.util.get_current_placement_group()
 
+        # Local DP size defaults to global DP size if not set.
+        data_parallel_size_local = self.data_parallel_size if (
+            self.data_parallel_size_local
+            is None) else self.data_parallel_size_local
+
+        # DP address, used in multi-node case for torch distributed group
+        # and ZMQ sockets.
+        data_parallel_address = self.data_parallel_address if (
+            self.data_parallel_address
+            is not None) else ParallelConfig.data_parallel_master_ip
+
+        # This port is only used when there are remote data parallel engines,
+        # otherwise the local IPC transport is used.
+        data_parallel_rpc_port = self.data_parallel_rpc_port if (
+            self.data_parallel_rpc_port
+            is not None) else ParallelConfig.data_parallel_rpc_port
+
         parallel_config = ParallelConfig(
             pipeline_parallel_size=self.pipeline_parallel_size,
             tensor_parallel_size=self.tensor_parallel_size,
             data_parallel_size=self.data_parallel_size,
+            data_parallel_size_local=data_parallel_size_local,
+            data_parallel_master_ip=data_parallel_address,
+            data_parallel_rpc_port=data_parallel_rpc_port,
             enable_expert_parallel=self.enable_expert_parallel,
             max_parallel_loading_workers=self.max_parallel_loading_workers,
             disable_custom_all_reduce=self.disable_custom_all_reduce,
diff --git a/vllm/entrypoints/cli/serve.py b/vllm/entrypoints/cli/serve.py
index 5c8781b50d2c..04be7c033998 100644
--- a/vllm/entrypoints/cli/serve.py
+++ b/vllm/entrypoints/cli/serve.py
@@ -1,14 +1,24 @@
 # SPDX-License-Identifier: Apache-2.0
 
 import argparse
+import signal
 
 import uvloop
 
+import vllm.envs as envs
+from vllm import AsyncEngineArgs
 from vllm.entrypoints.cli.types import CLISubcommand
 from vllm.entrypoints.openai.api_server import run_server
 from vllm.entrypoints.openai.cli_args import (make_arg_parser,
                                               validate_parsed_serve_args)
-from vllm.utils import FlexibleArgumentParser
+from vllm.logger import init_logger
+from vllm.usage.usage_lib import UsageContext
+from vllm.utils import FlexibleArgumentParser, get_tcp_uri
+from vllm.v1.engine.core import EngineCoreProc
+from vllm.v1.engine.core_client import CoreEngineProcManager
+from vllm.v1.executor.abstract import Executor
+
+logger = init_logger(__name__)
 
 
 class ServeSubcommand(CLISubcommand):
@@ -24,7 +34,10 @@ def cmd(args: argparse.Namespace) -> None:
         if hasattr(args, 'model_tag') and args.model_tag is not None:
             args.model = args.model_tag
 
-        uvloop.run(run_server(args))
+        if args.headless:
+            run_headless(args)
+        else:
+            uvloop.run(run_server(args))
 
     def validate(self, args: argparse.Namespace) -> None:
         validate_parsed_serve_args(args)
@@ -42,6 +55,18 @@ def subparser_init(
                                   nargs='?',
                                   help="The model tag to serve "
                                   "(optional if specified in config)")
+        serve_parser.add_argument(
+            "--headless",
+            action='store_true',
+            default=False,
+            help="Run in headless mode. See multi-node data parallel "
+            "documentation for more details.")
+        serve_parser.add_argument(
+            '--data-parallel-start-rank',
+            '-dpr',
+            type=int,
+            default=0,
+            help='Starting data parallel rank for secondary nodes.')
         serve_parser.add_argument(
             "--config",
             type=str,
@@ -57,3 +82,55 @@ def subparser_init(
 
 def cmd_init() -> list[CLISubcommand]:
     return [ServeSubcommand()]
+
+
+def run_headless(args: argparse.Namespace):
+
+    # Create the EngineConfig.
+    engine_args = AsyncEngineArgs.from_cli_args(args)
+    usage_context = UsageContext.OPENAI_API_SERVER
+    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
+
+    if not envs.VLLM_USE_V1:
+        raise RuntimeError("Headless mode is only supported for V1")
+
+    parallel_config = vllm_config.parallel_config
+    local_engine_count = parallel_config.data_parallel_size_local
+    host = parallel_config.data_parallel_master_ip
+    port = engine_args.data_parallel_rpc_port  # add to config too
+    input_address = get_tcp_uri(host, port)
+
+    if local_engine_count <= 0:
+        raise RuntimeError("data_parallel_size_local must be > 0 in "
+                           "headless mode")
+
+    # Catch SIGTERM and SIGINT to allow graceful shutdown.
+    def signal_handler(signum, frame):
+        logger.debug("Received %d signal.", signum)
+        raise SystemExit
+
+    signal.signal(signal.SIGTERM, signal_handler)
+    signal.signal(signal.SIGINT, signal_handler)
+
+    logger.info(
+        "Launching %d data parallel engine(s) in headless mode, "
+        "with head node address %s.", local_engine_count, input_address)
+
+    # Create the engines.
+    engine_manager = CoreEngineProcManager(
+        target_fn=EngineCoreProc.run_engine_core,
+        local_engine_count=local_engine_count,
+        start_index=args.data_parallel_start_rank,
+        local_start_index=0,
+        vllm_config=vllm_config,
+        on_head_node=False,
+        input_address=input_address,
+        executor_class=Executor.get_class(vllm_config),
+        log_stats=not engine_args.disable_log_stats,
+    )
+
+    try:
+        engine_manager.join_first()
+    finally:
+        logger.info("Shutting down.")
+        engine_manager.close()
diff --git a/vllm/utils.py b/vllm/utils.py
index 59635a25eb32..9a7da8067ba4 100644
--- a/vllm/utils.py
+++ b/vllm/utils.py
@@ -613,6 +613,10 @@ def is_valid_ipv6_address(address: str) -> bool:
 
 
 def get_distributed_init_method(ip: str, port: int) -> str:
+    return get_tcp_uri(ip, port)
+
+
+def get_tcp_uri(ip: str, port: int) -> str:
     # Brackets are not permitted in ipv4 addresses,
     # see https://github.com/python/cpython/issues/103848
     return f"tcp://[{ip}]:{port}" if ":" in ip else f"tcp://{ip}:{port}"
diff --git a/vllm/v1/engine/core.py b/vllm/v1/engine/core.py
index bc410befbdad..edc79ae20b9f 100644
--- a/vllm/v1/engine/core.py
+++ b/vllm/v1/engine/core.py
@@ -1,5 +1,4 @@
 # SPDX-License-Identifier: Apache-2.0
-import json
 import os
 import queue
 import signal
@@ -23,7 +22,7 @@
 from vllm.lora.request import LoRARequest
 from vllm.transformers_utils.config import (
     maybe_register_config_serialize_by_value)
-from vllm.utils import resolve_obj_by_qualname, zmq_socket_ctx
+from vllm.utils import make_zmq_socket, resolve_obj_by_qualname, zmq_socket_ctx
 from vllm.v1.core.kv_cache_utils import (get_kv_cache_config,
                                          unify_kv_cache_configs)
 from vllm.v1.core.sched.interface import SchedulerInterface
@@ -43,6 +42,7 @@
 logger = init_logger(__name__)
 
 POLLING_TIMEOUT_S = 2.5
+HANDSHAKE_TIMEOUT_MINS = 5
 
 _R = TypeVar('_R')  # Return type for collective_rpc
 
@@ -348,9 +348,9 @@ class EngineCoreProc(EngineCore):
 
     def __init__(
         self,
-        input_path: str,
-        output_path: str,
         vllm_config: VllmConfig,
+        on_head_node: bool,
+        input_address: str,
         executor_class: type[Executor],
         log_stats: bool,
         engine_index: int = 0,
@@ -360,28 +360,91 @@ def __init__(
         executor_fail_callback = lambda: input_queue.put_nowait(
             (EngineCoreRequestType.EXECUTOR_FAILED, b''))
 
-        super().__init__(vllm_config, executor_class, log_stats,
-                         executor_fail_callback)
-
-        self.step_fn = (self.step if self.batch_queue is None else
-                        self.step_with_batch_queue)
-        self.engines_running = False
-
-        # Background Threads and Queues for IO. These enable us to
-        # overlap ZMQ socket IO with GPU since they release the GIL,
-        # and to overlap some serialization/deserialization with the
-        # model forward pass.
-        # Threads handle Socket <-> Queues and core_busy_loop uses Queue.
-        self.input_queue = input_queue
-        self.output_queue = queue.Queue[Union[EngineCoreOutputs, bytes]]()
-        threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
-                         daemon=True).start()
-        self.output_thread = threading.Thread(
-            target=self.process_output_socket,
-            args=(output_path, engine_index),
-            daemon=True)
-        self.output_thread.start()
+        # Create input socket.
+        input_ctx = zmq.Context()
+        identity = engine_index.to_bytes(length=2, byteorder="little")
+        input_socket = make_zmq_socket(input_ctx,
+                                       input_address,
+                                       zmq.DEALER,
+                                       identity=identity,
+                                       bind=False)
+        try:
+            # Register engine with front-end.
+            output_address = self.startup_handshake(
+                input_socket, on_head_node, vllm_config.parallel_config)
+
+            # Update config which may have changed from the handshake.
+            vllm_config.__post_init__()
+
+            # Set up data parallel environment.
+            self._init_data_parallel(vllm_config)
+
+            # Initialize engine core and model.
+            super().__init__(vllm_config, executor_class, log_stats,
+                             executor_fail_callback)
+
+            self.step_fn = (self.step if self.batch_queue is None else
+                            self.step_with_batch_queue)
+            self.engines_running = False
+
+            # Send ready message.
+            num_gpu_blocks = vllm_config.cache_config.num_gpu_blocks
+            input_socket.send(
+                msgspec.msgpack.encode({
+                    "status": "READY",
+                    "local": on_head_node,
+                    "num_gpu_blocks": num_gpu_blocks,
+                }))
+
+            # Background Threads and Queues for IO. These enable us to
+            # overlap ZMQ socket IO with GPU since they release the GIL,
+            # and to overlap some serialization/deserialization with the
+            # model forward pass.
+            # Threads handle Socket <-> Queues and core_busy_loop uses Queue.
+            self.input_queue = input_queue
+            self.output_queue = queue.Queue[Union[EngineCoreOutputs, bytes]]()
+            threading.Thread(target=self.process_input_socket,
+                             args=(input_socket, ),
+                             daemon=True).start()
+            input_socket = None
+            self.output_thread = threading.Thread(
+                target=self.process_output_socket,
+                args=(output_address, engine_index),
+                daemon=True)
+            self.output_thread.start()
+        finally:
+            if input_socket is not None:
+                input_socket.close(linger=0)
+
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        if not input_socket.poll(timeout=HANDSHAKE_TIMEOUT_MINS * 60 * 1000):
+            raise RuntimeError("Did not receive response from front-end "
+                               f"process within {HANDSHAKE_TIMEOUT_MINS} "
+                               f"minutes")
+        init_bytes = input_socket.recv()
+        init_message = msgspec.msgpack.decode(init_bytes)
+        logger.debug("Received init message: %s", init_message)
+
+        output_socket_address = init_message["output_socket_address"]
+        #TBD(nick) maybe replace IP with configured head node address
+
+        received_parallel_config = init_message["parallel_config"]
+        for key, value in received_parallel_config.items():
+            setattr(parallel_config, key, value)
+
+        return output_socket_address
 
     @staticmethod
     def run_engine_core(*args,
@@ -412,7 +475,7 @@ def signal_handler(signum, frame):
         try:
             parallel_config: ParallelConfig = kwargs[
                 "vllm_config"].parallel_config
-            if parallel_config.data_parallel_size > 1:
+            if parallel_config.data_parallel_size > 1 or dp_rank > 0:
                 # Set data parallel rank for this engine process.
                 parallel_config.data_parallel_rank = dp_rank
                 parallel_config.data_parallel_rank_local = local_dp_rank
@@ -436,6 +499,9 @@ def signal_handler(signum, frame):
             if engine_core is not None:
                 engine_core.shutdown()
 
+    def _init_data_parallel(self, vllm_config: VllmConfig):
+        pass
+
     def run_busy_loop(self):
         """Core busy loop of the EngineCore."""
 
@@ -527,40 +593,25 @@ def _send_engine_dead(self):
             logger.fatal("vLLM shutdown signal from EngineCore failed "
                          "to send. Please report this issue.")
 
-    def process_input_socket(self, input_path: str, engine_index: int):
+    def process_input_socket(self, input_socket: zmq.Socket):
         """Input socket IO thread."""
 
         # Msgpack serialization decoding.
         add_request_decoder = MsgpackDecoder(EngineCoreRequest)
         generic_decoder = MsgpackDecoder()
-        identity = engine_index.to_bytes(length=2, byteorder="little")
 
-        with zmq_socket_ctx(input_path,
-                            zmq.DEALER,
-                            identity=identity,
-                            bind=False) as socket:
-
-            # Send ready message to front-end once input socket is connected.
-            message_dict = {
-                'type': 'READY',
-                'num_gpu_blocks': self.vllm_config.cache_config.num_gpu_blocks,
-            }
-            message = json.dumps(message_dict).encode('utf-8')
-            socket.send(message)
-
-            while True:
-                # (RequestType, RequestData)
-                type_frame, *data_frames = socket.recv_multipart(copy=False)
-                request_type = EngineCoreRequestType(bytes(type_frame.buffer))
+        while True:
+            # (RequestType, RequestData)
+            type_frame, *data_frames = input_socket.recv_multipart(copy=False)
+            request_type = EngineCoreRequestType(bytes(type_frame.buffer))
 
-                # Deserialize the request data.
-                decoder = add_request_decoder if (
-                    request_type
-                    == EngineCoreRequestType.ADD) else generic_decoder
-                request = decoder.decode(data_frames)
+            # Deserialize the request data.
+            decoder = add_request_decoder if (
+                request_type == EngineCoreRequestType.ADD) else generic_decoder
+            request = decoder.decode(data_frames)
 
-                # Push to input queue for core busy loop.
-                self.input_queue.put_nowait((request_type, request))
+            # Push to input queue for core busy loop.
+            self.input_queue.put_nowait((request_type, request))
 
     def process_output_socket(self, output_path: str, engine_index: int):
         """Output socket IO thread."""
@@ -609,9 +660,9 @@ class DPEngineCoreProc(EngineCoreProc):
 
     def __init__(
         self,
-        input_path: str,
-        output_path: str,
         vllm_config: VllmConfig,
+        on_head_node: bool,
+        input_address: str,
         executor_class: type[Executor],
         log_stats: bool,
     ):
@@ -623,8 +674,20 @@ def __init__(
         _add_prefix(sys.stdout, process_name, pid)
         _add_prefix(sys.stderr, process_name, pid)
 
-        dp_size = vllm_config.parallel_config.data_parallel_size
+        # Counts forward-passes of the model so that we can synchronize
+        # finished with DP peers every N steps.
+        self.counter = 0
+
+        # Initialize the engine.
+        dp_rank = vllm_config.parallel_config.data_parallel_rank
+        super().__init__(vllm_config, on_head_node, input_address,
+                         executor_class, log_stats, dp_rank)
+
+    def _init_data_parallel(self, vllm_config: VllmConfig):
+
+        # Configure GPUs and stateless process group for data parallel.
         dp_rank = vllm_config.parallel_config.data_parallel_rank
+        dp_size = vllm_config.parallel_config.data_parallel_size
         local_dp_rank = vllm_config.parallel_config.data_parallel_rank_local
 
         assert dp_size > 1
@@ -632,24 +695,16 @@ def __init__(
 
         from vllm.platforms import current_platform
         device_control_env_var = current_platform.device_control_env_var
-        tp_size = vllm_config.parallel_config.tensor_parallel_size
+        world_size = vllm_config.parallel_config.world_size
         os.environ[device_control_env_var] = ",".join(
             str(current_platform.device_id_to_physical_device_id(i))
-            for i in range(local_dp_rank * tp_size, (local_dp_rank + 1) *
-                           tp_size))
+            for i in range(local_dp_rank * world_size, (local_dp_rank + 1) *
+                           world_size))
 
         self.local_dp_rank = local_dp_rank
         self.dp_group = vllm_config.parallel_config.stateless_init_dp_group()
         self.current_wave = 0
 
-        # Initialize the engine after setting up environment.
-        super().__init__(input_path, output_path, vllm_config, executor_class,
-                         log_stats, dp_rank)
-
-        # Counts forward-passes of the model so that we can synchronize
-        # finished with DP peers every N steps.
-        self.counter = 0
-
     def shutdown(self):
         super().shutdown()
         if dp_group := getattr(self, "dp_group", None):
diff --git a/vllm/v1/engine/core_client.py b/vllm/v1/engine/core_client.py
index c33317edcbb0..0d52bc9a6814 100644
--- a/vllm/v1/engine/core_client.py
+++ b/vllm/v1/engine/core_client.py
@@ -1,7 +1,6 @@
 # SPDX-License-Identifier: Apache-2.0
 import asyncio
 import contextlib
-import json
 import queue
 import uuid
 import weakref
@@ -9,25 +8,27 @@
 from collections import deque
 from collections.abc import Awaitable, Sequence
 from concurrent.futures import Future
-from dataclasses import dataclass, field
+from dataclasses import dataclass
+from enum import Enum, auto
 from threading import Thread
 from typing import Any, Callable, Optional, TypeVar, Union
 
+import msgspec
 import zmq
 import zmq.asyncio
 
-from vllm.config import VllmConfig
+from vllm.config import ParallelConfig, VllmConfig
 from vllm.logger import init_logger
 from vllm.lora.request import LoRARequest
-from vllm.utils import (get_open_zmq_inproc_path, get_open_zmq_ipc_path,
-                        make_zmq_socket)
+from vllm.utils import (get_open_port, get_open_zmq_inproc_path,
+                        get_open_zmq_ipc_path, get_tcp_uri, make_zmq_socket)
 from vllm.v1.engine import (EngineCoreOutputs, EngineCoreRequest,
                             EngineCoreRequestType, UtilityOutput)
 from vllm.v1.engine.core import EngineCore, EngineCoreProc
 from vllm.v1.engine.exceptions import EngineDeadError
 from vllm.v1.executor.abstract import Executor
 from vllm.v1.serial_utils import MsgpackDecoder, MsgpackEncoder, bytestr
-from vllm.v1.utils import BackgroundProcHandle
+from vllm.v1.utils import CoreEngineProcManager
 
 logger = init_logger(__name__)
 
@@ -264,45 +265,22 @@ def collective_rpc(self,
         return self.engine_core.collective_rpc(method, timeout, args, kwargs)
 
 
+class CoreEngineState(Enum):
+    NEW = auto()
+    CONNECTED = auto()
+    READY = auto()
+
+
 class CoreEngine:
     """One per data parallel rank."""
 
-    def __init__(
-        self,
-        vllm_config: VllmConfig,
-        executor_class: type[Executor],
-        log_stats: bool,
-        input_path: str,
-        output_path: str,
-        index: int = 0,
-        local_dp_rank: int = 0,
-    ):
+    def __init__(self, index: int = 0, local: bool = True):
+        self.local = local
         self.index = index
         self.identity = index.to_bytes(length=2, byteorder="little")
-        try:
-            # Start EngineCore in background process.
-            self.proc_handle = BackgroundProcHandle(
-                input_path=input_path,
-                output_path=output_path,
-                process_name=f"EngineCore_{index}",
-                target_fn=EngineCoreProc.run_engine_core,
-                process_kwargs={
-                    "vllm_config": vllm_config,
-                    "dp_rank": index,
-                    "local_dp_rank": local_dp_rank,
-                    "executor_class": executor_class,
-                    "log_stats": log_stats,
-                })
-
-            self.num_reqs_in_flight = 0
-        finally:
-            if not hasattr(self, "num_reqs_in_flight"):
-                # Ensure socket is closed if process fails to start.
-                self.close()
 
-    def close(self):
-        if proc_handle := getattr(self, "proc_handle", None):
-            proc_handle.shutdown()
+        self.state = CoreEngineState.NEW
+        self.num_reqs_in_flight = 0
 
 
 @dataclass
@@ -311,7 +289,7 @@ class BackgroundResources:
     circular reference back to the client object."""
 
     ctx: Union[zmq.Context]
-    core_engines: list[CoreEngine] = field(default_factory=list)
+    local_engine_manager: Optional[CoreEngineProcManager] = None
     output_socket: Optional[Union[zmq.Socket, zmq.asyncio.Socket]] = None
     input_socket: Optional[Union[zmq.Socket, zmq.asyncio.Socket]] = None
     output_queue_task: Optional[asyncio.Task] = None
@@ -325,8 +303,8 @@ def __call__(self):
         """Clean up background resources."""
 
         self.engine_dead = True
-        for core_engine in self.core_engines:
-            core_engine.close()
+        if self.local_engine_manager is not None:
+            self.local_engine_manager.close()
 
         if self.output_queue_task is not None:
             self.output_queue_task.cancel()
@@ -388,25 +366,56 @@ def __init__(
         self._finalizer = weakref.finalize(self, self.resources)
         success = False
         try:
-            # Paths and sockets for IPC.
-            self.output_path = get_open_zmq_ipc_path()
-            input_path = get_open_zmq_ipc_path()
-            self.input_socket = make_zmq_socket(self.ctx,
-                                                input_path,
-                                                zmq.ROUTER,
-                                                bind=True)
-            self.resources.input_socket = self.input_socket
-
-            new_core_engine = lambda index, local_dp_rank=None: CoreEngine(
-                vllm_config, executor_class, log_stats, input_path, self.
-                output_path, index, local_dp_rank)
-
-            # Start engine core process(es).
-            self._init_core_engines(vllm_config, new_core_engine,
-                                    self.resources.core_engines)
+            parallel_config = vllm_config.parallel_config
+            local_engine_count = parallel_config.data_parallel_size_local
+            start_index = parallel_config.data_parallel_rank
+            local_start_index = parallel_config.data_parallel_rank_local
+
+            # SPMD mode is where there is an LLM instance per DP rank and
+            # one core engine per LLM, see
+            # examples/offline_inference/data_parallel.py.
+            spmd_mode = local_start_index is not None
+            if spmd_mode:
+                assert local_engine_count == 1
+                self.core_engines = [
+                    CoreEngine(index=local_start_index, local=True)
+                ]
+            else:
+                assert start_index == 0
+                local_start_index = 0
+                self.core_engines = [
+                    CoreEngine(index=i, local=(i < local_engine_count))
+                    for i in range(parallel_config.data_parallel_size)
+                ]
+
+            input_address, output_address = self._get_zmq_addresses(
+                parallel_config, spmd_mode)
+
+            # Create input and output sockets.
+            self.input_socket = self.resources.input_socket = make_zmq_socket(
+                self.ctx, input_address, zmq.ROUTER, bind=True)
+
+            self.resources.output_socket = make_zmq_socket(
+                self.ctx, output_address, zmq.constants.PULL)
+            # Start local engines.
+            if local_engine_count:
+                # In server mode, start_index and local_start_index will
+                # both be 0.
+                self.resources.local_engine_manager = CoreEngineProcManager(
+                    EngineCoreProc.run_engine_core,
+                    vllm_config=vllm_config,
+                    executor_class=executor_class,
+                    log_stats=log_stats,
+                    input_address=input_address,
+                    on_head_node=True,
+                    local_engine_count=local_engine_count,
+                    start_index=start_index,
+                    local_start_index=local_start_index)
+
+            self.core_engine = self.core_engines[0]
 
             # Wait for engine core process(es) to start.
-            self._wait_for_engine_startup()
+            self._wait_for_engine_startup(output_address, parallel_config)
 
             self.utility_results: dict[int, AnyFuture] = {}
 
@@ -420,56 +429,116 @@ def __init__(
             if not success:
                 self._finalizer()
 
-    def _wait_for_engine_startup(self):
+    @staticmethod
+    def _get_zmq_addresses(parallel_config: ParallelConfig,
+                           spmd_mode: bool) -> tuple[str, str]:
+        """Returns (input_address, output_address)."""
+        dp_size = parallel_config.data_parallel_size
+        local_engine_count = parallel_config.data_parallel_size_local
+
+        if local_engine_count == dp_size or spmd_mode:
+            input_address = get_open_zmq_ipc_path()
+            output_address = get_open_zmq_ipc_path()
+        else:
+            host = parallel_config.data_parallel_master_ip
+            input_port = parallel_config.data_parallel_rpc_port
+            output_port = get_open_port()
+            input_address = get_tcp_uri(host, input_port)
+            output_address = get_tcp_uri(host, output_port)
+
+        return input_address, output_address
+
+    def _wait_for_engine_startup(self, output_address: str,
+                                 parallel_config: ParallelConfig):
         # Get a sync handle to the socket which can be sync or async.
         sync_input_socket = zmq.Socket.shadow(self.input_socket)
 
         # Wait for engine core process(es) to send ready messages.
-        identities = set(eng.index for eng in self.resources.core_engines)
+        local_count = parallel_config.data_parallel_size_local
+        remote_count = len(self.core_engines) - local_count
+        # [local, remote] counts
+        conn_pending, start_pending = [local_count, remote_count], [0, 0]
+
         poller = zmq.Poller()
         poller.register(sync_input_socket, zmq.POLLIN)
-        for eng in self.resources.core_engines:
-            poller.register(eng.proc_handle, zmq.POLLIN)
-        while identities:
+        proc_manager = self.resources.local_engine_manager
+        if proc_manager is not None:
+            for sentinel in proc_manager.sentinels():
+                poller.register(sentinel, zmq.POLLIN)
+        while any(conn_pending) or any(start_pending):
             events = poller.poll(STARTUP_POLL_PERIOD_MS)
             if not events:
-                logger.debug("Waiting for %d core engine proc(s) to start: %s",
-                             len(identities), identities)
+                if any(conn_pending):
+                    logger.debug(
+                        "Waiting for %d local, %d remote core engine proc(s) "
+                        "to connect.", *conn_pending)
+                if any(start_pending):
+                    logger.debug(
+                        "Waiting for %d local, %d remote core engine proc(s) "
+                        "to start.", *start_pending)
                 continue
             if len(events) > 1 or events[0][0] != sync_input_socket:
-                # One of the core processes exited.
+                # One of the local core processes exited.
+                finished = proc_manager.finished_procs(
+                ) if proc_manager else {}
                 raise RuntimeError("Engine core initialization failed. "
-                                   "See root cause above.")
-
-            eng_id_bytes, data = sync_input_socket.recv_multipart()
-            eng_id = int.from_bytes(eng_id_bytes, byteorder="little")
-            if eng_id not in identities:
-                raise RuntimeError(f"Unexpected or duplicate engine: {eng_id}")
-            message_dict = json.loads(data.decode('utf-8'))
-            if message_dict['type'] != 'READY':
-                raise RuntimeError(f"Engine {eng_id} failed: {data.decode()}")
-            logger.info("Core engine process %d ready.", eng_id)
-            identities.discard(eng_id)
-            # Setup KV cache config with initialization state from
-            # engine core process. Sum values from all engines in DP case.
-            num_gpu_blocks = self.vllm_config.cache_config.num_gpu_blocks or 0
-            num_gpu_blocks += message_dict['num_gpu_blocks']
-            self.vllm_config.cache_config.num_gpu_blocks = num_gpu_blocks
-
-    def _init_core_engines(
-        self,
-        vllm_config: VllmConfig,
-        new_core_engine: Callable[[int, Optional[int]], CoreEngine],
-        core_engines: list[CoreEngine],
-    ) -> None:
-
-        # Default case - single core engine.
-        core_engine = new_core_engine(
-            vllm_config.parallel_config.data_parallel_rank,
-            vllm_config.parallel_config.data_parallel_rank_local,
-        )
-        core_engines.append(core_engine)
-        self.core_engine = core_engine
+                                   "See root cause above. "
+                                   f"Failed core proc(s): {finished}")
+
+            # Receive HELLO and READY messages from the input socket.
+            eng_identity, ready_msg_bytes = sync_input_socket.recv_multipart()
+            eng_index = int.from_bytes(eng_identity, byteorder="little")
+            engine = next(
+                (e for e in self.core_engines if e.identity == eng_identity),
+                None)
+            if engine is None:
+                raise RuntimeError(f"Message from engine with unexpected data "
+                                   f"parallel rank: {eng_index}")
+            msg = msgspec.msgpack.decode(ready_msg_bytes)
+            status, local = msg["status"], msg["local"]
+            if local != engine.local:
+                raise RuntimeError(f"{status} message from "
+                                   f"{'local' if local else 'remote'} "
+                                   f"engine {eng_index}, expected it to be "
+                                   f"{'local' if engine.local else 'remote'}")
+
+            if status == "HELLO" and engine.state == CoreEngineState.NEW:
+
+                # Send init message with DP config info.
+                init_message = self.encoder.encode({
+                    "output_socket_address": output_address,
+                    "parallel_config": {
+                        "data_parallel_master_ip":
+                        parallel_config.data_parallel_master_ip,
+                        "data_parallel_master_port":
+                        parallel_config.data_parallel_master_port,
+                        "data_parallel_size":
+                        parallel_config.data_parallel_size,
+                    },
+                })
+                sync_input_socket.send_multipart((eng_identity, *init_message),
+                                                 copy=False)
+                conn_pending[0 if local else 1] -= 1
+                start_pending[0 if local else 1] += 1
+                engine.state = CoreEngineState.CONNECTED
+            elif status == "READY" and (engine.state
+                                        == CoreEngineState.CONNECTED):
+                # Setup KV cache config with initialization state from
+                # engine core process. Sum values from all engines in DP case.
+                cache_config = self.vllm_config.cache_config
+                num_gpu_blocks = cache_config.num_gpu_blocks or 0
+                num_gpu_blocks += msg['num_gpu_blocks']
+                cache_config.num_gpu_blocks = num_gpu_blocks
+
+                start_pending[0 if local else 1] -= 1
+                engine.state = CoreEngineState.READY
+            else:
+                raise RuntimeError(f"Unexpected {status} message for "
+                                   f"{'local' if local else 'remote'} engine "
+                                   f"{eng_index} in {engine.state} state.")
+
+            logger.debug("%s from %s core engine process %s.", status,
+                         "local" if local else "remote", eng_index)
 
     def shutdown(self):
         # Terminate background resources.
@@ -520,7 +589,8 @@ def __init__(self, vllm_config: VllmConfig, executor_class: type[Executor],
         # Ensure that the outputs socket processing thread does not have
         # a ref to the client which prevents gc.
         ctx = self.ctx
-        output_path = self.output_path
+        out_socket = self.resources.output_socket
+        assert out_socket is not None
         decoder = self.decoder
         utility_results = self.utility_results
         outputs_queue = self.outputs_queue
@@ -531,7 +601,6 @@ def __init__(self, vllm_config: VllmConfig, executor_class: type[Executor],
 
         def process_outputs_socket():
             shutdown_socket = ctx.socket(zmq.PAIR)
-            out_socket = make_zmq_socket(ctx, output_path, zmq.constants.PULL)
             try:
                 shutdown_socket.bind(shutdown_path)
                 poller = zmq.Poller()
@@ -566,6 +635,9 @@ def process_outputs_socket():
                                           daemon=True)
         self.output_queue_thread.start()
 
+        # The thread takes on responsibility for closing the socket.
+        self.resources.output_socket = None
+
     def get_output(self) -> EngineCoreOutputs:
         # If an exception arises in process_outputs_socket task,
         # it is forwarded to the outputs_queue so we can raise it
@@ -693,10 +765,8 @@ def _ensure_output_queue_task(self):
                                               self.__class__,
                                               "process_engine_outputs", None)
         _self_ref = weakref.ref(self) if output_handler else None
-        output_path = self.output_path
-        output_socket = make_zmq_socket(self.ctx, output_path,
-                                        zmq.constants.PULL)
-        resources.output_socket = output_socket
+        output_socket = resources.output_socket
+        assert output_socket is not None
 
         async def process_outputs_socket():
             try:
@@ -861,21 +931,6 @@ def __init__(self, vllm_config: VllmConfig, executor_class: type[Executor],
 
         assert len(self.core_engines) > 1
 
-    def _init_core_engines(
-        self,
-        vllm_config: VllmConfig,
-        new_core_engine: Callable[[int, Optional[int]], CoreEngine],
-        core_engines: list[CoreEngine],
-    ) -> None:
-
-        # Launch a core engine for each data parallel rank.
-        dp_size = vllm_config.parallel_config.data_parallel_size
-        for i in range(dp_size):
-            # Multi-node not yet supported so local_dp_rank == dp_rank.
-            core_engines.append(new_core_engine(i, i))
-
-        self.core_engines = core_engines
-
     async def call_utility_async(self, method: str, *args) -> Any:
         # Only the result from the first engine is returned.
         return (await asyncio.gather(*[
diff --git a/vllm/v1/utils.py b/vllm/v1/utils.py
index 9c238c3aad8e..0758747a83cc 100644
--- a/vllm/v1/utils.py
+++ b/vllm/v1/utils.py
@@ -1,20 +1,23 @@
 # SPDX-License-Identifier: Apache-2.0
 
 import os
+import time
 import weakref
 from collections import defaultdict
 from collections.abc import Sequence
-from multiprocessing import Process
-from typing import (TYPE_CHECKING, Any, Callable, Generic, Optional, TypeVar,
-                    Union, overload)
+from multiprocessing import Process, connection
+from typing import (TYPE_CHECKING, Callable, Generic, Optional, TypeVar, Union,
+                    overload)
 
 import torch
 
+from vllm.config import VllmConfig
 from vllm.logger import init_logger
 from vllm.model_executor.models.utils import extract_layer_index
 from vllm.usage.usage_lib import (UsageContext, is_usage_stats_enabled,
                                   usage_message)
 from vllm.utils import get_mp_context, kill_process_tree
+from vllm.v1.executor.abstract import Executor
 
 if TYPE_CHECKING:
     from vllm.attention.layer import Attention
@@ -92,7 +95,7 @@ def __repr__(self):
         return f"ConstantList({self._x})"
 
 
-class BackgroundProcHandle:
+class CoreEngineProcManager:
     """
     Utility class to handle creation, readiness, and shutdown
     of background processes used by the AsyncLLM and LLMEngine.
@@ -100,49 +103,91 @@ class BackgroundProcHandle:
 
     def __init__(
         self,
-        input_path: str,
-        output_path: str,
-        process_name: str,
         target_fn: Callable,
-        process_kwargs: dict[Any, Any],
+        local_engine_count: int,
+        start_index: int,
+        local_start_index: int,
+        vllm_config: VllmConfig,
+        on_head_node: bool,
+        input_address: str,
+        executor_class: type[Executor],
+        log_stats: bool,
     ):
         context = get_mp_context()
+        common_kwargs = {
+            "vllm_config": vllm_config,
+            "on_head_node": on_head_node,
+            "input_address": input_address,
+            "executor_class": executor_class,
+            "log_stats": log_stats,
+        }
+
+        self.processes: list[Process] = []
+        for index in range(local_engine_count):
+            local_index = local_start_index + index
+            global_index = start_index + index
+            # Start EngineCore in background process.
+            self.processes.append(
+                context.Process(target=target_fn,
+                                name=f"EngineCore_{global_index}",
+                                kwargs=common_kwargs | {
+                                    "dp_rank": global_index,
+                                    "local_dp_rank": local_index,
+                                }))
+
+        self._finalizer = weakref.finalize(self, shutdown, self.processes,
+                                           input_address)
+        try:
+            for proc in self.processes:
+                proc.start()
+        finally:
+            # Kill other procs if not all are running.
+            if self.finished_procs():
+                self.close()
+
+    def close(self):
+        """Shutdown all procs."""
+        self._finalizer()
 
-        assert ("input_path" not in process_kwargs
-                and "output_path" not in process_kwargs)
-        process_kwargs["input_path"] = input_path
-        process_kwargs["output_path"] = output_path
-
-        # Run busy loop in background process.
-        self.proc: Process = context.Process(target=target_fn,
-                                             kwargs=process_kwargs,
-                                             name=process_name)
-        self._finalizer = weakref.finalize(self, shutdown, self.proc,
-                                           input_path, output_path)
-        self.proc.start()
+    def join_first(self):
+        """Wait for any process to exit."""
+        connection.wait(proc.sentinel for proc in self.processes)
 
-    def fileno(self):
-        return self.proc.sentinel
+    def sentinels(self) -> list:
+        return [proc.sentinel for proc in self.processes]
 
-    def shutdown(self):
-        self._finalizer()
+    def finished_procs(self) -> dict[str, int]:
+        """Returns dict of proc name -> exit code for any finished procs."""
+        return {
+            proc.name: proc.exitcode
+            for proc in self.processes if proc.exitcode is not None
+        }
 
 
 # Note(rob): shutdown function cannot be a bound method,
-# else the gc cannot collect the object.
-def shutdown(proc: Process, input_path: str, output_path: str):
+# else the gc cannot collect the objedecoupct.
+def shutdown(procs: list[Process], input_address: str):
     # Shutdown the process.
-    if proc.is_alive():
-        proc.terminate()
-        proc.join(5)
-
+    for proc in procs:
+        if proc.is_alive():
+            proc.terminate()
+
+    # Allow 5 seconds for remaining procs to terminate.
+    deadline = time.monotonic() + 5
+    for proc in procs:
+        remaining = deadline - time.monotonic()
+        if remaining <= 0:
+            break
+        if proc.is_alive():
+            proc.join(remaining)
+
+    for proc in procs:
         if proc.is_alive() and (pid := proc.pid) is not None:
             kill_process_tree(pid)
 
     # Remove zmq ipc socket files.
-    ipc_sockets = [output_path, input_path]
-    for ipc_socket in ipc_sockets:
-        socket_file = ipc_socket.replace("ipc://", "")
+    if input_address.startswith("ipc://"):
+        socket_file = input_address[len("ipc://"):]
         if os and os.path.exists(socket_file):
             os.remove(socket_file)
 


Reviews:
user: login: youkaichao
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T16:51:44Z
user: login: youkaichao
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T16:52:35Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T23:59:25Z
user: login: afeldman-nm
body: Hi @njhill  , I'm probably going to spend a little more time reviewing this but I left some initial nits.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T21:24:57Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T21:29:41Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T21:30:34Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T21:31:08Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T21:32:00Z
user: login: afeldman-nm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T21:34:42Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T21:35:09Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T21:37:32Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T22:48:57Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T23:06:46Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T23:11:19Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T23:17:45Z
user: login: afeldman-nm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T20:54:57Z
user: login: afeldman-nm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T20:55:27Z
user: login: russellb
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-11T15:25:51Z
user: login: russellb
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T17:59:20Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T19:21:37Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR decouples engine process management from IPC and adds support for running a mix of local and remote engines, including a new headless mode for secondary nodes. Key changes include:
- Refactoring of the background process management (renaming BackgroundProcHandle to CoreEngineProcManager and updating its process lifecycle management logic)
- Updates to engine core initialization and handshake protocols to support both local and remote engines
- Introduction of new CLI arguments and configuration updates for headless mode and multi-node data parallel setups

### Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

<details>
<summary>Show a summary per file</summary>

| File                              | Description                                                              |
| --------------------------------- | ------------------------------------------------------------------------ |
| vllm/v1/utils.py                  | Refactored process manager and shutdown behavior                         |
| vllm/v1/engine/core_client.py      | Updated engine core initialization and handshake state management         |
| vllm/v1/engine/core.py             | Revised engine core handshake, process startup, and data parallel setup     |
| vllm/utils.py                     | Added get_tcp_uri utility for constructing TCP URIs                       |
| vllm/entrypoints/cli/serve.py      | Added headless mode along with new CLI arguments and engine manager usage   |
| vllm/engine/arg_utils.py           | Introduced new CLI arguments supporting local data parallel configuration   |
| vllm/config.py                    | Updated ParallelConfig with new fields and validation for local DP settings  |
</details>



<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**vllm/v1/utils.py:143**
* [nitpick] The current check for finished processes in the process manager's initialization may trigger a shutdown if any process exits unexpectedly early. Please verify that this behavior aligns with the intended process management strategy.
```
if self.finished_procs():
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:37:43Z
user: login: simon-mo
body: Great work. I have mostly done a code level review. The design make sense to me, especially given this is an intermediate state. I would imagine the scale out PR will enhance the design quite a bit. 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-17T23:49:09Z
user: login: youkaichao
body: approve to unblock, given this is an intermediate state.

can you please leave some doc/tutorial on how to use it?
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-18T12:40:39Z
user: login: russellb
body: We've talked about this some offline, but for the record, I would prefer that we don't introduce any additional multi-node interfaces that are not secured in any way, and worse, allow arbitrary code execution via `pickle`. I understand the argument that PyTorch says its distributed communications are insecure, but that's not a great reason to make our own code worse and harder to fix in the future.
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-04-29T20:58:41Z
user: login: ruisearch42
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T21:41:25Z
user: login: njhill
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T17:38:21Z

Review comments:
diff_hunk: @@ -434,6 +438,29 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                             'MoE layers will be sharded according to the '
                             'product of the tensor-parallel-size and '
                             'data-parallel-size.')
+        parser.add_argument('--data-parallel-size-local',
+                            '-dpl',
+                            type=int,
+                            default=EngineArgs.data_parallel_size_local,
+                            help='Number of data parallel replicas to run on '
+                            'this node.')
+        parser.add_argument('--data-parallel-start-rank',
+                            '-dpr',
+                            type=int,
+                            default=EngineArgs.data_parallel_start_rank,
+                            help='Starting data parallel rank for secondary '
+                            'nodes.')
+        parser.add_argument('--data-parallel-address',
path: vllm/engine/arg_utils.py
user: login: youkaichao
body: we have `data_parallel_master_ip` and `data_parallel_master_port`. can we use those two field and rename the cli arg? i feel `data_parallel_master_ip` and `data_parallel_master_port` are easier to understand compared to `--data-parallel-address` and `--data-parallel-rpc-port`.

in addition, I reserved 10 ports starting from `data_parallel_master_port`, so you can use, say `data_parallel_master_port + 2`, as `--data-parallel-rpc-port`, which will make the user-interface easier to use.

see https://github.com/vllm-project/vllm/blob/2386803f2a2e3df1f29ea05212eaf68590b85805/vllm/utils.py#L581 
created_at: 2025-04-04T16:51:44Z
author_association: MEMBER
diff_hunk: @@ -434,6 +438,29 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                             'MoE layers will be sharded according to the '
                             'product of the tensor-parallel-size and '
                             'data-parallel-size.')
+        parser.add_argument('--data-parallel-size-local',
+                            '-dpl',
+                            type=int,
+                            default=EngineArgs.data_parallel_size_local,
+                            help='Number of data parallel replicas to run on '
+                            'this node.')
+        parser.add_argument('--data-parallel-start-rank',
+                            '-dpr',
+                            type=int,
+                            default=EngineArgs.data_parallel_start_rank,
+                            help='Starting data parallel rank for secondary '
+                            'nodes.')
path: vllm/engine/arg_utils.py
user: login: youkaichao
body: these two args should go to `vllm/entrypoints/cli/serve.py` since they are for online serving only.
created_at: 2025-04-04T16:52:34Z
author_association: MEMBER
diff_hunk: @@ -434,6 +438,29 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                             'MoE layers will be sharded according to the '
                             'product of the tensor-parallel-size and '
                             'data-parallel-size.')
+        parser.add_argument('--data-parallel-size-local',
+                            '-dpl',
+                            type=int,
+                            default=EngineArgs.data_parallel_size_local,
+                            help='Number of data parallel replicas to run on '
+                            'this node.')
+        parser.add_argument('--data-parallel-start-rank',
+                            '-dpr',
+                            type=int,
+                            default=EngineArgs.data_parallel_start_rank,
+                            help='Starting data parallel rank for secondary '
+                            'nodes.')
+        parser.add_argument('--data-parallel-address',
path: vllm/engine/arg_utils.py
user: login: njhill
body: I could rename `data-parallel-address` to `data-parallel-ip-address`? or `data-parallel-head-ip`? I was also trying to avoid using the "master" term :)

Re the port, the specified port in this case is actually the zmq socket port, the torch process group is assigned an available port arbitrarily (which is propagated to the other nodes).
created_at: 2025-04-04T23:59:25Z
author_association: MEMBER
diff_hunk: @@ -309,20 +309,44 @@ class EngineCoreProc(EngineCore):
 
     def __init__(
         self,
-        input_path: str,
-        output_path: str,
         vllm_config: VllmConfig,
+        on_head_node: bool,
+        input_address: str,
         executor_class: type[Executor],
         log_stats: bool,
         engine_index: int = 0,
     ):
+        # Create input socket.
+        input_ctx = zmq.Context()  # type: ignore[attr-defined]
+        identity = engine_index.to_bytes(length=2, byteorder="little")
+        input_socket = make_zmq_socket(input_ctx,
+                                       input_address,
+                                       zmq.DEALER,
+                                       identity=identity,
+                                       bind=False)
+
+        # Register engine with front-end.
+        output_address = self.startup_handshake(input_socket, on_head_node,
+                                                vllm_config.parallel_config)
+
+        # Set up data parallel environment.
+        self._init_data_parallel(vllm_config)
+
+        # Initialize engine core and model.
         super().__init__(vllm_config, executor_class, log_stats)
 
         self.step_fn = (self.step if self.batch_queue is None else
                         self.step_with_batch_queue)
 
         self.global_unfinished_reqs = False
 
+        # Send ready message.
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: assigning `on_head_node` to the `local` field suggests that `local` has an absolute meaning, i.e. "on the head node" as opposed to on any other node. However, I thought that "local" had a relative meaning, i.e. "on the same node as a particular engine core instance"?
created_at: 2025-04-07T18:42:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -309,20 +309,44 @@ class EngineCoreProc(EngineCore):
 
     def __init__(
         self,
-        input_path: str,
-        output_path: str,
         vllm_config: VllmConfig,
+        on_head_node: bool,
+        input_address: str,
         executor_class: type[Executor],
         log_stats: bool,
         engine_index: int = 0,
     ):
+        # Create input socket.
+        input_ctx = zmq.Context()  # type: ignore[attr-defined]
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: nit: jw is there a reason we need to ignore type here?
created_at: 2025-04-07T18:48:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -332,12 +356,39 @@ def __init__(
                                             Any]] = queue.Queue()
         self.output_queue: queue.Queue[EngineCoreOutputs] = queue.Queue()
         threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
+                         args=(input_socket, ),
                          daemon=True).start()
         threading.Thread(target=self.process_output_socket,
-                         args=(output_path, engine_index),
+                         args=(output_address, engine_index),
                          daemon=True).start()
 
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        input_socket.poll(timeout=5 * 60 * 1000)
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: Do we need error handling for the timeout scenario?
created_at: 2025-04-07T20:03:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -332,12 +356,39 @@ def __init__(
                                             Any]] = queue.Queue()
         self.output_queue: queue.Queue[EngineCoreOutputs] = queue.Queue()
         threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
+                         args=(input_socket, ),
                          daemon=True).start()
         threading.Thread(target=self.process_output_socket,
-                         args=(output_path, engine_index),
+                         args=(output_address, engine_index),
                          daemon=True).start()
 
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        input_socket.poll(timeout=5 * 60 * 1000)
+        init_bytes = input_socket.recv()
+        init_message = msgspec.msgpack.decode(init_bytes)
+        logger.debug("Received init message: %s", init_message)
+
+        output_socket_address = init_message["output_socket_address"]
+        #TBD maybe replace IP with configured head node address
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: Nit: not sure what the rule is that we follow, but should we put someone's name on this TODO? i.e. `TODO (nick)` or someone else
created_at: 2025-04-07T20:10:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -332,12 +356,39 @@ def __init__(
                                             Any]] = queue.Queue()
         self.output_queue: queue.Queue[EngineCoreOutputs] = queue.Queue()
         threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
+                         args=(input_socket, ),
                          daemon=True).start()
         threading.Thread(target=self.process_output_socket,
-                         args=(output_path, engine_index),
+                         args=(output_address, engine_index),
                          daemon=True).start()
 
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        input_socket.poll(timeout=5 * 60 * 1000)
+        init_bytes = input_socket.recv()
+        init_message = msgspec.msgpack.decode(init_bytes)
+        logger.debug("Received init message: %s", init_message)
+
+        output_socket_address = init_message["output_socket_address"]
+        #TBD maybe replace IP with configured head node address
+
+        received_parallel_config = init_message["parallel_config"]
+        for key, value in received_parallel_config.items():
+            setattr(parallel_config, key, value)
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: maybe consider a helper function for unpacking the init message and setting parallel config fields?
created_at: 2025-04-07T20:11:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -255,45 +258,21 @@ def collective_rpc(self,
         return self.engine_core.collective_rpc(method, timeout, args, kwargs)
 
 
+class CoreEngineState(Enum):
+    NEW = auto()
+    CONNECTED = auto()
+    READY = auto()
+
+
 class CoreEngine:
     """One per data parallel rank."""
 
-    def __init__(
-        self,
-        vllm_config: VllmConfig,
-        executor_class: type[Executor],
-        log_stats: bool,
-        input_path: str,
-        output_path: str,
-        index: int = 0,
-        local_dp_rank: int = 0,
-    ):
-        self.index = index
+    def __init__(self, index: int = 0, local: bool = True):
path: vllm/v1/engine/core_client.py
user: login: afeldman-nm
body: Nit: Unclear what local means here; see my previous comment on this topic. Is it possible that we have overloaded the word "local"?
created_at: 2025-04-07T20:45:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -255,45 +258,21 @@ def collective_rpc(self,
         return self.engine_core.collective_rpc(method, timeout, args, kwargs)
 
 
+class CoreEngineState(Enum):
+    NEW = auto()
+    CONNECTED = auto()
+    READY = auto()
+
+
 class CoreEngine:
path: vllm/v1/engine/core_client.py
user: login: afeldman-nm
body: Nit: with your recent changes, it seems like `CoreEngine` is reduced to a state management class; what about renaming `CoreEngineState` -> `CoreEngineStatusEnum` (or just `CoreEngineStatus`) and `CoreEngine` to `CoreEngineState`
created_at: 2025-04-07T20:50:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -383,67 +362,152 @@ def sigusr1_handler(signum, frame):
         self.resources = BackgroundResources(ctx=sync_ctx)
         self._finalizer = weakref.finalize(self, self.resources)
 
-        # Paths and sockets for IPC.
-        self.output_path = get_open_zmq_ipc_path()
-        input_path = get_open_zmq_ipc_path()
-        self.input_socket = make_zmq_socket(self.ctx,
-                                            input_path,
-                                            zmq.ROUTER,
-                                            bind=True)
-        self.resources.input_socket = self.input_socket
-
-        new_core_engine = lambda index, local_dp_rank=None: CoreEngine(
-            vllm_config, executor_class, log_stats, input_path, self.
-            output_path, index, local_dp_rank)
-
-        # Start engine core process(es).
-        self._init_core_engines(vllm_config, new_core_engine,
-                                self.resources.core_engines)
+        parallel_config = vllm_config.parallel_config
+        local_engine_count = parallel_config.data_parallel_size_local
+        start_index = parallel_config.data_parallel_rank
+        local_start_index = parallel_config.data_parallel_rank_local
+
+        # SPMD mode is where there is an LLM instance per DP rank and one
+        # core engine per LLM, see examples/offline_inference/data_parallel.py.
path: vllm/v1/engine/core_client.py
user: login: afeldman-nm
body: Nit: I checked `examples/offline_inference/data_parallel.py` on main and there appears to be no reference to SPMD, nor do I see a tweak of the example code in this PR. Would it make sense to add a comment mentioning SPMD in the example?
created_at: 2025-04-07T20:54:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -541,8 +585,20 @@ def __init__(
         _add_prefix(sys.stdout, process_name, pid)
         _add_prefix(sys.stderr, process_name, pid)
 
-        dp_size = vllm_config.parallel_config.data_parallel_size
+        # Counts forward-passes of the model so that we can synchronize
+        # finished with DP peers every N steps.
+        self.counter = 0
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: Probably a nit: was there a reason for moving this line?
created_at: 2025-04-07T21:01:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -309,20 +309,44 @@ class EngineCoreProc(EngineCore):
 
     def __init__(
         self,
-        input_path: str,
-        output_path: str,
         vllm_config: VllmConfig,
+        on_head_node: bool,
+        input_address: str,
         executor_class: type[Executor],
         log_stats: bool,
         engine_index: int = 0,
     ):
+        # Create input socket.
+        input_ctx = zmq.Context()  # type: ignore[attr-defined]
+        identity = engine_index.to_bytes(length=2, byteorder="little")
+        input_socket = make_zmq_socket(input_ctx,
+                                       input_address,
+                                       zmq.DEALER,
+                                       identity=identity,
+                                       bind=False)
+
+        # Register engine with front-end.
+        output_address = self.startup_handshake(input_socket, on_head_node,
+                                                vllm_config.parallel_config)
+
+        # Set up data parallel environment.
+        self._init_data_parallel(vllm_config)
+
+        # Initialize engine core and model.
         super().__init__(vllm_config, executor_class, log_stats)
 
         self.step_fn = (self.step if self.batch_queue is None else
                         self.step_with_batch_queue)
 
         self.global_unfinished_reqs = False
 
+        # Send ready message.
path: vllm/v1/engine/core.py
user: login: njhill
body: @afeldman-nm in this context it will be running on the front-end server which is always on the head node.
created_at: 2025-04-07T21:29:41Z
author_association: MEMBER
diff_hunk: @@ -332,12 +356,39 @@ def __init__(
                                             Any]] = queue.Queue()
         self.output_queue: queue.Queue[EngineCoreOutputs] = queue.Queue()
         threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
+                         args=(input_socket, ),
                          daemon=True).start()
         threading.Thread(target=self.process_output_socket,
-                         args=(output_path, engine_index),
+                         args=(output_address, engine_index),
                          daemon=True).start()
 
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        input_socket.poll(timeout=5 * 60 * 1000)
+        init_bytes = input_socket.recv()
+        init_message = msgspec.msgpack.decode(init_bytes)
+        logger.debug("Received init message: %s", init_message)
+
+        output_socket_address = init_message["output_socket_address"]
+        #TBD maybe replace IP with configured head node address
path: vllm/v1/engine/core.py
user: login: njhill
body: Can do, though it's not necessarily a TODO.
created_at: 2025-04-07T21:30:34Z
author_association: MEMBER
diff_hunk: @@ -332,12 +356,39 @@ def __init__(
                                             Any]] = queue.Queue()
         self.output_queue: queue.Queue[EngineCoreOutputs] = queue.Queue()
         threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
+                         args=(input_socket, ),
                          daemon=True).start()
         threading.Thread(target=self.process_output_socket,
-                         args=(output_path, engine_index),
+                         args=(output_address, engine_index),
                          daemon=True).start()
 
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        input_socket.poll(timeout=5 * 60 * 1000)
+        init_bytes = input_socket.recv()
+        init_message = msgspec.msgpack.decode(init_bytes)
+        logger.debug("Received init message: %s", init_message)
+
+        output_socket_address = init_message["output_socket_address"]
+        #TBD maybe replace IP with configured head node address
+
+        received_parallel_config = init_message["parallel_config"]
+        for key, value in received_parallel_config.items():
+            setattr(parallel_config, key, value)
path: vllm/v1/engine/core.py
user: login: njhill
body: It's only two lines and not done anywhere else that I'm aware of.
created_at: 2025-04-07T21:31:08Z
author_association: MEMBER
diff_hunk: @@ -255,45 +258,21 @@ def collective_rpc(self,
         return self.engine_core.collective_rpc(method, timeout, args, kwargs)
 
 
+class CoreEngineState(Enum):
+    NEW = auto()
+    CONNECTED = auto()
+    READY = auto()
+
+
 class CoreEngine:
     """One per data parallel rank."""
 
-    def __init__(
-        self,
-        vllm_config: VllmConfig,
-        executor_class: type[Executor],
-        log_stats: bool,
-        input_path: str,
-        output_path: str,
-        index: int = 0,
-        local_dp_rank: int = 0,
-    ):
-        self.index = index
+    def __init__(self, index: int = 0, local: bool = True):
path: vllm/v1/engine/core_client.py
user: login: njhill
body: Local means that it's local to the front-end (same node), otherwise it's remote.
created_at: 2025-04-07T21:32:00Z
author_association: MEMBER
diff_hunk: @@ -309,20 +309,44 @@ class EngineCoreProc(EngineCore):
 
     def __init__(
         self,
-        input_path: str,
-        output_path: str,
         vllm_config: VllmConfig,
+        on_head_node: bool,
+        input_address: str,
         executor_class: type[Executor],
         log_stats: bool,
         engine_index: int = 0,
     ):
+        # Create input socket.
+        input_ctx = zmq.Context()  # type: ignore[attr-defined]
+        identity = engine_index.to_bytes(length=2, byteorder="little")
+        input_socket = make_zmq_socket(input_ctx,
+                                       input_address,
+                                       zmq.DEALER,
+                                       identity=identity,
+                                       bind=False)
+
+        # Register engine with front-end.
+        output_address = self.startup_handshake(input_socket, on_head_node,
+                                                vllm_config.parallel_config)
+
+        # Set up data parallel environment.
+        self._init_data_parallel(vllm_config)
+
+        # Initialize engine core and model.
         super().__init__(vllm_config, executor_class, log_stats)
 
         self.step_fn = (self.step if self.batch_queue is None else
                         self.step_with_batch_queue)
 
         self.global_unfinished_reqs = False
 
+        # Send ready message.
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: I see. Maybe it's out of scope for this PR, but why not change the "local" field of the dict to "on_head_node" (and make equivalent terminological changes in other parts of the code)?

It may just be me but I find I am getting confused when local is used in an absolute sense to refer to being on the head node.
created_at: 2025-04-07T21:34:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -332,12 +356,39 @@ def __init__(
                                             Any]] = queue.Queue()
         self.output_queue: queue.Queue[EngineCoreOutputs] = queue.Queue()
         threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
+                         args=(input_socket, ),
                          daemon=True).start()
         threading.Thread(target=self.process_output_socket,
-                         args=(output_path, engine_index),
+                         args=(output_address, engine_index),
                          daemon=True).start()
 
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        input_socket.poll(timeout=5 * 60 * 1000)
path: vllm/v1/engine/core.py
user: login: njhill
body: Ah yes thanks I think I'd intended to add that here.
created_at: 2025-04-07T21:35:09Z
author_association: MEMBER
diff_hunk: @@ -541,8 +585,20 @@ def __init__(
         _add_prefix(sys.stdout, process_name, pid)
         _add_prefix(sys.stderr, process_name, pid)
 
-        dp_size = vllm_config.parallel_config.data_parallel_size
+        # Counts forward-passes of the model so that we can synchronize
+        # finished with DP peers every N steps.
+        self.counter = 0
path: vllm/v1/engine/core.py
user: login: njhill
body: It should be set before the engine is started (threads will be started in the superclass constructor and could begin doing work immediately).
created_at: 2025-04-07T21:37:32Z
author_association: MEMBER
diff_hunk: @@ -332,12 +356,39 @@ def __init__(
                                             Any]] = queue.Queue()
         self.output_queue: queue.Queue[EngineCoreOutputs] = queue.Queue()
         threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
+                         args=(input_socket, ),
                          daemon=True).start()
         threading.Thread(target=self.process_output_socket,
-                         args=(output_path, engine_index),
+                         args=(output_address, engine_index),
                          daemon=True).start()
 
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        input_socket.poll(timeout=5 * 60 * 1000)
path: vllm/v1/engine/core.py
user: login: njhill
body: Pushed an update for this.
created_at: 2025-04-07T22:48:57Z
author_association: MEMBER
diff_hunk: @@ -255,45 +258,21 @@ def collective_rpc(self,
         return self.engine_core.collective_rpc(method, timeout, args, kwargs)
 
 
+class CoreEngineState(Enum):
+    NEW = auto()
+    CONNECTED = auto()
+    READY = auto()
+
+
 class CoreEngine:
path: vllm/v1/engine/core_client.py
user: login: njhill
body: I think it's ok like this, it will likely go away anyhow in a subsequent update to support multiple API servers.
created_at: 2025-04-07T23:06:46Z
author_association: MEMBER
diff_hunk: @@ -309,20 +309,44 @@ class EngineCoreProc(EngineCore):
 
     def __init__(
         self,
-        input_path: str,
-        output_path: str,
         vllm_config: VllmConfig,
+        on_head_node: bool,
+        input_address: str,
         executor_class: type[Executor],
         log_stats: bool,
         engine_index: int = 0,
     ):
+        # Create input socket.
+        input_ctx = zmq.Context()  # type: ignore[attr-defined]
path: vllm/v1/engine/core.py
user: login: njhill
body: Looks like it's not needed, removed!
created_at: 2025-04-07T23:11:19Z
author_association: MEMBER
diff_hunk: @@ -434,6 +438,29 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
                             'MoE layers will be sharded according to the '
                             'product of the tensor-parallel-size and '
                             'data-parallel-size.')
+        parser.add_argument('--data-parallel-size-local',
+                            '-dpl',
+                            type=int,
+                            default=EngineArgs.data_parallel_size_local,
+                            help='Number of data parallel replicas to run on '
+                            'this node.')
+        parser.add_argument('--data-parallel-start-rank',
+                            '-dpr',
+                            type=int,
+                            default=EngineArgs.data_parallel_start_rank,
+                            help='Starting data parallel rank for secondary '
+                            'nodes.')
path: vllm/engine/arg_utils.py
user: login: njhill
body: @youkaichao I moved the start-rank one but the other one is used within the AsyncLLM impl which in theory could be used in other contexts (e.g. different front-end), so it makes sense for it to be in the config I think.

I could move it from `EngineArgs` into `AsyncEngineArgs` but then it would be on it's own so thought better to keep with the other data parallel args.
created_at: 2025-04-07T23:17:45Z
author_association: MEMBER
diff_hunk: @@ -332,12 +356,39 @@ def __init__(
                                             Any]] = queue.Queue()
         self.output_queue: queue.Queue[EngineCoreOutputs] = queue.Queue()
         threading.Thread(target=self.process_input_socket,
-                         args=(input_path, engine_index),
+                         args=(input_socket, ),
                          daemon=True).start()
         threading.Thread(target=self.process_output_socket,
-                         args=(output_path, engine_index),
+                         args=(output_address, engine_index),
                          daemon=True).start()
 
+    @staticmethod
+    def startup_handshake(input_socket: zmq.Socket, on_head_node: bool,
+                          parallel_config: ParallelConfig) -> str:
+
+        # Send registration message.
+        input_socket.send(
+            msgspec.msgpack.encode({
+                "status": "HELLO",
+                "local": on_head_node,
+            }))
+
+        # Receive initialization message.
+        logger.info("Waiting for init message from front-end.")
+        input_socket.poll(timeout=5 * 60 * 1000)
+        init_bytes = input_socket.recv()
+        init_message = msgspec.msgpack.decode(init_bytes)
+        logger.debug("Received init message: %s", init_message)
+
+        output_socket_address = init_message["output_socket_address"]
+        #TBD maybe replace IP with configured head node address
+
+        received_parallel_config = init_message["parallel_config"]
+        for key, value in received_parallel_config.items():
+            setattr(parallel_config, key, value)
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: Okay sounds good.
created_at: 2025-04-08T20:54:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -541,8 +585,20 @@ def __init__(
         _add_prefix(sys.stdout, process_name, pid)
         _add_prefix(sys.stderr, process_name, pid)
 
-        dp_size = vllm_config.parallel_config.data_parallel_size
+        # Counts forward-passes of the model so that we can synchronize
+        # finished with DP peers every N steps.
+        self.counter = 0
path: vllm/v1/engine/core.py
user: login: afeldman-nm
body: Okay sounds good.
created_at: 2025-04-08T20:55:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -383,72 +362,161 @@ def sigusr1_handler(signum, frame):
         self.resources = BackgroundResources(ctx=sync_ctx)
         self._finalizer = weakref.finalize(self, self.resources)
 
-        # Paths and sockets for IPC.
-        self.output_path = get_open_zmq_ipc_path()
-        input_path = get_open_zmq_ipc_path()
-        self.input_socket = make_zmq_socket(self.ctx,
-                                            input_path,
-                                            zmq.ROUTER,
-                                            bind=True)
-        self.resources.input_socket = self.input_socket
-
-        new_core_engine = lambda index, local_dp_rank=None: CoreEngine(
-            vllm_config, executor_class, log_stats, input_path, self.
-            output_path, index, local_dp_rank)
-
-        # Start engine core process(es).
-        self._init_core_engines(vllm_config, new_core_engine,
-                                self.resources.core_engines)
+        parallel_config = vllm_config.parallel_config
+        local_engine_count = parallel_config.data_parallel_size_local
+        start_index = parallel_config.data_parallel_rank
+        local_start_index = parallel_config.data_parallel_rank_local
+
+        # SPMD mode is where there is an LLM instance per DP rank and one
+        # core engine per LLM, see examples/offline_inference/data_parallel.py.
+        spmd_mode = local_start_index is not None
+        if spmd_mode:
+            assert local_engine_count == 1
+            self.core_engines = [
+                CoreEngine(index=local_start_index, local=True)
+            ]
+        else:
+            assert start_index == 0
+            local_start_index = 0
+            self.core_engines = [
+                CoreEngine(index=i, local=(i < local_engine_count))
+                for i in range(parallel_config.data_parallel_size)
+            ]
+
+        input_address, output_address = self._get_zmq_addresses(
+            parallel_config, spmd_mode)
+
+        # Create input and output sockets.
+        self.input_socket = self.resources.input_socket = make_zmq_socket(
+            self.ctx, input_address, zmq.ROUTER, bind=True)
+
+        self.resources.output_socket = make_zmq_socket(self.ctx,
+                                                       output_address,
+                                                       zmq.constants.PULL)
+        # Start local engines.
+        if local_engine_count:
+            # In server mode, start_index and local_start_index will both be 0.
+            self.resources.local_engine_manager = CoreEngineProcManager(
+                EngineCoreProc.run_engine_core,
+                vllm_config=vllm_config,
+                executor_class=executor_class,
+                log_stats=log_stats,
+                input_address=input_address,
+                on_head_node=True,
+                local_engine_count=local_engine_count,
+                start_index=start_index,
+                local_start_index=local_start_index)
+
+        self.core_engine = self.core_engines[0]
 
         # Wait for engine core process(es) to start.
-        self._wait_for_engine_startup()
+        self._wait_for_engine_startup(output_address, parallel_config)
 
         self.utility_results: dict[int, AnyFuture] = {}
 
-    def _wait_for_engine_startup(self):
+    @staticmethod
+    def _get_zmq_addresses(parallel_config: ParallelConfig,
+                           spmd_mode: bool) -> tuple[str, str]:
+        """Returns (input_address, output_address)."""
+        dp_size = parallel_config.data_parallel_size
+        local_engine_count = parallel_config.data_parallel_size_local
+
+        if local_engine_count == dp_size or spmd_mode:
+            input_address = get_open_zmq_ipc_path()
+            output_address = get_open_zmq_ipc_path()
+        else:
+            host = parallel_config.data_parallel_master_ip
+            input_port = parallel_config.data_parallel_rpc_port
+            output_port = get_open_port()
+            input_address = f"tcp://{host}:{input_port}"
+            output_address = f"tcp://{host}:{output_port}"
path: vllm/v1/engine/core_client.py
user: login: russellb
body: This requires a bit more care for IPv6. If the address is IPv6, you need square brackets around `{host}`.

Similar code here: https://github.com/vllm-project/vllm/blob/34b2cf3b3366443e6b68a43572aee1800f6f2b90/vllm/distributed/device_communicators/shm_broadcast.py#L238-L241

Note that for zmq, the socket also needs an extra flag set to support IPv6: 

https://github.com/vllm-project/vllm/blob/34b2cf3b3366443e6b68a43572aee1800f6f2b90/vllm/distributed/device_communicators/shm_broadcast.py#L303-L304

though the latter is probably a fix to `make_zmq_socket`, which isn't new code in this PR. I can fix that part separately. At least for this PR, ensuring the zmq URLs are correct is enough.
created_at: 2025-04-11T15:25:44Z
author_association: MEMBER
diff_hunk: @@ -383,72 +362,161 @@ def sigusr1_handler(signum, frame):
         self.resources = BackgroundResources(ctx=sync_ctx)
         self._finalizer = weakref.finalize(self, self.resources)
 
-        # Paths and sockets for IPC.
-        self.output_path = get_open_zmq_ipc_path()
-        input_path = get_open_zmq_ipc_path()
-        self.input_socket = make_zmq_socket(self.ctx,
-                                            input_path,
-                                            zmq.ROUTER,
-                                            bind=True)
-        self.resources.input_socket = self.input_socket
-
-        new_core_engine = lambda index, local_dp_rank=None: CoreEngine(
-            vllm_config, executor_class, log_stats, input_path, self.
-            output_path, index, local_dp_rank)
-
-        # Start engine core process(es).
-        self._init_core_engines(vllm_config, new_core_engine,
-                                self.resources.core_engines)
+        parallel_config = vllm_config.parallel_config
+        local_engine_count = parallel_config.data_parallel_size_local
+        start_index = parallel_config.data_parallel_rank
+        local_start_index = parallel_config.data_parallel_rank_local
+
+        # SPMD mode is where there is an LLM instance per DP rank and one
+        # core engine per LLM, see examples/offline_inference/data_parallel.py.
+        spmd_mode = local_start_index is not None
+        if spmd_mode:
+            assert local_engine_count == 1
+            self.core_engines = [
+                CoreEngine(index=local_start_index, local=True)
+            ]
+        else:
+            assert start_index == 0
+            local_start_index = 0
+            self.core_engines = [
+                CoreEngine(index=i, local=(i < local_engine_count))
+                for i in range(parallel_config.data_parallel_size)
+            ]
+
+        input_address, output_address = self._get_zmq_addresses(
+            parallel_config, spmd_mode)
+
+        # Create input and output sockets.
+        self.input_socket = self.resources.input_socket = make_zmq_socket(
+            self.ctx, input_address, zmq.ROUTER, bind=True)
+
+        self.resources.output_socket = make_zmq_socket(self.ctx,
+                                                       output_address,
+                                                       zmq.constants.PULL)
+        # Start local engines.
+        if local_engine_count:
+            # In server mode, start_index and local_start_index will both be 0.
+            self.resources.local_engine_manager = CoreEngineProcManager(
+                EngineCoreProc.run_engine_core,
+                vllm_config=vllm_config,
+                executor_class=executor_class,
+                log_stats=log_stats,
+                input_address=input_address,
+                on_head_node=True,
+                local_engine_count=local_engine_count,
+                start_index=start_index,
+                local_start_index=local_start_index)
+
+        self.core_engine = self.core_engines[0]
 
         # Wait for engine core process(es) to start.
-        self._wait_for_engine_startup()
+        self._wait_for_engine_startup(output_address, parallel_config)
 
         self.utility_results: dict[int, AnyFuture] = {}
 
-    def _wait_for_engine_startup(self):
+    @staticmethod
+    def _get_zmq_addresses(parallel_config: ParallelConfig,
+                           spmd_mode: bool) -> tuple[str, str]:
+        """Returns (input_address, output_address)."""
+        dp_size = parallel_config.data_parallel_size
+        local_engine_count = parallel_config.data_parallel_size_local
+
+        if local_engine_count == dp_size or spmd_mode:
+            input_address = get_open_zmq_ipc_path()
+            output_address = get_open_zmq_ipc_path()
+        else:
+            host = parallel_config.data_parallel_master_ip
+            input_port = parallel_config.data_parallel_rpc_port
+            output_port = get_open_port()
+            input_address = f"tcp://{host}:{input_port}"
+            output_address = f"tcp://{host}:{output_port}"
path: vllm/v1/engine/core_client.py
user: login: russellb
body: The fix to `make_zmq_socket()` is here: https://github.com/vllm-project/vllm/pull/16506
created_at: 2025-04-11T17:59:20Z
author_association: MEMBER
diff_hunk: @@ -383,72 +362,161 @@ def sigusr1_handler(signum, frame):
         self.resources = BackgroundResources(ctx=sync_ctx)
         self._finalizer = weakref.finalize(self, self.resources)
 
-        # Paths and sockets for IPC.
-        self.output_path = get_open_zmq_ipc_path()
-        input_path = get_open_zmq_ipc_path()
-        self.input_socket = make_zmq_socket(self.ctx,
-                                            input_path,
-                                            zmq.ROUTER,
-                                            bind=True)
-        self.resources.input_socket = self.input_socket
-
-        new_core_engine = lambda index, local_dp_rank=None: CoreEngine(
-            vllm_config, executor_class, log_stats, input_path, self.
-            output_path, index, local_dp_rank)
-
-        # Start engine core process(es).
-        self._init_core_engines(vllm_config, new_core_engine,
-                                self.resources.core_engines)
+        parallel_config = vllm_config.parallel_config
+        local_engine_count = parallel_config.data_parallel_size_local
+        start_index = parallel_config.data_parallel_rank
+        local_start_index = parallel_config.data_parallel_rank_local
+
+        # SPMD mode is where there is an LLM instance per DP rank and one
+        # core engine per LLM, see examples/offline_inference/data_parallel.py.
+        spmd_mode = local_start_index is not None
+        if spmd_mode:
+            assert local_engine_count == 1
+            self.core_engines = [
+                CoreEngine(index=local_start_index, local=True)
+            ]
+        else:
+            assert start_index == 0
+            local_start_index = 0
+            self.core_engines = [
+                CoreEngine(index=i, local=(i < local_engine_count))
+                for i in range(parallel_config.data_parallel_size)
+            ]
+
+        input_address, output_address = self._get_zmq_addresses(
+            parallel_config, spmd_mode)
+
+        # Create input and output sockets.
+        self.input_socket = self.resources.input_socket = make_zmq_socket(
+            self.ctx, input_address, zmq.ROUTER, bind=True)
+
+        self.resources.output_socket = make_zmq_socket(self.ctx,
+                                                       output_address,
+                                                       zmq.constants.PULL)
+        # Start local engines.
+        if local_engine_count:
+            # In server mode, start_index and local_start_index will both be 0.
+            self.resources.local_engine_manager = CoreEngineProcManager(
+                EngineCoreProc.run_engine_core,
+                vllm_config=vllm_config,
+                executor_class=executor_class,
+                log_stats=log_stats,
+                input_address=input_address,
+                on_head_node=True,
+                local_engine_count=local_engine_count,
+                start_index=start_index,
+                local_start_index=local_start_index)
+
+        self.core_engine = self.core_engines[0]
 
         # Wait for engine core process(es) to start.
-        self._wait_for_engine_startup()
+        self._wait_for_engine_startup(output_address, parallel_config)
 
         self.utility_results: dict[int, AnyFuture] = {}
 
-    def _wait_for_engine_startup(self):
+    @staticmethod
+    def _get_zmq_addresses(parallel_config: ParallelConfig,
+                           spmd_mode: bool) -> tuple[str, str]:
+        """Returns (input_address, output_address)."""
+        dp_size = parallel_config.data_parallel_size
+        local_engine_count = parallel_config.data_parallel_size_local
+
+        if local_engine_count == dp_size or spmd_mode:
+            input_address = get_open_zmq_ipc_path()
+            output_address = get_open_zmq_ipc_path()
+        else:
+            host = parallel_config.data_parallel_master_ip
+            input_port = parallel_config.data_parallel_rpc_port
+            output_port = get_open_port()
+            input_address = f"tcp://{host}:{input_port}"
+            output_address = f"tcp://{host}:{output_port}"
path: vllm/v1/engine/core_client.py
user: login: njhill
body: Now addressed, thanks!
created_at: 2025-04-17T19:21:37Z
author_association: MEMBER
diff_hunk: @@ -57,3 +82,46 @@ def subparser_init(
 
 def cmd_init() -> list[CLISubcommand]:
     return [ServeSubcommand()]
+
+
+def run_headless(args: argparse.Namespace):
+
+    # Create the EngineConfig.
+    engine_args = AsyncEngineArgs.from_cli_args(args)
+    usage_context = UsageContext.OPENAI_API_SERVER
+    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
+
+    if not envs.VLLM_USE_V1:
+        raise RuntimeError("Headless mode is only supported for V1")
+
+    parallel_config = vllm_config.parallel_config
+    local_engine_count = parallel_config.data_parallel_size_local
+    host = parallel_config.data_parallel_master_ip
+    port = engine_args.data_parallel_rpc_port  # add to config too
+    input_address = f"tcp://{host}:{port}"
path: vllm/entrypoints/cli/serve.py
user: login: Copilot
body: For consistency and to ensure proper handling of IPv6 addresses, consider using the get_tcp_uri(host, port) helper instead of manually constructing the TCP URI.
```suggestion
    input_address = get_tcp_uri(host, port)
```
created_at: 2025-04-17T23:37:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -255,45 +258,21 @@ def collective_rpc(self,
         return self.engine_core.collective_rpc(method, timeout, args, kwargs)
 
 
+class CoreEngineState(Enum):
+    NEW = auto()
+    CONNECTED = auto()
+    READY = auto()
+
+
 class CoreEngine:
path: vllm/v1/engine/core_client.py
user: login: ruisearch42
body: If `CoreEngine` is per DP rank, does it make sense to call it `DPEngine`? It feels a bit confusing for `CoreEngine` vs `EngineCore`.
created_at: 2025-05-12T21:41:25Z
author_association: COLLABORATOR
diff_hunk: @@ -255,45 +258,21 @@ def collective_rpc(self,
         return self.engine_core.collective_rpc(method, timeout, args, kwargs)
 
 
+class CoreEngineState(Enum):
+    NEW = auto()
+    CONNECTED = auto()
+    READY = auto()
+
+
 class CoreEngine:
path: vllm/v1/engine/core_client.py
user: login: njhill
body: @ruisearch42 yeah I'm not sure what name is best. It does represent an engine core process but we already have a `EngineCore` class which is the engine itself.
created_at: 2025-05-13T17:38:21Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-03T01:19:05Z
body: 👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀
author_association: NONE
user: login: zxfan-cpu
created_at: 2025-04-03T02:46:35Z
body: My pr https://github.com/vllm-project/vllm/pull/15863 is based on Ray's support for multi-node DP. Could you provide some feedback for modifications so we can merge with your PR?
author_association: CONTRIBUTOR
user: login: youkaichao
created_at: 2025-04-04T16:22:57Z
body: is this "frontend" is one api server process?
author_association: MEMBER
user: login: njhill
created_at: 2025-04-04T18:15:03Z
body: > is this "frontend" is one api server process?

So far, yes. I've been trying to open incremental PRs, working on multi api server right now as the next one (on top of this PR). It won't change much in terms of the deployment semantics though - just maybe one additional arg to specify how many api server procs (applies only to head node and so mutually exclusive with `--headless`).
author_association: MEMBER
user: login: mergify[bot]
created_at: 2025-04-04T20:37:17Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: afeldman-nm
created_at: 2025-04-07T18:23:52Z
body: > 
> #### TODO (this PR):
> * [ ]  Feedback
> * [x]  Some code simplification
> * [x]  Fix offline DP compatiblity
> * [ ]  CI tests
> 
> #### Next PR:
> * API server scale-out

Hey @njhill , just want to clarify - when you include "CI tests" in the TODO, does this mean you will be adding new unit tests? Maybe it would be valuable to unit-test the communication process in scaled-out scenarios for example.
author_association: CONTRIBUTOR
user: login: njhill
created_at: 2025-04-07T21:33:42Z
body: > > #### TODO (this PR):
> > 
> > * [ ]   Feedback
> > * [x]   Some code simplification
> > * [x]   Fix offline DP compatiblity
> > * [ ]   CI tests
> > 
> > #### Next PR:
> > 
> > * API server scale-out
> 
> Hey @njhill , just want to clarify - when you include "CI tests" in the TODO, does this mean you will be adding new unit tests? Maybe it would be valuable to unit-test the communication process in scaled-out scenarios for example.

Yes, plan to add tests to cover launching some engine(s) remotely, but wanted to get some feedback on the design first.
author_association: MEMBER
user: login: mergify[bot]
created_at: 2025-04-11T04:33:29Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: njhill
created_at: 2025-04-18T14:32:38Z
body: Thanks @simon-mo, @youkaichao. Don't merge quite yet, I will push small updates and a multi-node CI test today.
author_association: MEMBER
user: login: mergify[bot]
created_at: 2025-04-23T04:37:02Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: russellb
created_at: 2025-04-29T21:16:01Z
body: related to my last comment -- I have some changes in a private branch that allow turning off the `pickle` fallback in our custom msgpack encoder/decoder. I'm going to push a PR with just those changes. We can try using that here to see if anything breaks and perhaps even have an environment variable that allows turning it back on if something unexpected breaks -- `VLLM_ALLOW_INSECURE_PICKLE_FALLBACK` or something ...
author_association: MEMBER
user: login: russellb
created_at: 2025-04-30T01:02:22Z
body: posted https://github.com/vllm-project/vllm/pull/17427 with the first step -- allowing the pickle fallback to be turned off in the custom msgpack encoder/decoder plus an env var to force it back on if something breaks unexpectedly
author_association: MEMBER
user: login: mergify[bot]
created_at: 2025-04-30T10:21:47Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: mergify[bot]
created_at: 2025-05-08T05:31:02Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: njhill
created_at: 2025-05-11T21:59:11Z
body: > We've talked about this some offline, but for the record, I would prefer that we don't introduce any additional multi-node interfaces that are not secured in any way, and worse, allow arbitrary code execution via `pickle`. I understand the argument that PyTorch says its distributed communications are insecure, but that's not a great reason to make our own code worse and harder to fix in the future.

The security concerns are hopefully addressed now that we have https://github.com/vllm-project/vllm/pull/17490 merged.
author_association: MEMBER
user: login: simon-mo
created_at: 2025-05-11T23:46:52Z
body: @russellb @youkaichao  can you please help final round of review? 
author_association: COLLABORATOR
user: login: russellb
created_at: 2025-05-12T01:48:19Z
body: > > We've talked about this some offline, but for the record, I would prefer that we don't introduce any additional multi-node interfaces that are not secured in any way, and worse, allow arbitrary code execution via `pickle`. I understand the argument that PyTorch says its distributed communications are insecure, but that's not a great reason to make our own code worse and harder to fix in the future.
> 
> The security concerns are hopefully addressed now that we have #17490 merged.

Confirmed, yes -- thank you for working with me on this!
author_association: MEMBER
user: login: njhill
created_at: 2025-05-12T15:24:57Z
body: Fixing rebase issues
author_association: MEMBER
user: login: njhill
created_at: 2025-05-12T17:51:02Z
body: I've opened a separate PR for one of the new CI test failures which is unrelated: https://github.com/vllm-project/vllm/pull/18007
author_association: MEMBER
user: login: mergify[bot]
created_at: 2025-05-12T19:10:28Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: GGKOP
created_at: 2025-05-23T08:32:59Z
body: When will vllm.entrypoints.openai.api_server support --headless and --data-parallel-start-rank? @njhill 
author_association: NONE
user: login: SparkJiao
created_at: 2025-05-31T09:12:18Z
body: Hi, does currently this approach support IPv6 address? 
author_association: NONE
user: login: russellb
created_at: 2025-05-31T19:27:00Z
body: > Hi, does currently this approach support IPv6 address?

I've been trying to fix IPv6 support anywhere I see that doesn't work correctly. If you see it not working anywhere, let me know. I looked at it in this PR and think it should be OK now, though I didn't try it myself.
author_association: MEMBER
user: login: SparkJiao
created_at: 2025-06-01T01:43:41Z
body: @russellb Hi, I think this PR can fix the problem: https://github.com/vllm-project/vllm/pull/18991. I have tried it.
author_association: NONE