Repository: vllm-project/vllm
Pull request #17827: [Feature] Support Pipeline Parallism in torchrun SPMD offline inference for V1
Author: login: luccafong
Labels: name: documentation
description: Improvements or additions to documentation
name: ready
description: ONLY add when PR is ready to merge/full CI is needed
name: ci/build
name: v1
Created: 2025-05-08T04:20:03Z | Merged: 2025-05-16T05:28:27Z

This PR add support for PP on torchrun offline inference, note this now does not support overlapping mircobatches so not the most efficient way to ublock PP use cases, will improve as follow ups.

```
torchrun --nnodes 2 --nproc-per-node 2 --rdzv-id=random12345 --rdzv-backend=c10d --rdzv-endpoint=<host:port>  examples/offline_inference/torchrun_example.py
```

output
```
--------------------------------------------------
Prompt: 'Hello, my name is'
Generated text: ' Hilary and I have been a full time Licensed Massage Therapist since 199'

--------------------------------------------------
Prompt: 'The president of the United States is'
Generated text: ' a leader of the free world. His actions and statements have a direct impact on'

--------------------------------------------------
Prompt: 'The capital of France is'
Generated text: ' the largest city in the country and a major European metropolis. It is also'

--------------------------------------------------
Prompt: 'The future of AI is'
Generated text: ' not as promising as you might think\nWith a preponderance of artificial intelligence'
```



Diff:
diff --git a/.buildkite/test-pipeline.yaml b/.buildkite/test-pipeline.yaml
index 1040d1e1b801..414e4d0ee75b 100644
--- a/.buildkite/test-pipeline.yaml
+++ b/.buildkite/test-pipeline.yaml
@@ -148,6 +148,8 @@ steps:
   # test with tp=2 and external_dp=2
   - VLLM_USE_V1=0 torchrun --nproc-per-node=4 distributed/test_torchrun_example.py
   - torchrun --nproc-per-node=4 distributed/test_torchrun_example.py
+  # test with tp=2 and pp=2
+  - PP_SIZE=2 torchrun --nproc-per-node=4 distributed/test_torchrun_example.py
   # test with internal dp
   - python3 ../examples/offline_inference/data_parallel.py
   - TP_SIZE=2 DP_SIZE=2 pytest -v -s v1/test_async_llm_dp.py
diff --git a/examples/offline_inference/torchrun_example.py b/examples/offline_inference/torchrun_example.py
index c6d9e6b47e21..bb61a0a29e32 100644
--- a/examples/offline_inference/torchrun_example.py
+++ b/examples/offline_inference/torchrun_example.py
@@ -8,6 +8,8 @@
 see `tests/distributed/test_torchrun_example.py` for the unit test.
 """
 
+import torch.distributed as dist
+
 from vllm import LLM, SamplingParams
 
 # Create prompts, the same across all ranks
@@ -27,23 +29,26 @@
 # all ranks have the same random seed, so that sampling can be
 # deterministic across ranks.
 llm = LLM(
-    model="facebook/opt-125m",
+    model="meta-llama/Llama-3.1-8B",
     tensor_parallel_size=2,
+    pipeline_parallel_size=2,
     distributed_executor_backend="external_launcher",
-    seed=0,
+    max_model_len=32768,
+    seed=1,
 )
 
 outputs = llm.generate(prompts, sampling_params)
 
 # all ranks will have the same outputs
-print("-" * 50)
-for output in outputs:
-    prompt = output.prompt
-    generated_text = output.outputs[0].text
-    print(f"Prompt: {prompt!r}\n"
-          f"Generated text: {generated_text!r}")
+if dist.get_rank() == 0:
     print("-" * 50)
-"""
+    for output in outputs:
+        prompt = output.prompt
+        generated_text = output.outputs[0].text
+        print(f"Prompt: {prompt!r}\n"
+              f"Generated text: {generated_text!r}\n")
+        print("-" * 50)
+    """
 Further tips:
 
 1. to communicate control messages across all ranks, use the cpu group,
diff --git a/tests/distributed/test_torchrun_example.py b/tests/distributed/test_torchrun_example.py
index 0420a6454d46..bb38e908b734 100644
--- a/tests/distributed/test_torchrun_example.py
+++ b/tests/distributed/test_torchrun_example.py
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 # unit test for `examples/offline_inference/torchrun_example.py`
-
+import os
 import random
 
 import torch.distributed as dist
@@ -25,6 +25,7 @@
 # to test if all ranks agree on the same kv cache configuration.
 llm = LLM(model="facebook/opt-125m",
           tensor_parallel_size=2,
+          pipeline_parallel_size=int(os.getenv("PP_SIZE", 1)),
           distributed_executor_backend="external_launcher",
           gpu_memory_utilization=random.uniform(0.7, 0.9),
           swap_space=random.randint(1, 4),
diff --git a/vllm/config.py b/vllm/config.py
index 09e89c1116f1..300a99c9cec1 100644
--- a/vllm/config.py
+++ b/vllm/config.py
@@ -1690,7 +1690,6 @@ class ParallelConfig:
     """Port of the data parallel master."""
     enable_expert_parallel: bool = False
     """Use expert parallelism instead of tensor parallelism for MoE layers."""
-
     max_parallel_loading_workers: Optional[int] = None
     """Maximum number of parallel loading workers when loading model
     sequentially in multiple batches. To avoid RAM OOM when using tensor
diff --git a/vllm/distributed/device_communicators/custom_all_reduce.py b/vllm/distributed/device_communicators/custom_all_reduce.py
index 7567161b6ac7..5c2dbcc27b13 100644
--- a/vllm/distributed/device_communicators/custom_all_reduce.py
+++ b/vllm/distributed/device_communicators/custom_all_reduce.py
@@ -265,7 +265,8 @@ def custom_all_reduce(self, input: torch.Tensor) -> Optional[torch.Tensor]:
 
     def close(self):
         if not self.disabled and self._ptr:
-            ops.dispose(self._ptr)
+            if ops is not None:
+                ops.dispose(self._ptr)
             self._ptr = 0
             self.free_shared_buffer(self.meta_ptrs, rank=self.rank)
             self.free_shared_buffer(self.buffer_ptrs, rank=self.rank)
@@ -298,4 +299,5 @@ def free_shared_buffer(pointers: list[int],
                            rank: Optional[int] = 0) -> None:
         if rank is None:
             rank = dist.get_rank(group=group)
-        ops.free_shared_buffer(pointers[rank])
+        if ops is not None:
+            ops.free_shared_buffer(pointers[rank])
diff --git a/vllm/engine/arg_utils.py b/vllm/engine/arg_utils.py
index 240142a1c5d1..9dcb68323d49 100644
--- a/vllm/engine/arg_utils.py
+++ b/vllm/engine/arg_utils.py
@@ -1376,9 +1376,10 @@ def _is_v1_supported_oracle(self, model_config: ModelConfig) -> bool:
             return False
 
         if (self.pipeline_parallel_size > 1
-                and self.distributed_executor_backend not in ["ray", "mp"]):
+                and self.distributed_executor_backend
+                not in ("ray", "mp", "external_launcher")):
             name = "Pipeline Parallelism without Ray distributed executor " \
-                    "or multiprocessing executor"
+                    "or multiprocessing executor or external launcher"
             _raise_or_fallback(feature_name=name, recommend_to_remove=False)
             return False
 
diff --git a/vllm/executor/uniproc_executor.py b/vllm/executor/uniproc_executor.py
index 2e4b47c1e24a..1d3a6e443a80 100644
--- a/vllm/executor/uniproc_executor.py
+++ b/vllm/executor/uniproc_executor.py
@@ -86,9 +86,6 @@ class ExecutorWithExternalLauncher(UniProcExecutor):
     def _init_executor(self) -> None:
         """Initialize the worker and load the model.
         """
-        assert self.vllm_config.parallel_config.pipeline_parallel_size == 1, \
-            ("ExecutorWithExternalLauncher does not "
-            "support pipeline parallelism.")
         assert self.vllm_config.scheduler_config.delay_factor == 0.0, \
             ("ExecutorWithExternalLauncher needs deterministic "
             "execution, so it"
diff --git a/vllm/v1/worker/gpu_model_runner.py b/vllm/v1/worker/gpu_model_runner.py
index 1b16f273a6de..0257a4209687 100644
--- a/vllm/v1/worker/gpu_model_runner.py
+++ b/vllm/v1/worker/gpu_model_runner.py
@@ -20,7 +20,8 @@
                                           has_kv_transfer_group)
 from vllm.distributed.kv_transfer.kv_connector.v1 import KVConnectorBase_V1
 from vllm.distributed.parallel_state import (
-    get_pp_group, graph_capture, prepare_communication_buffer_for_model)
+    get_pp_group, get_tp_group, graph_capture,
+    prepare_communication_buffer_for_model)
 from vllm.forward_context import get_forward_context, set_forward_context
 from vllm.logger import init_logger
 from vllm.model_executor.layers.rotary_embedding import MRotaryEmbedding
@@ -1168,13 +1169,32 @@ def execute_model(
             hidden_states, aux_hidden_states = model_output
         else:
             hidden_states = model_output
-
+        # Broadcast PP output for external_launcher (torchrun)
+        # to make sure we are synced across pp ranks
+        # TODO: Support overlapping mirco-batches
+        # https://github.com/vllm-project/vllm/issues/18019
+        broadcast_pp_output = \
+            self.parallel_config.distributed_executor_backend \
+            == "external_launcher" and len(get_pp_group().ranks) > 0
         if not get_pp_group().is_last_rank:
             # For mid-pipeline stages, return the hidden states.
-            return hidden_states
-
-        sample_hidden_states = hidden_states[logits_indices]
-        logits = self.model.compute_logits(sample_hidden_states, None)
+            if not broadcast_pp_output:
+                return hidden_states
+            assert isinstance(hidden_states, IntermediateTensors)
+            get_pp_group().send_tensor_dict(hidden_states.tensors,
+                                            all_gather_group=get_tp_group())
+            logits = None
+        else:
+            sample_hidden_states = hidden_states[logits_indices]
+            logits = self.model.compute_logits(sample_hidden_states, None)
+        if broadcast_pp_output:
+            model_output_broadcast_data = {
+                "logits": logits.contiguous(),
+            } if logits is not None else {}
+            model_output_broadcast_data = get_pp_group().broadcast_tensor_dict(
+                model_output_broadcast_data, src=len(get_pp_group().ranks) - 1)
+            assert model_output_broadcast_data is not None
+            logits = model_output_broadcast_data["logits"]
 
         # Apply structured output bitmasks if present
         if scheduler_output.grammar_bitmask is not None:
@@ -1192,6 +1212,7 @@ def execute_model(
             # creates a new tensor with separate storage from the original
             # logits tensor. This means any in-place operations on bonus_logits
             # won't affect the original logits tensor.
+            assert logits is not None
             bonus_logits = logits[spec_decode_metadata.bonus_logits_indices]
             sampler_output = self.sampler(
                 logits=bonus_logits,
diff --git a/vllm/v1/worker/gpu_worker.py b/vllm/v1/worker/gpu_worker.py
index d85701fa93df..93129d987940 100644
--- a/vllm/v1/worker/gpu_worker.py
+++ b/vllm/v1/worker/gpu_worker.py
@@ -275,13 +275,13 @@ def execute_model(
 
         output = self.model_runner.execute_model(scheduler_output,
                                                  intermediate_tensors)
-
-        if not get_pp_group().is_last_rank:
+        parallel_config = self.vllm_config.parallel_config
+        if parallel_config.distributed_executor_backend != "external_launcher" \
+            and not get_pp_group().is_last_rank:
             assert isinstance(output, IntermediateTensors)
             get_pp_group().send_tensor_dict(output.tensors,
                                             all_gather_group=get_tp_group())
             return None
-
         assert isinstance(output, ModelRunnerOutput)
         return output if self.is_driver_worker else None
 


Reviews:
user: login: houseroad
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T20:12:31Z
user: login: comaniac
body: Overall LGTM. The main question is not sure why we need `pipeline_parallel_broadcast_output` and cannot make it always true.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T22:02:26Z
user: login: luccafong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-10T00:35:37Z
user: login: luccafong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-10T00:37:21Z
user: login: youkaichao
body: for efficient pp, we would want different stages of pp from different batches run concurrently. would this broadcast make them serialized? like there will be only one batch and one stage running for the full engine.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-10T09:16:44Z
user: login: houseroad
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T05:46:33Z
user: login: houseroad
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T05:47:57Z
user: login: luccafong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T17:19:35Z
user: login: luccafong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T17:31:41Z
user: login: houseroad
body: As discussed with @luccafong , landing this first to unblock some use case. 

Will enhance the perf in the following PR.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-12T19:30:18Z
user: login: ruisearch42
body: LG with a couple comments
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T23:26:43Z
user: login: luccafong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T02:20:42Z
user: login: luccafong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T02:21:23Z

Review comments:
diff_hunk: @@ -1645,7 +1645,9 @@ def data_parallel_rank_local(self, value: int) -> None:
     """Port of the data parallel master."""
     enable_expert_parallel: bool = False
     """Use expert parallelism instead of tensor parallelism for MoE layers."""
-
+    pipeline_parallel_broadcast_output: bool = False
path: vllm/config.py
user: login: houseroad
body: Shall we just always broadcast in any pp > 1 case? Do we really need a new config?
created_at: 2025-05-09T20:12:31Z
author_association: COLLABORATOR
diff_hunk: @@ -1645,7 +1645,9 @@ def data_parallel_rank_local(self, value: int) -> None:
     """Port of the data parallel master."""
     enable_expert_parallel: bool = False
     """Use expert parallelism instead of tensor parallelism for MoE layers."""
-
+    pipeline_parallel_broadcast_output: bool = False
path: vllm/config.py
user: login: comaniac
body: +1
created_at: 2025-05-09T22:02:00Z
author_association: COLLABORATOR
diff_hunk: @@ -1341,9 +1341,10 @@ def _is_v1_supported_oracle(self, model_config: ModelConfig) -> bool:
             return False
 
         if (self.pipeline_parallel_size > 1
-                and self.distributed_executor_backend not in ["ray", "mp"]):
+                and self.distributed_executor_backend
+                not in ["ray", "mp", "external_launcher"]):
path: vllm/engine/arg_utils.py
user: login: comaniac
body: nit: Use immutable tuple.

```suggestion
                not in ("ray", "mp", "external_launcher")):
```
created_at: 2025-05-09T22:02:22Z
author_association: COLLABORATOR
diff_hunk: @@ -1645,7 +1645,9 @@ def data_parallel_rank_local(self, value: int) -> None:
     """Port of the data parallel master."""
     enable_expert_parallel: bool = False
     """Use expert parallelism instead of tensor parallelism for MoE layers."""
-
+    pipeline_parallel_broadcast_output: bool = False
path: vllm/config.py
user: login: luccafong
body: @comaniac  this may break the online serving? was this by design before not to broadcast?
created_at: 2025-05-10T00:35:37Z
author_association: COLLABORATOR
diff_hunk: @@ -1645,7 +1645,9 @@ def data_parallel_rank_local(self, value: int) -> None:
     """Port of the data parallel master."""
     enable_expert_parallel: bool = False
     """Use expert parallelism instead of tensor parallelism for MoE layers."""
-
+    pipeline_parallel_broadcast_output: bool = False
path: vllm/config.py
user: login: luccafong
body: considering in the ray approach,it just sent back to rank 0 through grpc or shared memory?, so others do not need it, so it may introduce overhead?
created_at: 2025-05-10T00:37:21Z
author_association: COLLABORATOR
diff_hunk: @@ -1136,13 +1137,27 @@ def execute_model(
             hidden_states, aux_hidden_states = output
         else:
             hidden_states = output
-
         if not get_pp_group().is_last_rank:
             # For mid-pipeline stages, return the hidden states.
-            return hidden_states
-
-        sample_hidden_states = hidden_states[logits_indices]
-        logits = self.model.compute_logits(sample_hidden_states, None)
+            if not self.parallel_config.pipeline_parallel_broadcast_output:
+                return hidden_states
+            assert isinstance(hidden_states, IntermediateTensors)
+            get_pp_group().send_tensor_dict(hidden_states.tensors,
+                                            all_gather_group=get_tp_group())
+            model_output_broadcast_data = {}
+        else:
+            sample_hidden_states = hidden_states[logits_indices]
+            logits = self.model.compute_logits(sample_hidden_states, None)
+            model_output_broadcast_data = {
+                "logits": logits,
+            }
+        pp_group_ranks = len(get_pp_group().ranks)
+        last_rank_in_group = pp_group_ranks - 1
+        if len(get_pp_group().ranks) > 0:
+            model_output_broadcast_data = get_pp_group().broadcast_tensor_dict(
+                model_output_broadcast_data, src=last_rank_in_group)
+        assert model_output_broadcast_data is not None
path: vllm/v1/worker/gpu_model_runner.py
user: login: houseroad
body: should we return earlier if it's not first rank?
created_at: 2025-05-12T05:46:33Z
author_association: COLLABORATOR
diff_hunk: @@ -1645,7 +1645,9 @@ def data_parallel_rank_local(self, value: int) -> None:
     """Port of the data parallel master."""
     enable_expert_parallel: bool = False
     """Use expert parallelism instead of tensor parallelism for MoE layers."""
-
+    pipeline_parallel_broadcast_output: bool = False
path: vllm/config.py
user: login: houseroad
body: Shall we just use pp_rank() > 0 + external_laucher == True to replace this flag?
created_at: 2025-05-12T05:47:57Z
author_association: COLLABORATOR
diff_hunk: @@ -1645,7 +1645,9 @@ def data_parallel_rank_local(self, value: int) -> None:
     """Port of the data parallel master."""
     enable_expert_parallel: bool = False
     """Use expert parallelism instead of tensor parallelism for MoE layers."""
-
+    pipeline_parallel_broadcast_output: bool = False
path: vllm/config.py
user: login: luccafong
body: Sure, removed the flag now.
created_at: 2025-05-12T17:19:35Z
author_association: COLLABORATOR
diff_hunk: @@ -1136,13 +1137,27 @@ def execute_model(
             hidden_states, aux_hidden_states = output
         else:
             hidden_states = output
-
         if not get_pp_group().is_last_rank:
             # For mid-pipeline stages, return the hidden states.
-            return hidden_states
-
-        sample_hidden_states = hidden_states[logits_indices]
-        logits = self.model.compute_logits(sample_hidden_states, None)
+            if not self.parallel_config.pipeline_parallel_broadcast_output:
+                return hidden_states
+            assert isinstance(hidden_states, IntermediateTensors)
+            get_pp_group().send_tensor_dict(hidden_states.tensors,
+                                            all_gather_group=get_tp_group())
+            model_output_broadcast_data = {}
+        else:
+            sample_hidden_states = hidden_states[logits_indices]
+            logits = self.model.compute_logits(sample_hidden_states, None)
+            model_output_broadcast_data = {
+                "logits": logits,
+            }
+        pp_group_ranks = len(get_pp_group().ranks)
+        last_rank_in_group = pp_group_ranks - 1
+        if len(get_pp_group().ranks) > 0:
+            model_output_broadcast_data = get_pp_group().broadcast_tensor_dict(
+                model_output_broadcast_data, src=last_rank_in_group)
+        assert model_output_broadcast_data is not None
path: vllm/v1/worker/gpu_model_runner.py
user: login: luccafong
body: still need it for complete sync in current solution
created_at: 2025-05-12T17:31:41Z
author_association: COLLABORATOR
diff_hunk: @@ -1171,10 +1172,27 @@ def execute_model(
 
         if not get_pp_group().is_last_rank:
             # For mid-pipeline stages, return the hidden states.
-            return hidden_states
-
-        sample_hidden_states = hidden_states[logits_indices]
-        logits = self.model.compute_logits(sample_hidden_states, None)
+            if self.parallel_config.distributed_executor_backend \
+                != "external_launcher":
+                return hidden_states
+            assert isinstance(hidden_states, IntermediateTensors)
+            get_pp_group().send_tensor_dict(hidden_states.tensors,
+                                            all_gather_group=get_tp_group())
+            model_output_broadcast_data = {}
+        else:
+            sample_hidden_states = hidden_states[logits_indices]
+            logits = self.model.compute_logits(sample_hidden_states, None)
+            model_output_broadcast_data = {
+                "logits": logits,
+            }
+        pp_group_ranks = len(get_pp_group().ranks)
+        last_rank_in_group = pp_group_ranks - 1
+        if self.parallel_config.distributed_executor_backend \
+            == "external_launcher" and len(get_pp_group().ranks) > 0:
+            model_output_broadcast_data = get_pp_group().broadcast_tensor_dict(
path: vllm/v1/worker/gpu_model_runner.py
user: login: ruisearch42
body: Could you explain why we need this broadcast?
created_at: 2025-05-12T23:25:01Z
author_association: COLLABORATOR
diff_hunk: @@ -1327,9 +1327,10 @@ def _is_v1_supported_oracle(self, model_config: ModelConfig) -> bool:
             return False
 
         if (self.pipeline_parallel_size > 1
-                and self.distributed_executor_backend not in ["ray", "mp"]):
+                and self.distributed_executor_backend
path: vllm/engine/arg_utils.py
user: login: ruisearch42
body: Could we add a unit test? There is test_pipeline_parallel.py
created_at: 2025-05-12T23:25:59Z
author_association: COLLABORATOR
diff_hunk: @@ -1327,9 +1327,10 @@ def _is_v1_supported_oracle(self, model_config: ModelConfig) -> bool:
             return False
 
         if (self.pipeline_parallel_size > 1
-                and self.distributed_executor_backend not in ["ray", "mp"]):
+                and self.distributed_executor_backend
path: vllm/engine/arg_utils.py
user: login: luccafong
body: test_pipeline_parallel.py mainly for online serving, and we need torchrun for the tests, I added in test_torchrun_examples
created_at: 2025-05-13T02:20:42Z
author_association: COLLABORATOR
diff_hunk: @@ -1171,10 +1172,27 @@ def execute_model(
 
         if not get_pp_group().is_last_rank:
             # For mid-pipeline stages, return the hidden states.
-            return hidden_states
-
-        sample_hidden_states = hidden_states[logits_indices]
-        logits = self.model.compute_logits(sample_hidden_states, None)
+            if self.parallel_config.distributed_executor_backend \
+                != "external_launcher":
+                return hidden_states
+            assert isinstance(hidden_states, IntermediateTensors)
+            get_pp_group().send_tensor_dict(hidden_states.tensors,
+                                            all_gather_group=get_tp_group())
+            model_output_broadcast_data = {}
+        else:
+            sample_hidden_states = hidden_states[logits_indices]
+            logits = self.model.compute_logits(sample_hidden_states, None)
+            model_output_broadcast_data = {
+                "logits": logits,
+            }
+        pp_group_ranks = len(get_pp_group().ranks)
+        last_rank_in_group = pp_group_ranks - 1
+        if self.parallel_config.distributed_executor_backend \
+            == "external_launcher" and len(get_pp_group().ranks) > 0:
+            model_output_broadcast_data = get_pp_group().broadcast_tensor_dict(
path: vllm/v1/worker/gpu_model_runner.py
user: login: luccafong
body: Yes, added as comments, now we enable by sync all ranks, will improve to reduce pp bubles in following PR.
created_at: 2025-05-13T02:21:23Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-08T04:20:10Z
body: 👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀
author_association: NONE
user: login: mergify[bot]
created_at: 2025-05-12T16:48:07Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @luccafong.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: luccafong
created_at: 2025-05-12T17:30:31Z
body: > for efficient pp, we would want different stages of pp from different batches run concurrently. would this broadcast make them serialized? like there will be only one batch and one stage running for the full engine.

thanks @youkaichao,  right now running torch.run in a pure synced way to align with SPMD, so this is not the ideal solution for efficient PP, only one batch and one stage. I will think about how to design and improve it so we have multi-batches can overlap while compatible with torch.run as a follow up.
author_association: COLLABORATOR
user: login: mergify[bot]
created_at: 2025-05-14T03:04:12Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @luccafong.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: luccafong
created_at: 2025-05-15T16:23:26Z
body: CI build failure not related, also pulled latest change, @houseroad 
author_association: COLLABORATOR
user: login: houseroad
created_at: 2025-05-15T17:57:39Z
body: Actually v1 test failure may be related:  E       AssertionError: pipeline model parallel group is not initialized

cc: @luccafong 
author_association: COLLABORATOR
user: login: luccafong
created_at: 2025-05-15T19:23:13Z
body: > Actually v1 test failure may be related: E AssertionError: pipeline model parallel group is not initialized
> 
> cc: @luccafong

this is a trunk failure

Added a fix here:
https://github.com/vllm-project/vllm/pull/18223
author_association: COLLABORATOR