Repository: vllm-project/vllm
Issue category: Feature Request

Issue:
[Feature]: Allow Ray placement group more time to wait for resources to be ready
### 🚀 The feature, motivation and pitch

```
ray start --head --dashboard-host 0.0.0.0 --disable-usage-stats --block  --num-gpus 1

python -m vllm.entrypoints.openai.api_server     --model Qwen/Qwen2.5-1.5B-Instruct     --trust-remote-code     --distributed-executor-backend ray     --tensor-parallel-size 2
```

I am using a ray cluster as the distributed executor backend. However, my cluster sometimes takes time to be ready. there are multiple reasons
- image downloading speed is different, some node has image cache and some does not.
- node short of resources, node level autoscaler will make decision to bring up more nodes but it takes time for new ray node to join and form a cluster

I do not want the engine to early exit. 

![image](https://github.com/user-attachments/assets/7a621963-21fb-4f16-9573-71299e951bad)

![image](https://github.com/user-attachments/assets/5b06c173-1465-4675-a5a0-2627a3b8b18d)


### Alternatives

_No response_

### Additional context

_No response_

### Before submitting a new issue...

- [X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.


Hints:

Patch:
diff --git a/vllm/executor/ray_utils.py b/vllm/executor/ray_utils.py
index 4f28efd639084..426aa1b5c728f 100644
--- a/vllm/executor/ray_utils.py
+++ b/vllm/executor/ray_utils.py
@@ -277,10 +277,14 @@ def initialize_ray_cluster(
                 f"Total number of devices: {device_bundles}.")
     else:
         num_devices_in_cluster = ray.cluster_resources().get(device_str, 0)
+        # Log a warning message and delay resource allocation failure response.
+        # Avoid immediate rejection to allow user-initiated placement group
+        # created and wait cluster to be ready
         if parallel_config.world_size > num_devices_in_cluster:
-            raise ValueError(
-                f"The number of required {device_str}s exceeds the total "
-                f"number of available {device_str}s in the placement group.")
+            logger.warning(
+                "The number of required %ss exceeds the total "
+                "number of available %ss in the placement group.", device_str,
+                device_str)
         # Create a new placement group
         placement_group_specs: List[Dict[str, float]] = ([{
             device_str: 1.0


Test patch: