Repository: vllm-project/vllm
Pull request: [Hardware] add platform-specific request validation api
This PR adds a generic `validate_request` api to the platform interface. This allows platforms to implement any runtime checks on each request to ensure that all the requested features are supported before scheduling it. There is already one existing check in this category, `supports_structured_output`, and I'd like to avoid a proliferation of more platform apis for individual features like this.

Currently, the spyre plugin needs to implement some extra validation around the shape of inputs, as we have more constraints on valid prompt lengths and max token requests. This new api would let us do that without needing to hack around rejecting requests from the scheduler.

FIX https://github.com/vllm-project/vllm-spyre/issues/77

<!--- pyml disable-next-line no-emphasis-as-heading -->


Problem statement:
Add platform api for request validation to reject requests that don't fit warmup shapes
From talking to @njhill, there's already precedent for doing hardware-specific request validation with the `supports_structured_output` platform api. For spyre we need a way to reject requests that don't fit a warmup shape, so we should propose a more general api to validate requests up front. Maybe something like:

```
def validate_request(
        cls,
        prompt: PromptType,
        params: Union[SamplingParams, PoolingParams],
        lora_request: Optional[LoRARequest] = None,
) -> None:
        """Raises if this request is unsupported on this platform"""
```

Hints from issue discussion:

Commit message: :sparkles: add platform-specific request validation

Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

Patch to review:
diff --git a/vllm/platforms/interface.py b/vllm/platforms/interface.py
index 2bb543bd73f7..a6425a7ce895 100644
--- a/vllm/platforms/interface.py
+++ b/vllm/platforms/interface.py
@@ -1,5 +1,4 @@
 # SPDX-License-Identifier: Apache-2.0
-
 import enum
 import platform
 import random
@@ -9,10 +8,14 @@
 import numpy as np
 import torch
 
+from vllm.inputs import PromptType
 from vllm.logger import init_logger
 
 if TYPE_CHECKING:
     from vllm.config import ModelConfig, VllmConfig
+    from vllm.lora.request import LoRARequest
+    from vllm.pooling_params import PoolingParams
+    from vllm.sampling_params import SamplingParams
     from vllm.utils import FlexibleArgumentParser
 else:
     ModelConfig = None
@@ -393,6 +396,15 @@ def use_custom_allreduce(cls) -> bool:
         """
         return False
 
+    @classmethod
+    def validate_request(
+        cls,
+        prompt: PromptType,
+        params: Union[SamplingParams, PoolingParams],
+        lora_request: Optional[LoRARequest] = None,
+    ) -> None:
+        """Raises if this request is unsupported on this platform"""
+
 
 class UnspecifiedPlatform(Platform):
     _enum = PlatformEnum.UNSPECIFIED
diff --git a/vllm/v1/engine/processor.py b/vllm/v1/engine/processor.py
index 403edddfcbee..35b436419f41 100644
--- a/vllm/v1/engine/processor.py
+++ b/vllm/v1/engine/processor.py
@@ -183,6 +183,12 @@ def process_inputs(
         # TODO(woosuk): Support pooling models.
         # TODO(woosuk): Support encoder-decoder models.
 
+        from vllm.platforms import current_platform
+        current_platform.validate_request(
+            prompt=prompt,
+            params=params,
+            lora_request=lora_request,
+        )
         self._validate_lora(lora_request)
         self._validate_params(params)
         if priority != 0:


Code review comments:
diff_hunk: @@ -393,6 +396,15 @@ def use_custom_allreduce(cls) -> bool:
         """
         return False
 
+    @classmethod
+    def validate_request(
+        cls,
+        prompt: PromptType,
+        params: Union[SamplingParams, PoolingParams],
+        lora_request: Optional[LoRARequest] = None,
path: vllm/platforms/interface.py
text: @user1:
Do we need to include lora_request here? Wouldn't a platform either support lora or not, and if not isn't this something that could be checked at startup time?

@author:
Ah, yeah that's true. I was thinking there might be a case where something about the adapter needs validation but I think you're right that anything about supporting lora would be checked either at boot time, or at adapter load time