Repository: vllm-project/vllm
Pull request: [bugfix] add supports_v1 platform interface
This PR re-allows platform plugins to run on V1. I'm assuming that all out-of-tree platforms are responsible for all of their own validation at startup to ensure the provided configuration will work, so vllm shouldn't disallow them from running.

Our plugin currently doesn't boot with v0.8.1 (and presumably won't with the upcoming v0.8.2), this little fix allows it to start again.

FIX https://github.com/vllm-project/vllm-spyre/issues/38



Problem statement:
Plugin won't boot with vllm v0.8.1
Trying to boot with vllm 0.8.1 with `VLLM_SPYRE_DYNAMO_BACKEND=eager` and `VLLM_USE_V1=1` gives the fun error:

```
INFO 03-19 22:31:45 [config.py:2595] Downcasting torch.float32 to torch.float16.
INFO 03-19 22:31:49 [config.py:583] This model supports multiple tasks: {'score', 'reward', 'classify', 'embed', 'generate'}. Defaulting to 'generate'.
Traceback (most recent call last):
  File "/home/senuser/my-vllm/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/cli/main.py", line 75, in main
    args.dispatch_function(args)
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/cli/serve.py", line 33, in cmd
    uvloop.run(run_server(args))
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1012, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/lib64/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 141, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib64/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 161, in build_async_engine_client_from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/engine/arg_utils.py", line 1215, in create_engine_config
    if try_v1 and self._is_v1_supported_oracle(model_config):
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/engine/arg_utils.py", line 1584, in _is_v1_supported_oracle
    _raise_or_fallback(
  File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/engine/arg_utils.py", line 1758, in _raise_or_fallback
    raise NotImplementedError(
NotImplementedError: VLLM_USE_V1=1 is not supported with device type=cpu.
```

Hints from issue discussion:

Commit message: :bug: re-allow OOT platforms on V1

Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

Patch to review:
diff --git a/vllm/engine/arg_utils.py b/vllm/engine/arg_utils.py
index 38a47a846df7..8d96c836fbdc 100644
--- a/vllm/engine/arg_utils.py
+++ b/vllm/engine/arg_utils.py
@@ -1670,8 +1670,10 @@ def _is_v1_supported_oracle(self, model_config: ModelConfig) -> bool:
             return False
 
         # No support for device type other than CUDA, AMD (experiemntal) or
-        # TPU (experimental) so far.
-        if not (current_platform.is_cuda_alike() or current_platform.is_tpu()):
+        # TPU (experimental) so far. Out-of-tree device support plugins can
+        # maintain their own v1 compatibility checks.
+        if not (current_platform.is_cuda_alike() or current_platform.is_tpu()
+                or current_platform.is_out_of_tree()):
             _raise_or_fallback(
                 feature_name=f"device type={current_platform.device_type}",
                 recommend_to_remove=False)


Code review comments:
diff_hunk: @@ -1670,8 +1670,10 @@ def _is_v1_supported_oracle(self, model_config: ModelConfig) -> bool:
             return False
 
         # No support for device type other than CUDA, AMD (experiemntal) or
-        # TPU (experimental) so far.
-        if not (current_platform.is_cuda_alike() or current_platform.is_tpu()):
+        # TPU (experimental) so far. Out-of-tree device support plugins can
+        # maintain their own v1 compatibility checks.
+        if not (current_platform.is_cuda_alike() or current_platform.is_tpu()
+                or current_platform.is_out_of_tree()):
path: vllm/engine/arg_utils.py
text: @user1:
Maybe we should add a `supports_v1()` to `Platform` (defaulting to false) instead of this, so we don't break fallbacks for other out-of-tree platforms

@author:
Yeah I'd be fine with that as well. I didn't do that here at first because I didn't want to push this centralized bit of logic back out to the different platforms in-tree, but that's not a huge deal.

> so we don't break fallbacks

Yeah... I get that we don't want to turn on V1 where it won't work, but this explicitly denies it where in our case it does work! 😅 It should be the responsibility of the plugin to determine compatibility, so I'm fine with either this or a `platform.supports_v1()` flag. I'l leave some time for more comments and check back on this tomorrow morning

@user2:
I have a comment here https://github.com/vllm-project/vllm/issues/15263#issuecomment-2745034522

we should add `platform.supports_v1()`

@user2:
in addition, to give oot platforms the same ability of falling back, we should pass the config object to the `supports_v1` function.

@user3:
> in addition, to give oot platforms the same ability of falling back, we should pass the config object to the `supports_v1` function.

Agree 👍 

@author:
Cool, on it 👍