Repository: vllm-project/vllm-spyre
Pull request #157: :memo: Document plugin configuration
Author: login: joerunde
Created: 2025-05-13T22:49:05Z | Merged: 2025-05-14T20:25:52Z

This PR exposes our `envs.py` file in our docs, and adds a page describing how to configure the plugin


Diff:
diff --git a/docs/source/index.md b/docs/source/index.md
index 51e459074..4a9eecbf0 100644
--- a/docs/source/index.md
+++ b/docs/source/index.md
@@ -22,6 +22,8 @@ installation
 :::{toctree}
 :caption: User Guide
 :maxdepth: 1
+user_guide/configuration
+user_guide/env_vars
 user_guide/supported_features
 user_guide/supported_models
 :::
diff --git a/docs/source/user_guide/configuration.md b/docs/source/user_guide/configuration.md
new file mode 100644
index 000000000..e6c3193cc
--- /dev/null
+++ b/docs/source/user_guide/configuration.md
@@ -0,0 +1,53 @@
+# Configuration
+
+For a complete list of configuration options, see <project:./env_vars.md>.
+
+## Backend Selection
+
+The torch.compile backend can be configured with the `VLLM_SPYRE_DYNAMO_BACKEND` environment variable.
+
+All models can be tested on CPU by setting this to `eager`.
+To run inference on IBM Spyre Accelerators, the backend should be set as:
+
+| Model type | vLLM backend | `VLLM_SPYRE_DYNAMO_BACKEND` configuration | Notes |
+| --- | --- | --- | --- |
+| Decoder | v0 | sendnn_decoder | V0 support for decoder models is deprecated |
+| Decoder | v1 | sendnn_decoder | |
+| Embedding | v0 | sendnn | |
+| Embedding | v1 | N/A | Embedding models are not yet supported on V1 |
+
+## Batching Modes
+
+When running decoder models, vLLM Spyre supports a static batching mode and a continuous batching mode.
+
+### Static Batching
+
+With static batching, graphs are pre-compiled for the configured batch shapes and each batch must finish processing before a new batch can be scheduled. This adds extra constraints on the sizes of inputs and outputs for each request, and requests that do not fit the precompiled graphs will be rejected.
+
+Static batching mode is enabled by default, and can be explicitly enabled by setting `VLLM_USE_CB=0`.
+
+:::{caution}
+There are no up-front checks that the compiled graphs will fit into the available memory on the Spyre cards. If the graphs are too large for the available memory, vllm will crash during model warmup.
+:::
+
+The batch shapes are configured with the `VLLM_SPYRE_WARMUP_*` environment variables. For example, to warm up two graph shapes for one single large request and four smaller requests you could use:
+
+```shell
+export VLLM_SPYRE_WARMUP_BATCH_SIZES=1,4
+export VLLM_SPYRE_WARMUP_PROMPT_LENS=4096,1024
+export VLLM_SPYRE_WARMUP_NEW_TOKENS=1024,256
+```
+
+### Continuous Batching
+
+:::{attention}
+Continuous batching is not currently supported on IBM Spyre Accelerators. A CPU-only implementation is available by setting `VLLM_SPYRE_DYNAMO_BACKEND=eager`. Continuous batching can be enabled with `VLLM_USE_CB=1`.
+:::
+
+Continuous batching works much more like other accelerator implementations on vLLM. Requests can be continually appended to a running batch, and requests that finish generating can be evicted from the batch to make room for more requests. Neither chunked prefill nor prefix caching are currently supported though, so when a request is added to the running batch it must first be paused for a full prefill of the incoming prompt.
+
+## Caching Compiled Graphs
+
+`torch_sendnn` supports caching compiled model graphs, which can vastly speed up warmup time when loading models in a distributed setting.
+
+To enable this, set `TORCH_SENDNN_CACHE_ENABLE=1` and configure `TORCH_SENDNN_CACHE_DIR` to a directory to hold the cache files. By default, this feature is disabled.
diff --git a/docs/source/user_guide/env_vars.md b/docs/source/user_guide/env_vars.md
new file mode 100644
index 000000000..d32f0ebf7
--- /dev/null
+++ b/docs/source/user_guide/env_vars.md
@@ -0,0 +1,9 @@
+# Environment Variables
+
+vLLM Spyre uses the following environment variables to configure the system:
+
+:::{literalinclude} ../../../vllm_spyre/envs.py
+:end-before: end-env-vars-definition
+:language: python
+:start-after: begin-env-vars-definition
+:::
diff --git a/vllm_spyre/envs.py b/vllm_spyre/envs.py
index 9fa4a3066..2c51d7e5e 100644
--- a/vllm_spyre/envs.py
+++ b/vllm_spyre/envs.py
@@ -11,23 +11,28 @@
     VLLM_SPYRE_PERF_METRIC_LOGGING_ENABLED: int = 0
     VLLM_SPYRE_PERF_METRIC_LOGGING_DIR: str = "/tmp"
 
+# begin-env-vars-definition
+
 environment_variables: dict[str, Callable[[], Any]] = {
     # Defines the prompt lengths the Spyre accelerator should be prepared
-    # for, formatted as comma separated list.
+    # for, formatted as comma separated list. Only applicable in static batching
+    # mode (VLLM_SPYRE_USE_CB=0).
     "VLLM_SPYRE_WARMUP_PROMPT_LENS":
     lambda: [
         int(p) for p in os.getenv(key='VLLM_SPYRE_WARMUP_PROMPT_LENS',
                                   default='64').split(',')
     ],
     # Defines the max output tokens the Spyre accelerator should be prepared
-    # for, formatted as comma separated list.
+    # for, formatted as comma separated list. Only applicable in static batching
+    # mode (VLLM_SPYRE_USE_CB=0).
     "VLLM_SPYRE_WARMUP_NEW_TOKENS":
     lambda: [
         int(d) for d in os.getenv(key='VLLM_SPYRE_WARMUP_NEW_TOKENS',
                                   default='20').split(',')
     ],
     # Defines the batch sizes the Spyre accelerator should be prepared
-    # for, formatted as comma separated list.
+    # for, formatted as comma separated list. Only applicable in static batching
+    # mode (VLLM_SPYRE_USE_CB=0).
     "VLLM_SPYRE_WARMUP_BATCH_SIZES":
     lambda: [
         int(b) for b in os.getenv(key='VLLM_SPYRE_WARMUP_BATCH_SIZES',
@@ -41,15 +46,17 @@
     # - "sendnn": Compile for execution on Spyre hardware for
     #   encoder models
     # - "inductor": Compile for execution on CPU (for debug and testing)
-    # - "eager": Skip compile entirely (for debug and testing
+    # - "eager": Skip compile entirely (for debug and testing)
     "VLLM_SPYRE_DYNAMO_BACKEND":
     lambda: os.getenv("VLLM_SPYRE_DYNAMO_BACKEND", "sendnn_decoder"),
 
-    # If set, use the V1 continuous batching implementation
+    # If set, use the V1 continuous batching implementation. Otherwise, static
+    # batching mode will be enabled.
     "VLLM_SPYRE_USE_CB":
     lambda: bool(int(os.getenv("VLLM_SPYRE_USE_CB", "0"))),
 
-    # If set, remove redundant (left) padded blocks
+    # If set, remove redundant (left) padded blocks. Only applicable in
+    # continuous batching mode.
     "VLLM_SPYRE_RM_PADDED_BLOCKS":
     lambda: bool(int(os.getenv("VLLM_SPYRE_RM_PADDED_BLOCKS", "0"))),
 
@@ -64,6 +71,8 @@
     lambda: os.getenv("VLLM_SPYRE_PERF_METRIC_LOGGING_DIR", "/tmp"),
 }
 
+# end-env-vars-definition
+
 
 def __getattr__(name: str):
     # lazy evaluation of environment variables


Reviews:
user: login: rafvasq
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T00:34:28Z
user: login: joerunde
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T16:20:45Z
user: login: joerunde
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T16:21:14Z
user: login: rafvasq
body: Just two missing periods and LGTM!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-14T19:58:43Z

Review comments:
diff_hunk: @@ -0,0 +1,47 @@
+# Configuration
+
+For a complete list of configuration options, see :link: ./env_vars :link-type: doc
+
+## Backend selection
path: docs/source/user_guide/configuration.md
user: login: rafvasq
body: ```suggestion
## Backend Selection
```
created_at: 2025-05-14T00:14:53Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+# Configuration
+
+For a complete list of configuration options, see :link: ./env_vars :link-type: doc
+
+## Backend selection
+
+The torch.compile backend can be configured with the `VLLM_SPYRE_DYNAMO_BACKEND` environment variable.
+
+All models can be tested on CPU by setting this to `eager`.
+To run inference on IBM Spyre Accelerators, the backend should be set as:
+
+| Model type | vLLM backend | `VLLM_SPYRE_DYNAMO_BACKEND` configuration | Notes |
+| --- | --- | --- | --- |
+| Decoder | v0 | sendnn_decoder | V0 support for decoder models is deprecated |
+| Decoder | v1 | sendnn_decoder | |
+| Embedding | v0 | sendnn | |
+| Embedding | v1 | N/A | Embedding models are not yet supported on V1 |
+
+## Batching modes
path: docs/source/user_guide/configuration.md
user: login: rafvasq
body: ```suggestion
## Batching Modes
```
created_at: 2025-05-14T00:15:02Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+# Configuration
+
+For a complete list of configuration options, see :link: ./env_vars :link-type: doc
+
+## Backend selection
+
+The torch.compile backend can be configured with the `VLLM_SPYRE_DYNAMO_BACKEND` environment variable.
+
+All models can be tested on CPU by setting this to `eager`.
+To run inference on IBM Spyre Accelerators, the backend should be set as:
+
+| Model type | vLLM backend | `VLLM_SPYRE_DYNAMO_BACKEND` configuration | Notes |
+| --- | --- | --- | --- |
+| Decoder | v0 | sendnn_decoder | V0 support for decoder models is deprecated |
+| Decoder | v1 | sendnn_decoder | |
+| Embedding | v0 | sendnn | |
+| Embedding | v1 | N/A | Embedding models are not yet supported on V1 |
+
+## Batching modes
+
+When running decoder models, vLLM Spyre supports a static batching mode and a continuous batching mode.
+
+### Static batching
path: docs/source/user_guide/configuration.md
user: login: rafvasq
body: ```suggestion
### Static Batching
```
created_at: 2025-05-14T00:15:08Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+# Configuration
+
+For a complete list of configuration options, see :link: ./env_vars :link-type: doc
path: docs/source/user_guide/configuration.md
user: login: rafvasq
body: This link didn't render properly, and I think it should be done using one of the options [here](https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html).
created_at: 2025-05-14T00:26:10Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+# Configuration
+
+For a complete list of configuration options, see :link: ./env_vars :link-type: doc
+
+## Backend selection
+
+The torch.compile backend can be configured with the `VLLM_SPYRE_DYNAMO_BACKEND` environment variable.
+
+All models can be tested on CPU by setting this to `eager`.
+To run inference on IBM Spyre Accelerators, the backend should be set as:
+
+| Model type | vLLM backend | `VLLM_SPYRE_DYNAMO_BACKEND` configuration | Notes |
+| --- | --- | --- | --- |
+| Decoder | v0 | sendnn_decoder | V0 support for decoder models is deprecated |
+| Decoder | v1 | sendnn_decoder | |
+| Embedding | v0 | sendnn | |
+| Embedding | v1 | N/A | Embedding models are not yet supported on V1 |
+
+## Batching modes
+
+When running decoder models, vLLM Spyre supports a static batching mode and a continuous batching mode.
+
+### Static batching
+
+With static batching, graphs are pre-compiled for the configured batch shapes and each batch must finish processing before a new batch can be scheduled. This adds extra constraints on the sizes of inputs and outputs for each request, and requests that do not fit the precompiled graphs will be rejected.
+
+Static batching mode is enabled by default, and can be explicitly enabled by setting `VLLM_USE_CB=0`.
+
+The batch shapes are configured with the `VLLM_SPYRE_WARMUP_*` environment variables. For example, to warm up two graph shapes for one single large requests and four smaller requests you could use:
path: docs/source/user_guide/configuration.md
user: login: rafvasq
body: ```suggestion
The batch shapes are configured with the `VLLM_SPYRE_WARMUP_*` environment variables. For example, to warm up two graph shapes for one single large request and four smaller requests you could use:
```
created_at: 2025-05-14T00:28:15Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+# Configuration
+
+For a complete list of configuration options, see :link: ./env_vars :link-type: doc
+
+## Backend selection
+
+The torch.compile backend can be configured with the `VLLM_SPYRE_DYNAMO_BACKEND` environment variable.
+
+All models can be tested on CPU by setting this to `eager`.
+To run inference on IBM Spyre Accelerators, the backend should be set as:
+
+| Model type | vLLM backend | `VLLM_SPYRE_DYNAMO_BACKEND` configuration | Notes |
+| --- | --- | --- | --- |
+| Decoder | v0 | sendnn_decoder | V0 support for decoder models is deprecated |
+| Decoder | v1 | sendnn_decoder | |
+| Embedding | v0 | sendnn | |
+| Embedding | v1 | N/A | Embedding models are not yet supported on V1 |
+
+## Batching modes
+
+When running decoder models, vLLM Spyre supports a static batching mode and a continuous batching mode.
+
+### Static batching
+
+With static batching, graphs are pre-compiled for the configured batch shapes and each batch must finish processing before a new batch can be scheduled. This adds extra constraints on the sizes of inputs and outputs for each request, and requests that do not fit the precompiled graphs will be rejected.
+
+Static batching mode is enabled by default, and can be explicitly enabled by setting `VLLM_USE_CB=0`.
+
+The batch shapes are configured with the `VLLM_SPYRE_WARMUP_*` environment variables. For example, to warm up two graph shapes for one single large requests and four smaller requests you could use:
+
+```shell
+export VLLM_SPYRE_WARMUP_BATCH_SIZES=1,4
+export VLLM_SPYRE_WARMUP_PROMPT_LENS=4096,1024
+export VLLM_SPYRE_WARMUP_NEW_TOKENS=1024,256
+```
+
+### Continuous Batching
+
+NB: Continuous batching is not currently supported on IBM Spyre Accelerators. A CPU-only implementation is available by setting `VLLM_SPYRE_DYNAMO_BACKEND=eager`. Continuous batching can be enabled with `VLLM_USE_CB=1`
path: docs/source/user_guide/configuration.md
user: login: rafvasq
body: Maybe this can be highlighted as a [callout](https://myst-parser.readthedocs.io/en/latest/syntax/admonitions.html#admonition-types) instead of using NB
created_at: 2025-05-14T00:32:12Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+# Configuration
+
+For a complete list of configuration options, see :link: ./env_vars :link-type: doc
+
+## Backend selection
+
+The torch.compile backend can be configured with the `VLLM_SPYRE_DYNAMO_BACKEND` environment variable.
+
+All models can be tested on CPU by setting this to `eager`.
+To run inference on IBM Spyre Accelerators, the backend should be set as:
+
+| Model type | vLLM backend | `VLLM_SPYRE_DYNAMO_BACKEND` configuration | Notes |
+| --- | --- | --- | --- |
+| Decoder | v0 | sendnn_decoder | V0 support for decoder models is deprecated |
+| Decoder | v1 | sendnn_decoder | |
+| Embedding | v0 | sendnn | |
+| Embedding | v1 | N/A | Embedding models are not yet supported on V1 |
+
+## Batching modes
+
+When running decoder models, vLLM Spyre supports a static batching mode and a continuous batching mode.
+
+### Static batching
+
+With static batching, graphs are pre-compiled for the configured batch shapes and each batch must finish processing before a new batch can be scheduled. This adds extra constraints on the sizes of inputs and outputs for each request, and requests that do not fit the precompiled graphs will be rejected.
+
+Static batching mode is enabled by default, and can be explicitly enabled by setting `VLLM_USE_CB=0`.
+
+The batch shapes are configured with the `VLLM_SPYRE_WARMUP_*` environment variables. For example, to warm up two graph shapes for one single large requests and four smaller requests you could use:
+
+```shell
+export VLLM_SPYRE_WARMUP_BATCH_SIZES=1,4
+export VLLM_SPYRE_WARMUP_PROMPT_LENS=4096,1024
+export VLLM_SPYRE_WARMUP_NEW_TOKENS=1024,256
+```
+
+### Continuous Batching
+
+NB: Continuous batching is not currently supported on IBM Spyre Accelerators. A CPU-only implementation is available by setting `VLLM_SPYRE_DYNAMO_BACKEND=eager`. Continuous batching can be enabled with `VLLM_USE_CB=1`
path: docs/source/user_guide/configuration.md
user: login: joerunde
body: ah nice, yeah that's much better
created_at: 2025-05-14T16:20:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,47 @@
+# Configuration
+
+For a complete list of configuration options, see :link: ./env_vars :link-type: doc
path: docs/source/user_guide/configuration.md
user: login: joerunde
body: Thanks! I didn't know how links work here and just tried to follow an example from vllm
created_at: 2025-05-14T16:21:14Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,53 @@
+# Configuration
+
+For a complete list of configuration options, see <project:./env_vars.md>
path: docs/source/user_guide/configuration.md
user: login: rafvasq
body: ```suggestion
For a complete list of configuration options, see <project:./env_vars.md>.
```
created_at: 2025-05-14T19:57:15Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,53 @@
+# Configuration
+
+For a complete list of configuration options, see <project:./env_vars.md>
+
+## Backend Selection
+
+The torch.compile backend can be configured with the `VLLM_SPYRE_DYNAMO_BACKEND` environment variable.
+
+All models can be tested on CPU by setting this to `eager`.
+To run inference on IBM Spyre Accelerators, the backend should be set as:
+
+| Model type | vLLM backend | `VLLM_SPYRE_DYNAMO_BACKEND` configuration | Notes |
+| --- | --- | --- | --- |
+| Decoder | v0 | sendnn_decoder | V0 support for decoder models is deprecated |
+| Decoder | v1 | sendnn_decoder | |
+| Embedding | v0 | sendnn | |
+| Embedding | v1 | N/A | Embedding models are not yet supported on V1 |
+
+## Batching Modes
+
+When running decoder models, vLLM Spyre supports a static batching mode and a continuous batching mode.
+
+### Static Batching
+
+With static batching, graphs are pre-compiled for the configured batch shapes and each batch must finish processing before a new batch can be scheduled. This adds extra constraints on the sizes of inputs and outputs for each request, and requests that do not fit the precompiled graphs will be rejected.
+
+Static batching mode is enabled by default, and can be explicitly enabled by setting `VLLM_USE_CB=0`.
+
+:::{caution}
+There are no up-front checks that the compiled graphs will fit into the available memory on the Spyre cards. If the graphs are too large for the available memory, vllm will crash during model warmup.
+:::
+
+The batch shapes are configured with the `VLLM_SPYRE_WARMUP_*` environment variables. For example, to warm up two graph shapes for one single large request and four smaller requests you could use:
+
+```shell
+export VLLM_SPYRE_WARMUP_BATCH_SIZES=1,4
+export VLLM_SPYRE_WARMUP_PROMPT_LENS=4096,1024
+export VLLM_SPYRE_WARMUP_NEW_TOKENS=1024,256
+```
+
+### Continuous Batching
+
+:::{attention}
+Continuous batching is not currently supported on IBM Spyre Accelerators. A CPU-only implementation is available by setting `VLLM_SPYRE_DYNAMO_BACKEND=eager`. Continuous batching can be enabled with `VLLM_USE_CB=1`
path: docs/source/user_guide/configuration.md
user: login: rafvasq
body: ```suggestion
Continuous batching is not currently supported on IBM Spyre Accelerators. A CPU-only implementation is available by setting `VLLM_SPYRE_DYNAMO_BACKEND=eager`. Continuous batching can be enabled with `VLLM_USE_CB=1`.
```
created_at: 2025-05-14T19:58:21Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-13T22:49:14Z
body: 👋 Hi! Thank you for contributing to vLLM support on Spyre.
 Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run `format.sh` and commit the changes. This can be done with `uv` directly: 
```
uv sync --frozen --group lint
```
 Or this can be done with `pip`: 
```
uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh
```
Now you are good to go 🚀
author_association: NONE
user: login: joerunde
created_at: 2025-05-13T22:54:07Z
body: Preview docs are up here: https://vllm-spyre--157.org.readthedocs.build/en/157/
author_association: COLLABORATOR
user: login: joerunde
created_at: 2025-05-14T19:59:53Z
body: I am far too young to put periods everywhere by default 🙃 
author_association: COLLABORATOR
user: login: rafvasq
created_at: 2025-05-14T20:13:02Z
body: @joerunde I might start asking for [two spaces after periods](https://www.cultofpedagogy.com/two-spaces-after-period/) too 😆 
author_association: COLLABORATOR
user: login: joerunde
created_at: 2025-05-14T20:19:56Z
body: > I might start asking for [two spaces after periods](https://www.cultofpedagogy.com/two-spaces-after-period/) too 😆



![nooooo](https://media2.giphy.com/media/v1.Y2lkPTc5MGI3NjExbGpzZWtiMHNpaDF3cW96Mmp2OTRia2N6aDJnaG9neGl2M3phY2cyNiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/8vUEXZA2me7vnuUvrs/giphy.gif)

author_association: COLLABORATOR