Repository: ray-project/ray
Pull request: [data.llm] Deserialize arrays for sampling_params
<!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. -->

## Why are these changes needed?
This solves the issue described in https://github.com/ray-project/ray/issues/52630
Some vllm sampling params such as `stop_token_ids` are lists and hence get converted to np.arrays when sent from one process to another in ray data. This causes a deserialization error in the vLLM engine since it expects a list, not a np.array.

## Related issue number
Closes #52630 

## Checks
- [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [x] I've included any doc changes needed for https://docs.ray.io/en/master/.
    - [-] I've added any new APIs to the API Reference. For example, if I added a
           method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(


Problem statement:
[`build_llm_processor` throws deserialization error for  `stop_token_ids`: LLM>]
### What happened + What you expected to happen

Thanks for the awesome LLM abstractions. I found a bug in it that needs a simple fix.
Some vllm sampling params such as `stop_token_ids` are lists and hence get converted to np.arrays. We need to call the `self._maybe_convert_ndarray_to_list` on the sampling params to deserialize them properly before passing to vLLM so that the engine doesn't error. I have opened a PR for this fix!


Script to replicate:
`python bug.py`
```python
import ray
from ray.data.llm import vLLMEngineProcessorConfig, build_llm_processor
import numpy as np

config = vLLMEngineProcessorConfig(
    model_source="unsloth/Llama-3.1-8B-Instruct",
    engine_kwargs={
        "enable_chunked_prefill": True,
        "max_num_batched_tokens": 4096,
        "max_model_len": 16384,
    },
    concurrency=1,
    batch_size=64,
)
processor = build_llm_processor(
    config,
    preprocess=lambda row: dict(
        messages=[
            {"role": "system", "content": "You are a bot that responds with haikus."},
            {"role": "user", "content": row["item"]}
        ],
        sampling_params=dict(
            temperature=0.3,
            max_tokens=250,
            stop_token_ids=[1,2,3], # NOTE: this gets serialized to a np.array
        )
    ),
    postprocess=lambda row: dict(
        answer=row["generated_text"],
        **row  # This will return all the original columns in the dataset.
    ),
)

ds = ray.data.from_items(["Start of the haiku is: Complete this for me..."])

ds = processor(ds)
ds.show(limit=1)
```

```bash
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)   File "/home/arthur/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)     self.run()
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)   File "/home/arthur/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)     self._target(*self._args, **self._kwargs)
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)   File "/home/arthur/code/bug/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 429, in process_input_socket
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)     request = decoder.decode(data_frame.buffer)
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)   File "/home/arthur/code/bug/.venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 34, in decode
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)     return self.decoder.decode(obj)
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323)            ^^^^^^^^^^^^^^^^^^^^^^^^
(MapWorker(MapBatches(vLLMEngineStageUDF)) pid=421323) msgspec.ValidationError: Expected `int`, got `ext` - at `$[6].stop_token_ids[1]`
```

### Versions / Dependencies

```

```

### Reproduction script

uv add ray[data,llm] vllm
```
Package                           Version
--------------------------------- -------------
aiohappyeyeballs                  2.6.1
aiohttp                           3.11.18
aiohttp-cors                      0.8.1
aiosignal                         1.3.2
airportsdata                      20250224
annotated-types                   0.7.0
anyio                             4.9.0
astor                             0.8.1
attrs                             25.3.0
blake3                            1.0.4
cachetools                        5.5.2
certifi                           2025.4.26
charset-normalizer                3.4.1
click                             8.1.8
cloudpickle                       3.1.1
colorful                          0.5.6
compressed-tensors                0.9.2
cupy-cuda12x                      13.4.1
depyf                             0.18.0
dill                              0.4.0
diskcache                         5.6.3
distlib                           0.3.9
distro                            1.9.0
dnspython                         2.7.0
einops                            0.8.1
email-validator                   2.2.0
fastapi                           0.115.12
fastapi-cli                       0.0.7
fastrlock                         0.8.3
filelock                          3.18.0
frozenlist                        1.6.0
fsspec                            2025.3.2
gguf                              0.10.0
google-api-core                   2.24.2
google-auth                       2.39.0
googleapis-common-protos          1.70.0
grpcio                            1.71.0
h11                               0.16.0
httpcore                          1.0.9
httptools                         0.6.4
httpx                             0.28.1
huggingface-hub                   0.30.2
idna                              3.10
importlib-metadata                8.6.1
interegular                       0.3.3
jinja2                            3.1.6
jiter                             0.9.0
jsonref                           1.1.0
jsonschema                        4.23.0
jsonschema-specifications         2025.4.1
lark                              1.2.2
llguidance                        0.7.19
llvmlite                          0.43.0
lm-format-enforcer                0.10.11
markdown-it-py                    3.0.0
markupsafe                        3.0.2
mdurl                             0.1.2
mistral-common                    1.5.4
mpmath                            1.3.0
msgpack                           1.1.0
msgspec                           0.19.0
multidict                         6.4.3
nest-asyncio                      1.6.0
networkx                          3.4.2
ninja                             1.11.1.4
numba                             0.60.0
numpy                             1.26.4
nvidia-cublas-cu12                12.4.5.8
nvidia-cuda-cupti-cu12            12.4.127
nvidia-cuda-nvrtc-cu12            12.4.127
nvidia-cuda-runtime-cu12          12.4.127
nvidia-cudnn-cu12                 9.1.0.70
nvidia-cufft-cu12                 11.2.1.3
nvidia-curand-cu12                10.3.5.147
nvidia-cusolver-cu12              11.6.1.9
nvidia-cusparse-cu12              12.3.1.170
nvidia-cusparselt-cu12            0.6.2
nvidia-nccl-cu12                  2.21.5
nvidia-nvjitlink-cu12             12.4.127
nvidia-nvtx-cu12                  12.4.127
openai                            1.76.0
opencensus                        0.11.4
opencensus-context                0.1.3
opencv-python-headless            4.11.0.86
outlines                          0.1.11
outlines-core                     0.1.26
packaging                         25.0
pandas                            2.2.3
partial-json-parser               0.2.1.1.post5
pillow                            11.2.1
platformdirs                      4.3.7
prometheus-client                 0.21.1
prometheus-fastapi-instrumentator 7.1.0
propcache                         0.3.1
proto-plus                        1.26.1
protobuf                          6.30.2
psutil                            7.0.0
py-cpuinfo                        9.0.0
py-spy                            0.4.0
pyarrow                           17.0.0
pyasn1                            0.6.1
pyasn1-modules                    0.4.2
pycountry                         24.6.1
pydantic                          2.11.3
pydantic-core                     2.33.1
pygments                          2.19.1
python-dateutil                   2.9.0.post0
python-dotenv                     1.1.0
python-json-logger                3.3.0
python-multipart                  0.0.20
pytz                              2025.2
pyyaml                            6.0.2
pyzmq                             26.4.0
ray                               2.44.1
referencing                       0.36.2
regex                             2024.11.6
requests                          2.32.3
rich                              14.0.0
rich-toolkit                      0.14.3
rpds-py                           0.24.0
rsa                               4.9.1
safetensors                       0.5.3
scipy                             1.15.2
sentencepiece                     0.2.0
setuptools                        79.0.1
shellingham                       1.5.4
six                               1.17.0
smart-open                        7.1.0
sniffio                           1.3.1
starlette                         0.46.2
sympy                             1.13.1
tiktoken                          0.9.0
tokenizers                        0.21.1
torch                             2.6.0
torchaudio                        2.6.0
torchvision                       0.21.0
tqdm                              4.67.1
transformers                      4.51.3
triton                            3.2.0
typer                             0.15.2
typing-extensions                 4.13.2
typing-inspection                 0.4.0
tzdata                            2025.2
urllib3                           2.4.0
uvicorn                           0.34.2
uvloop                            0.21.0
virtualenv                        20.30.0
vllm                              0.8.2
watchfiles                        1.0.5
websockets                        15.0.1
wrapt                             1.17.2
xformers                          0.0.29.post2
xgrammar                          0.1.16
yarl                              1.20.0
zipp                              3.21.0
```

### Issue Severity

High: It blocks me from completing my task.

Hints from issue discussion:

Commit message: fix: deserialize arrays in vllm sampling_params

Signed-off-by: Arthur <atte.book@gmail.com>

Patch to review:
diff --git a/python/ray/llm/_internal/batch/stages/vllm_engine_stage.py b/python/ray/llm/_internal/batch/stages/vllm_engine_stage.py
index d073205f7f265..47bd681b16d60 100644
--- a/python/ray/llm/_internal/batch/stages/vllm_engine_stage.py
+++ b/python/ray/llm/_internal/batch/stages/vllm_engine_stage.py
@@ -301,7 +301,7 @@ async def _prepare_llm_request(self, row: Dict[str, Any]) -> vLLMEngineRequest:
             else:
                 guided_decoding = None
             params = vllm.SamplingParams(
-                **sampling_params,
+                **self._maybe_convert_ndarray_to_list(sampling_params),
                 guided_decoding=guided_decoding,
             )
         elif self.task_type == vLLMTaskType.EMBED:


Code review comments:
diff_hunk: @@ -301,7 +301,7 @@ async def _prepare_llm_request(self, row: Dict[str, Any]) -> vLLMEngineRequest:
             else:
                 guided_decoding = None
             params = vllm.SamplingParams(
-                **sampling_params,
+                **self._maybe_convert_ndarray_to_list(sampling_params),
path: python/ray/llm/_internal/batch/stages/vllm_engine_stage.py
text: @user1:
Can you do a similar thing for sglang engine stage as well? Thanks.