Repository: vllm-project/vllm
Issue category: Bug Report

Issue:
[Bug]: idefics3 doesn't stream
### Your current environment

<details>
<summary>The output of `python collect_env.py`</summary>

```text
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 14.2.1 20240910
Clang version: 18.1.8
CMake version: version 3.30.0
Libc version: glibc-2.40

Python version: 3.12.7 (main, Oct  1 2024, 11:15:50) [GCC 14.2.1 20240910] (64-bit runtime)
Python platform: Linux-6.10.10-arch1-1-kvm-local-x86_64-with-glibc2.40
Is CUDA available: True
CUDA runtime version: 12.6.85
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 Ti
Nvidia driver version: 565.57.01
cuDNN version: Probably one of the following:
/usr/lib/libcudnn.so.9.5.1
/usr/lib/libcudnn_adv.so.9.5.1
/usr/lib/libcudnn_cnn.so.9.5.1
/usr/lib/libcudnn_engines_precompiled.so.9.5.1
/usr/lib/libcudnn_engines_runtime_compiled.so.9.5.1
/usr/lib/libcudnn_graph.so.9.5.1
/usr/lib/libcudnn_heuristic.so.9.5.1
/usr/lib/libcudnn_ops.so.9.5.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               24
On-line CPU(s) list:                  0-23
Vendor ID:                            GenuineIntel
Model name:                           12th Gen Intel(R) Core(TM) i9-12900K
CPU family:                           6
Model:                                151
Thread(s) per core:                   2
Core(s) per socket:                   16
Socket(s):                            1
Stepping:                             2
CPU(s) scaling MHz:                   50%
CPU max MHz:                          5200.0000
CPU min MHz:                          800.0000
BogoMIPS:                             6374.40
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr flush_l1d arch_capabilities
Virtualization:                       VT-x
L1d cache:                            640 KiB (16 instances)
L1i cache:                            768 KiB (16 instances)
L2 cache:                             14 MiB (10 instances)
L3 cache:                             30 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-23

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu124torch2.4
[pip3] mypy==1.11.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] sentence-transformers==3.2.0
[pip3] torch==2.5.1
[pip3] torchac_cuda==0.2.5
[pip3] torchaudio==2.5.0
[pip3] torchvision==0.20.1
[pip3] transformers==4.46.2
[pip3] transformers-stream-generator==0.0.5
[pip3] triton==3.1.0
[conda] No relevant packages
ROCM Version: 6.2.41134-0
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post2.dev194+gd2ce1de02.d20241129
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
[4mGPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID[0m
GPU0	 X 	0-23	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

MAX_JOBS=16
NCCL_SOCKET_IFNAME=br0
CUDA_PATH=/opt/cuda
CUDA_SDK=/opt/cuda/targets/x86_64-linux
CUDA_SDK_ROOT_DIR=/opt/cuda/targets/x86_64-linux
CUDA_HOME=/opt/cuda
CUDA_HOME=/opt/cuda
LD_LIBRARY_PATH=/home/jeff/envs/python/virtualenvs/vllm312/lib/python3.12/site-packages/cv2/../../lib64:
CUDA_MODULE_LOADING=LAZY
```

</details>


### Model Input Dumps

_No response_

### 🐛 Describe the bug

Trying to stream with Idefics3-based models (both `Idefics3-8B-Llama3` and `SmolVLM`), I immediately get this traceback:

```
INFO 11-28 22:33:35 preprocess.py:215] Your model uses the legacy input pipeline instead of the new multi-modal processor. Please note that the legacy pipeline will be removed in a future release. For more details, see: https://github.com/vllm-project/vllm/issues/10114
INFO 11-28 22:33:35 engine.py:285] Aborted request chatcmpl-9aaf2452b75e4c54bc0b21685e6149d5.
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 188, in __call__
  |     await response(scope, wrapped_receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 222, in __call__
  |     async for chunk in self.body_iterator:
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in body_stream
  |     raise app_exc
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
  |     await self.app(scope, receive_or_disconnect, send_no_error)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 252, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    |     return await self.app(scope, receive, send)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    |     raise exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    |     with collapse_excgroups():
    |          ^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    |     self.gen.throw(value)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    |     raise exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 255, in wrap
    |     await func()
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 244, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
    |     async for res in result_generator:
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/utils.py", line 407, in iterate_with_cancellation
    |     item = await awaits[0]
    |            ^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/multiprocessing/client.py", line 633, in _process_request
    |     raise request_output
    | TypeError: argument of type 'NoneType' is not iterable
    +------------------------------------

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
  |     yield
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 186, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 259, in __call__
    |     await wrap(partial(self.listen_for_disconnect, receive))
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 255, in wrap
    |     await func()
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 232, in listen_for_disconnect
    |     message = await receive()
    |               ^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 118, in receive_or_disconnect
    |     async with anyio.create_task_group() as task_group:
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
    |     raise exc_val
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 126, in receive_or_disconnect
    |     message = await wrap(wrapped_receive)
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 121, in wrap
    |     result = await func()
    |              ^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 51, in wrapped_receive
    |     msg = await self.receive()
    |           ^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 555, in receive
    |     await self.message_event.wait()
    |   File "/usr/lib/python3.12/asyncio/locks.py", line 212, in wait
    |     await fut
    | asyncio.exceptions.CancelledError: Cancelled by cancel scope 70de11b4acc0
    |
    | During handling of the above exception, another exception occurred:
    |
    | Exception Group Traceback (most recent call last):
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 188, in __call__
    |     await response(scope, wrapped_receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 222, in __call__
    |     async for chunk in self.body_iterator:
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in body_stream
    |     raise app_exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    |     await self.app(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    |     await route.handle(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    |     await self.app(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
    |     await response(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 252, in __call__
    |     async with anyio.create_task_group() as task_group:
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
    |     raise BaseExceptionGroup(
    | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
      |     result = await app(  # type: ignore[func-returns-value]
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
      |     return await self.app(scope, receive, send)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
      |     raise exc
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
      |     await self.app(scope, receive, _send)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
      |     with collapse_excgroups():
      |          ^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
      |     self.gen.throw(value)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
      |     raise exc
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 255, in wrap
      |     await func()
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 244, in stream_response
      |     async for chunk in self.body_iterator:
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
      |     async for res in result_generator:
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/utils.py", line 407, in iterate_with_cancellation
      |     item = await awaits[0]
      |            ^^^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/multiprocessing/client.py", line 633, in _process_request
      |     raise request_output
      | TypeError: argument of type 'NoneType' is not iterable
      +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
         ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 255, in wrap
    await func()
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 244, in stream_response
    async for chunk in self.body_iterator:
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
    async for res in result_generator:
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/utils.py", line 407, in iterate_with_cancellation
    item = await awaits[0]
           ^^^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/multiprocessing/client.py", line 633, in _process_request
    raise request_output
TypeError: argument of type 'NoneType' is not iterable
```

The models appear to work with simple prompts in `vllm chat`.

### Before submitting a new issue...

- [X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.


Hints:
Please re-run this with `--disable-frontend-multiprocessing` to get a clearer stack trace.
Sure, here you go:

```
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 188, in __call__
  |     await response(scope, wrapped_receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 222, in __call__
  |     async for chunk in self.body_iterator:
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in body_stream
  |     raise app_exc
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
  |     await self.app(scope, receive_or_disconnect, send_no_error)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 252, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    |     return await self.app(scope, receive, send)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    |     raise exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    |     with collapse_excgroups():
    |          ^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    |     self.gen.throw(value)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    |     raise exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 255, in wrap
    |     await func()
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 244, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
    |     async for res in result_generator:
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/utils.py", line 407, in iterate_with_cancellation
    |     item = await awaits[0]
    |            ^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 1054, in generate
    |     async for output in await self.add_request(
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 113, in generator
    |     raise result
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
    |     return_value = task.result()
    |                    ^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 875, in run_engine_loop
    |     result = task.result()
    |              ^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 786, in engine_step
    |     await self.engine.add_request_async(**new_request)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 495, in add_request_async
    |     processed_inputs = self.input_processor(preprocessed_inputs)
    |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/inputs/registry.py", line 352, in process_input
    |     if is_encoder_decoder_inputs(processed_inputs):
    |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/inputs/parse.py", line 112, in is_encoder_decoder_inputs
    |     return "encoder" in inputs and "decoder" in inputs
    |            ^^^^^^^^^^^^^^^^^^^
    | TypeError: argument of type 'NoneType' is not iterable
    +------------------------------------

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
  |     yield
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 186, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 259, in __call__
    |     await wrap(partial(self.listen_for_disconnect, receive))
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 255, in wrap
    |     await func()
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 232, in listen_for_disconnect
    |     message = await receive()
    |               ^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 118, in receive_or_disconnect
    |     async with anyio.create_task_group() as task_group:
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 767, in __aexit__
    |     raise exc_val
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 126, in receive_or_disconnect
    |     message = await wrap(wrapped_receive)
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 121, in wrap
    |     result = await func()
    |              ^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 51, in wrapped_receive
    |     msg = await self.receive()
    |           ^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 555, in receive
    |     await self.message_event.wait()
    |   File "/usr/lib/python3.12/asyncio/locks.py", line 212, in wait
    |     await fut
    | asyncio.exceptions.CancelledError: Cancelled by cancel scope 79d48ebb8f50
    |
    | During handling of the above exception, another exception occurred:
    |
    | Exception Group Traceback (most recent call last):
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 188, in __call__
    |     await response(scope, wrapped_receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 222, in __call__
    |     async for chunk in self.body_iterator:
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in body_stream
    |     raise app_exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    |     await self.app(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    |     await route.handle(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    |     await self.app(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
    |     await response(scope, receive, send)
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 252, in __call__
    |     async with anyio.create_task_group() as task_group:
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
    |     raise BaseExceptionGroup(
    | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
      |     result = await app(  # type: ignore[func-returns-value]
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
      |     return await self.app(scope, receive, send)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
      |     raise exc
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
      |     await self.app(scope, receive, _send)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
      |     with collapse_excgroups():
      |          ^^^^^^^^^^^^^^^^^^^^
      |   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
      |     self.gen.throw(value)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
      |     raise exc
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 255, in wrap
      |     await func()
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 244, in stream_response
      |     async for chunk in self.body_iterator:
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
      |     async for res in result_generator:
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/utils.py", line 407, in iterate_with_cancellation
      |     item = await awaits[0]
      |            ^^^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 1054, in generate
      |     async for output in await self.add_request(
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 113, in generator
      |     raise result
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
      |     return_value = task.result()
      |                    ^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 875, in run_engine_loop
      |     result = task.result()
      |              ^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 786, in engine_step
      |     await self.engine.add_request_async(**new_request)
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 495, in add_request_async
      |     processed_inputs = self.input_processor(preprocessed_inputs)
      |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/inputs/registry.py", line 352, in process_input
      |     if is_encoder_decoder_inputs(processed_inputs):
      |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/inputs/parse.py", line 112, in is_encoder_decoder_inputs
      |     return "encoder" in inputs and "decoder" in inputs
      |            ^^^^^^^^^^^^^^^^^^^
      | TypeError: argument of type 'NoneType' is not iterable
      +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
         ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 255, in wrap
    await func()
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/starlette/responses.py", line 244, in stream_response
    async for chunk in self.body_iterator:
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 319, in chat_completion_stream_generator
    async for res in result_generator:
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/utils.py", line 407, in iterate_with_cancellation
    item = await awaits[0]
           ^^^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 1054, in generate
    async for output in await self.add_request(
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 113, in generator
    raise result
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
    return_value = task.result()
                   ^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 875, in run_engine_loop
    result = task.result()
             ^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 786, in engine_step
    await self.engine.add_request_async(**new_request)
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/engine/async_llm_engine.py", line 495, in add_request_async
    processed_inputs = self.input_processor(preprocessed_inputs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/inputs/registry.py", line 352, in process_input
    if is_encoder_decoder_inputs(processed_inputs):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeff/.virtualenvs/vllm312/lib/python3.12/site-packages/vllm/inputs/parse.py", line 112, in is_encoder_decoder_inputs
    return "encoder" in inputs and "decoder" in inputs
           ^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
```
cc @Isotr0py @jeejeelee 
I probably found the cause and am trying to solve it

Patch:
diff --git a/vllm/model_executor/models/idefics3.py b/vllm/model_executor/models/idefics3.py
index 014e27bc869d4..e5d2edbd81eb1 100644
--- a/vllm/model_executor/models/idefics3.py
+++ b/vllm/model_executor/models/idefics3.py
@@ -267,54 +267,56 @@ def input_processor_for_idefics3(ctx: InputContext,
     n_images_in_text = []
 
     text = inputs.get("prompt")
-    if text is not None:
-        if isinstance(text, str):
-            text = [text]
-        elif not isinstance(text, list) and not isinstance(text[0], str):
-            raise ValueError("Invalid input text. Please provide a string, "
-                             "or a list of strings")
-
-        fake_image_token = processor.fake_image_token.content
-        image_token = processor.image_token.content
-        global_img_token = processor.global_image_tag
-
-        prompt_strings = []
-        for sample, sample_rows, sample_cols in zip(text, image_rows,
-                                                    image_cols):
-            n_images_in_text.append(sample.count(image_token))
-
-            # Replace the image token with fake tokens around the expanded
-            # image token sequence of length `image_seq_len`
-            image_prompt_strings = []
-            for n_rows, n_cols in zip(sample_rows, sample_cols):
-                image_prompt_string = _get_image_prompt_string(
-                    n_rows,
-                    n_cols,
-                    processor.image_seq_len,
-                    image_token=image_token,
-                    fake_token_around_image=fake_image_token,
-                    global_img_token=global_img_token,
-                )
-                image_prompt_strings.append(image_prompt_string)
-
-            split_sample = sample.split(image_token)
-            if len(split_sample) == 0:
-                raise ValueError(
-                    "The image token should be present in the text.")
+    if text is None:
+        prompt_token_ids = inputs.get("prompt_token_ids", [])
+        assert prompt_token_ids
+        text = tokenizer.decode(prompt_token_ids)
+
+    if isinstance(text, str):
+        text = [text]
+    elif not isinstance(text, list) and not isinstance(text[0], str):
+        raise ValueError("Invalid input text. Please provide a string, "
+                         "or a list of strings")
+
+    fake_image_token = processor.fake_image_token.content
+    image_token = processor.image_token.content
+    global_img_token = processor.global_image_tag
+
+    prompt_strings = []
+    for sample, sample_rows, sample_cols in zip(text, image_rows, image_cols):
+        n_images_in_text.append(sample.count(image_token))
+
+        # Replace the image token with fake tokens around the expanded
+        # image token sequence of length `image_seq_len`
+        image_prompt_strings = []
+        for n_rows, n_cols in zip(sample_rows, sample_cols):
+            image_prompt_string = _get_image_prompt_string(
+                n_rows,
+                n_cols,
+                processor.image_seq_len,
+                image_token=image_token,
+                fake_token_around_image=fake_image_token,
+                global_img_token=global_img_token,
+            )
+            image_prompt_strings.append(image_prompt_string)
 
-            # Place in the image prompt strings where the image tokens are
-            sample = split_sample[0]
-            for i, image_prompt_string in enumerate(image_prompt_strings):
-                sample += image_prompt_string + split_sample[i + 1]
-            prompt_strings.append(sample)
+        split_sample = sample.split(image_token)
+        if len(split_sample) == 0:
+            raise ValueError("The image token should be present in the text.")
 
-        prompt_token_ids = tokenizer(text=prompt_strings[0]).input_ids
+        # Place in the image prompt strings where the image tokens are
+        sample = split_sample[0]
+        for i, image_prompt_string in enumerate(image_prompt_strings):
+            sample += image_prompt_string + split_sample[i + 1]
+        prompt_strings.append(sample)
 
-        return token_inputs(
-            prompt_token_ids=prompt_token_ids,
-            prompt=prompt_strings[0],
-            multi_modal_data=multi_modal_data,
-        )
+    prompt_token_ids = tokenizer(text=prompt_strings[0]).input_ids
+
+    return token_inputs(
+        prompt_token_ids=prompt_token_ids,
+        prompt=prompt_strings[0],
+        multi_modal_data=multi_modal_data,
+    )
 
 
 def _get_max_num_image_patch(image_processor: Idefics3ImageProcessor) -> int:


Test patch: