Repository: NVIDIA/TensorRT-LLM
Pull request #3757: feat: enhance trtllm serve multimodal
Author: login: yechank-nvidia
Created: 2025-04-22T05:58:05Z | Merged: 2025-05-15T23:16:32Z

This MR enhances the trtllm-serve multimdoal support

1. made the load_image and load_video asynchronous
2. add image_encoded input support to be compatible with genai-perf and add genai-perf test
3. support text-only on multimodal mdoels(currently, Qwen2-VL & Qwen2.5-VL)

After this PR, need to optimize E2E flow. Multimodal models are heavily dependent on input processor which contains Vision Model. So, when the input processor is slow the LLM is waiting its input to be ready. 


Diff:
diff --git a/docs/source/commands/trtllm-serve.rst b/docs/source/commands/trtllm-serve.rst
index 60620bb9ff1c..9c322ebbc656 100644
--- a/docs/source/commands/trtllm-serve.rst
+++ b/docs/source/commands/trtllm-serve.rst
@@ -78,6 +78,7 @@ First, create a configuration file:
    cat >./extra-llm-api-config.yml<<EOF
    kv_cache_config:
        enable_block_reuse: false
+       free_gpu_memory_fraction: 0.6
    EOF
 
 Then, start the server with the configuration file:
diff --git a/examples/serve/curl_chat_client_for_multimodal.sh b/examples/serve/curl_chat_client_for_multimodal.sh
index 25c7b0f1b070..3719a56569c7 100644
--- a/examples/serve/curl_chat_client_for_multimodal.sh
+++ b/examples/serve/curl_chat_client_for_multimodal.sh
@@ -1,10 +1,10 @@
 #! /usr/bin/env bash
 
-# Single image inference
+# SINGLE IMAGE INFERENCE
 curl http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json"  \
     -d '{
-        "model": "Qwen2-VL-7B-Instruct",
+        "model": "Qwen2.5-VL-3B-Instruct",
         "messages":[{
             "role": "system",
             "content": "You are a helpful assistant."
@@ -26,3 +26,63 @@ curl http://localhost:8000/v1/chat/completions \
         "max_tokens": 64,
         "temperature": 0
     }'
+
+# MULTI IMAGE INFERENCE
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "Qwen2.5-VL-3B-Instruct",
+        "messages":[{
+            "role": "system",
+            "content": "You are a helpful assistant."
+        }, {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text":"Tell me the difference between two images"
+                },
+                {
+                    "type":"image_url",
+                    "image_url": {
+                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
+                    }
+                },
+                {
+                    "type":"image_url",
+                    "image_url": {
+                        "url": "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
+                    }
+                }
+            ]
+        }],
+        "max_tokens": 64,
+        "temperature": 0
+    }'
+
+# SINGLE VIDEO INFERENCE
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "Qwen2.5-VL-3B-Instruct",
+        "messages":[{
+            "role": "system",
+            "content": "You are a helpful assistant."
+        }, {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text":"Tell me what you see in the video briefly."
+                },
+                {
+                    "type":"video_url",
+                    "video_url": {
+                        "url": "https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4"
+                    }
+                }
+            ]
+        }],
+        "max_tokens": 64,
+        "temperature": 0
+    }'
diff --git a/examples/serve/genai_perf_client.sh b/examples/serve/genai_perf_client.sh
index dd7afcd9fd2e..91a6308995cd 100755
--- a/examples/serve/genai_perf_client.sh
+++ b/examples/serve/genai_perf_client.sh
@@ -3,7 +3,6 @@
 genai-perf profile \
     -m TinyLlama-1.1B-Chat-v1.0 \
     --tokenizer TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
-    --service-kind openai \
     --endpoint-type chat \
     --random-seed 123 \
     --synthetic-input-tokens-mean 128 \
diff --git a/examples/serve/genai_perf_client_for_multimodal.sh b/examples/serve/genai_perf_client_for_multimodal.sh
new file mode 100644
index 000000000000..a04f424549f3
--- /dev/null
+++ b/examples/serve/genai_perf_client_for_multimodal.sh
@@ -0,0 +1,19 @@
+#! /usr/bin/env bash
+
+genai-perf profile \
+    -m Qwen2.5-VL-3B-Instruct \
+    --tokenizer Qwen/Qwen2.5-VL-3B-Instruct \
+    --endpoint-type multimodal \
+    --random-seed 123 \
+    --image-width-mean 64 \
+    --image-height-mean 64 \
+    --image-format png \
+    --synthetic-input-tokens-mean 128 \
+    --synthetic-input-tokens-stddev 0 \
+    --output-tokens-mean 128 \
+    --output-tokens-stddev 0 \
+    --request-count 5 \
+    --request-rate 1 \
+    --profile-export-file my_profile_export.json \
+    --url localhost:8000 \
+    --streaming
diff --git a/examples/serve/openai_chat_client_for_multimodal.py b/examples/serve/openai_chat_client_for_multimodal.py
index a790b1d9ffda..2372cb02162e 100644
--- a/examples/serve/openai_chat_client_for_multimodal.py
+++ b/examples/serve/openai_chat_client_for_multimodal.py
@@ -2,14 +2,16 @@
 
 from openai import OpenAI
 
+from tensorrt_llm.inputs import encode_base64_content_from_url
+
 client = OpenAI(
     base_url="http://localhost:8000/v1",
     api_key="tensorrt_llm",
 )
 
-# Single image inference
+# SINGLE IMAGE INFERENCE
 response = client.chat.completions.create(
-    model="Qwen2-VL-7B-Instruct",
+    model="Qwen2.5-VL-3B-Instruct",
     messages=[{
         "role": "system",
         "content": "you are a helpful assistant"
@@ -31,6 +33,82 @@
 )
 print(response)
 
-# TODO
-# multi-image inference
-# video inference
+# MULTI IMAGE INFERENCE
+response = client.chat.completions.create(
+    model="Qwen2.5-VL-3B-Instruct",
+    messages=[{
+        "role": "system",
+        "content": "you are a helpful assistant"
+    }, {
+        "role":
+        "user",
+        "content": [{
+            "type": "text",
+            "text": "Tell me the difference between two images"
+        }, {
+            "type": "image_url",
+            "image_url": {
+                "url":
+                "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
+            }
+        }, {
+            "type": "image_url",
+            "image_url": {
+                "url":
+                "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
+            }
+        }]
+    }],
+    max_tokens=64,
+)
+print(response)
+
+# SINGLE VIDEO INFERENCE
+response = client.chat.completions.create(
+    model="Qwen2.5-VL-3B-Instruct",
+    messages=[{
+        "role": "system",
+        "content": "you are a helpful assistant"
+    }, {
+        "role":
+        "user",
+        "content": [{
+            "type": "text",
+            "text": "Tell me what you see in the video briefly."
+        }, {
+            "type": "video_url",
+            "video_url": {
+                "url":
+                "https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4"
+            }
+        }]
+    }],
+    max_tokens=64,
+)
+print(response)
+
+# IMAGE EMBED INFERENCE
+image64 = encode_base64_content_from_url(
+    "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
+)
+response = client.chat.completions.create(
+    model="Qwen2.5-VL-3B-Instruct",
+    messages=[{
+        "role": "system",
+        "content": "you are a helpful assistant"
+    }, {
+        "role":
+        "user",
+        "content": [{
+            "type": "text",
+            "text": "Describe the natural environment in the image."
+        }, {
+            "type": "image_url",
+            "image_url": {
+                "url": "data:image/png;base64," + image64
+            }
+        }]
+    }],
+    max_tokens=64,
+)
+print(response)
diff --git a/tensorrt_llm/_torch/models/modeling_qwen2vl.py b/tensorrt_llm/_torch/models/modeling_qwen2vl.py
index 368a5bc80b6b..d2335530f9ab 100644
--- a/tensorrt_llm/_torch/models/modeling_qwen2vl.py
+++ b/tensorrt_llm/_torch/models/modeling_qwen2vl.py
@@ -318,12 +318,14 @@ def __call__(
         mm_processor_kwargs['do_rescale'] = False
         processed_inputs = self._preprocess(text_prompt, mm_data,
                                             mm_processor_kwargs).to(self.device)
-
-        mm_features = self._process(
-            processed_inputs.get('pixel_values', None),
-            processed_inputs.get('pixel_values_videos', None),
-            processed_inputs.get('image_grid_thw', None),
-            processed_inputs.get('video_grid_thw', None))
+        if mm_data:
+            mm_features = self._process(
+                processed_inputs.get('pixel_values', None),
+                processed_inputs.get('pixel_values_videos', None),
+                processed_inputs.get('image_grid_thw', None),
+                processed_inputs.get('video_grid_thw', None))
+        else:
+            mm_features = None
 
         input_ids = processed_inputs['input_ids']
 
diff --git a/tensorrt_llm/inputs/__init__.py b/tensorrt_llm/inputs/__init__.py
index a9afd9184a8a..0e1d20f82ea7 100644
--- a/tensorrt_llm/inputs/__init__.py
+++ b/tensorrt_llm/inputs/__init__.py
@@ -1,15 +1,17 @@
 from .data import PromptInputs, TextPrompt, TokensPrompt, prompt_inputs
 from .registry import (ExtraProcessedInputs, InputProcessor,
                        create_input_processor, register_input_processor)
-from .utils import (INPUT_FORMATTER_MAP, default_image_loader,
-                    default_video_loader, format_generic_input,
+from .utils import (INPUT_FORMATTER_MAP, async_load_image, async_load_video,
+                    default_image_loader, default_video_loader,
+                    encode_base64_content_from_url, format_generic_input,
                     format_qwen2_vl_input, format_vila_input, load_image,
                     load_video)
 
 __all__ = [
     "PromptInputs", "prompt_inputs", "TextPrompt", "TokensPrompt",
     "InputProcessor", "create_input_processor", "register_input_processor",
-    "ExtraProcessedInputs", "load_image", "load_video", "INPUT_FORMATTER_MAP",
-    "default_image_loader", "default_video_loader", "format_vila_input",
-    "format_generic_input", "format_qwen2_vl_input"
+    "ExtraProcessedInputs", "load_image", "load_video", "async_load_image",
+    "async_load_video", "INPUT_FORMATTER_MAP", "default_image_loader",
+    "default_video_loader", "format_vila_input", "format_generic_input",
+    "format_qwen2_vl_input", "encode_base64_content_from_url"
 ]
diff --git a/tensorrt_llm/inputs/utils.py b/tensorrt_llm/inputs/utils.py
index 0da8ec617513..4576a71af412 100644
--- a/tensorrt_llm/inputs/utils.py
+++ b/tensorrt_llm/inputs/utils.py
@@ -1,5 +1,12 @@
+import base64
+import tempfile
+from io import BytesIO
+from pathlib import Path
 from typing import List, Union
+from urllib.parse import urlparse
 
+import aiohttp
+import cv2
 import numpy as np
 import requests
 import torch
@@ -8,16 +15,67 @@
 from transformers import AutoProcessor
 
 
+def _load_and_convert_image(image):
+    image = Image.open(image)
+    image.load()
+    return image.convert("RGB")
+
+
 def load_image(image: str,
                format: str = "pt",
                device: str = "cuda") -> Union[Image.Image, torch.Tensor]:
     assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
 
-    if image.startswith("http://") or image.startswith("https://"):
-        image = Image.open(requests.get(image, stream=True, timeout=10).raw)
+    parsed_url = urlparse(image)
+
+    if parsed_url.scheme in ["http", "https"]:
+        image = requests.get(image, stream=True, timeout=10).raw
+        image = _load_and_convert_image(image)
+    elif parsed_url.scheme == "data":
+        data_spec, data = parsed_url.path.split(",", 1)
+        media_type, data_type = data_spec.split(";", 1)
+
+        if data_type != "base64":
+            msg = "Only base64 data URLs are supported for now."
+            raise NotImplementedError(msg)
+
+        content = base64.b64decode(data)
+        image = _load_and_convert_image(BytesIO(content))
     else:
-        image = Image.open(image)
-    image = image.convert("RGB")
+        image = _load_and_convert_image(image)
+
+    if format == "pt":
+        return ToTensor()(image).to(device=device)
+    else:
+        return image
+
+
+async def async_load_image(
+        image: str,
+        format: str = "pt",
+        device: str = "cuda") -> Union[Image.Image, torch.Tensor]:
+    assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
+
+    parsed_url = urlparse(image)
+
+    if parsed_url.scheme in ["http", "https"]:
+        async with aiohttp.ClientSession() as session:
+            async with session.get(image) as response:
+                content = await response.read()
+                image = _load_and_convert_image(BytesIO(content))
+    elif parsed_url.scheme == "data":
+        data_spec, data = parsed_url.path.split(",", 1)
+        media_type, data_type = data_spec.split(";", 1)
+
+        if data_type != "base64":
+            msg = "Only base64 data URLs are supported for now."
+            raise NotImplementedError(msg)
+
+        content = base64.b64decode(data)
+        image = _load_and_convert_image(BytesIO(content))
+    else:
+        image = _load_and_convert_image(Path(parsed_url.path))
+
     if format == "pt":
         return ToTensor()(image).to(device=device)
     else:
@@ -29,7 +87,6 @@ def load_video(
         num_frames: int = 10,
         format: str = "pt",
         device: str = "cuda") -> Union[List[Image.Image], List[torch.Tensor]]:
-    import cv2
 
     assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
 
@@ -71,6 +128,40 @@ def load_video(
     ]
 
 
+async def async_load_video(
+        video: str,
+        num_frames: int = 10,
+        format: str = "pt",
+        device: str = "cuda") -> Union[List[Image.Image], List[torch.Tensor]]:
+    assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
+
+    parsed_url = urlparse(video)
+
+    if parsed_url.scheme in ["http", "https"]:
+        async with aiohttp.ClientSession() as session:
+            async with session.get(video) as response:
+                with tempfile.NamedTemporaryFile(delete=False,
+                                                 suffix='.mp4') as tmp:
+                    tmp.write(await response.content.read())
+                    video_path = tmp.name
+    # TODO: add case for video encoded in base64
+    else:
+        video_path = video
+
+    return load_video(video_path, num_frames, format, device)
+
+
+# Copied from https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_client_for_multimodal.py#L38
+def encode_base64_content_from_url(content_url: str) -> str:
+    """Encode a content retrieved from a remote url to base64 format."""
+
+    with requests.get(content_url, timeout=10) as response:
+        response.raise_for_status()
+        result = base64.b64encode(response.content).decode('utf-8')
+
+    return result
+
+
 """
 VLM input preparation.
 """
@@ -169,7 +260,6 @@ def apply_template(prompt, multimodal_data):
                    }]
 
         conversation = [{"role": "user", "content": content}]
-        # print(conversation)
         return processor.apply_chat_template(
             conversation,
             tokenize=False,
diff --git a/tensorrt_llm/serve/chat_utils.py b/tensorrt_llm/serve/chat_utils.py
index 1d3423e52515..5220a797fd75 100644
--- a/tensorrt_llm/serve/chat_utils.py
+++ b/tensorrt_llm/serve/chat_utils.py
@@ -1,6 +1,8 @@
+import asyncio
+from collections import defaultdict
 from functools import partial
-from typing import (Any, Callable, Dict, Iterable, List, Literal, Optional,
-                    Tuple, TypeAlias, TypedDict, Union, cast)
+from typing import (Any, Callable, Coroutine, Dict, Iterable, List, Literal,
+                    Optional, Tuple, TypeAlias, TypedDict, Union, cast)
 
 from openai.types.chat import ChatCompletionContentPartImageParam
 from openai.types.chat import \
@@ -10,170 +12,199 @@
 from transformers import AutoConfig, ProcessorMixin
 from typing_extensions import Required
 
-from tensorrt_llm.inputs import load_image, load_video
+from tensorrt_llm.inputs import async_load_image, async_load_video
 from tensorrt_llm.llmapi.tokenizer import TokenizerBase
 from tensorrt_llm.logger import logger
 
 
 class VideoURL(TypedDict):
+    """Type definition for video URL structure."""
     url: Required[str]
 
 
 class ChatCompletionContentPartVideoParam(TypedDict, total=False):
+    """Type definition for video content part parameters."""
     video_url: Required[VideoURL]
     type: Required[Literal["video_url"]]
 
 
 class ConversationMessage(TypedDict):
+    """Type definition for conversation message structure."""
     role: str
     content: str
 
 
-_TextParser = partial(cast, ChatCompletionContentPartTextParam)
-_ImageParser = partial(cast, ChatCompletionContentPartImageParam)
-_VideoParser = partial(cast, ChatCompletionContentPartVideoParam)
-
+# Type Aliases and Constants
 ChatCompletionContentPartParam: TypeAlias = Union[
     OpenAIChatCompletionContentPartParam, ChatCompletionContentPartVideoParam,
     str]
 
-MM_PARSER_MAP: dict[
-    str,
-    Callable[[ChatCompletionContentPartParam], Union[str, dict[str, str]]],
-] = {
-    "text":
-    lambda part: _TextParser(part).get("text", None),
-    "image_url":
-    lambda part: _ImageParser(part).get("image_url", {}).get("url", None),
-    "video_url":
-    lambda part: _VideoParser(part).get("video_url", {}).get("url", None),
-}
-
 VALID_MESSAGE_CONTENT_MM_PART_TYPES = ["text", "image_url", "video_url"]
 
+# Parser Functions
+_TextParser = partial(cast, ChatCompletionContentPartTextParam)
+_ImageParser = partial(cast, ChatCompletionContentPartImageParam)
+_VideoParser = partial(cast, ChatCompletionContentPartVideoParam)
 
-def retrieve_multimodal_placeholder(
-    modality: str,
-    model_config: AutoConfig,
-    current_count: int,
-) -> Optional[str]:
-    model_type = model_config.model_type
-
-    if modality == "image":
-        if model_type in ("mllama", "llama4"):
-            return "
"
-        if model_type in ("qwen2_vl", "qwen2_5_vl"):
-            return "


"
-        raise TypeError(f"Unknown {modality} model type: {model_type}")
-
-    elif modality == "video":
-        if model_type in ("qwen2_vl", "qwen2_5_vl"):
-            return "


"
-        raise TypeError(f"Unknown {modality} model type: {model_type}")
-
-    raise TypeError(f"Unknown modality: {modality}")
-
-
-def add_multimodal_placeholders(
-    text_prompt: str,
-    mm_content_dict: dict[str, list[Any]],
-    model_config: AutoConfig,
-) -> str:
+MM_PARSER_MAP: dict[str, Callable[[ChatCompletionContentPartParam], Union[
+    str, dict[str, str]]]] = {
+        "text":
+        lambda part: _TextParser(part).get("text", None),
+        "image_url":
+        lambda part: _ImageParser(part).get("image_url", {}).get("url", None),
+        "video_url":
+        lambda part: _VideoParser(part).get("video_url", {}).get("url", None),
+    }
+
+
+class AsyncMultimodalDataTracker:
+    """Tracks and manages multimodal data for async processing."""
+
+    def __init__(self, model_config: AutoConfig):
+        self.model_config = model_config
+        self.mm_data = defaultdict[str](list)
+        self.mm_placeholder_counts = defaultdict[str](int)
+
+    async def retrieve_all_mm_data(self) -> Optional[Dict[str, List[Any]]]:
+        """Retrieve all collected multimodal data."""
+        if not self.mm_data:
+            return None
+
+        return {
+            modality: await asyncio.gather(*items)
+            for modality, items in self.mm_data.items()
+        }
+
+    def retrieve_multimodal_placeholder(self, modality: str,
+                                        current_count: int) -> Optional[str]:
+        """Get the appropriate placeholder for a given modality and model type."""
+        model_type = self.model_config.model_type
+
+        if modality == "image":
+            if model_type in ("qwen2_vl", "qwen2_5_vl"):
+                return "


"
+            elif model_type in ("mllama", "llama4"):
+                return "
"
+            raise TypeError(f"Unknown {modality} model type: {model_type}")
+        elif modality == "video":
+            if model_type in ("qwen2_vl", "qwen2_5_vl"):
+                return "


"
+            raise TypeError(f"Unknown {modality} model type: {model_type}")
+        raise TypeError(f"Unknown modality: {modality}")
+
+    def add_mm_data(self, media_type: str, data: Coroutine):
+        current_count = len(self.mm_data[media_type]) + 1
+        placeholder = self.retrieve_multimodal_placeholder(
+            media_type, current_count)
+        self.mm_data[media_type].append(data)
+        if placeholder:
+            self.mm_placeholder_counts[placeholder] += 1
+
+    def mm_data_counts(self) -> Dict[str, int]:
+        """Get the count of multimodal placeholders."""
+        return dict(self.mm_placeholder_counts)
+
+
+def add_multimodal_placeholders(text_prompt: str,
+                                mm_placeholder_counts: dict[str, int]) -> str:
+    """Add multimodal placeholders to the text prompt."""
     placeholders = []
-    counts = {}
-    for media_type, _ in mm_content_dict.items():
-        if media_type not in counts:
-            counts[media_type] = 0
-        counts[media_type] += 1
-        placeholder = retrieve_multimodal_placeholder(media_type, model_config,
-                                                      counts[media_type])
-        if placeholder is not None:
-            placeholders.append(placeholder)
+    for placeholder in mm_placeholder_counts:
+        placeholders.extend([placeholder] * mm_placeholder_counts[placeholder])
     return "\n".join(placeholders + [text_prompt])
 
 
 def _parse_chat_message_content_mm_part(
     part: ChatCompletionContentPartParam
 ) -> tuple[str, Union[str, dict[str, str]]]:
-    assert isinstance(
-        part, dict)  # This is needed to avoid mypy errors: part.get() from str
+    """Parse a single multimodal part of a chat message."""
+    assert isinstance(part, dict)
     part_type = part.get("type", None)
 
     if isinstance(part_type, str) and part_type in MM_PARSER_MAP:
-        content = MM_PARSER_MAP[part_type](part)
-        return part_type, content
+        return part_type, MM_PARSER_MAP[part_type](part)
 
     if not isinstance(part_type, str):
         raise ValueError("Invalid 'type' field in multimodal part.")
     return part_type, "unknown part_type content"
 
 
-def parse_chat_message_content_part(part: ChatCompletionMessageParam, ):
-    if isinstance(part, str):  # Handle plain text parts
+def parse_chat_message_content_part(
+    part: ChatCompletionMessageParam,
+    mm_data_tracker: AsyncMultimodalDataTracker,
+) -> Optional[str]:
+    """Parse a single part of a chat message."""
+    if isinstance(part, str):
         return part
 
-    # Handle structured dictionary parts
     part_type, content = _parse_chat_message_content_mm_part(part)
 
     # if part_type is text/image_url/video_url but content is None, log a warning and skip
     if part_type in VALID_MESSAGE_CONTENT_MM_PART_TYPES and content is None:
         logger.warning(
-            "Skipping multimodal part '%s' (type: '%s') "
-            "with empty / unparsable content.", part, part_type)
+            "Skipping multimodal part '%s' (type: '%s') with empty / unparsable content.",
+            part, part_type)
         return None
 
-    mm_content = None
     if part_type == "text":
-        str_content = cast(str, content)
-        return str_content, mm_content
-
-    # TODO: make them async on multimodal data as loading video/image is time consuming
+        return cast(str, content)
 
-    # Handle all non-text multimodal types
     if part_type == "image_url":
         str_content = cast(str, content)
-        mm_content = {"image": load_image(str_content)}
-    elif part_type == "video_url":
+
+        async def load_image_async():
+            try:
+                return await async_load_image(str_content)
+            except Exception as e:
+                logger.error(f"Failed to load image: {str(e)}")
+                return None
+
+        mm_data_tracker.add_mm_data("image", load_image_async())
+        return None
+
+    if part_type == "video_url":
         str_content = cast(str, content)
-        mm_content = {"video": load_video(str_content, num_frames=8)}
-    else:
-        raise NotImplementedError(f"Unknown part type: {part_type}")
 
-    return None, mm_content
+        async def load_video_async():
+            try:
+                return await async_load_video(str_content, num_frames=8)
+            except Exception as e:
+                logger.error(f"Failed to load video: {str(e)}")
+                return None
+
+        mm_data_tracker.add_mm_data("video", load_video_async())
+        return None
+
+    raise NotImplementedError(f"Unknown part type: {part_type}")
 
 
 def parse_chat_message_content_parts(
     role: str,
     parts: Iterable[ChatCompletionMessageParam],
-    model_config: AutoConfig,
+    mm_data_tracker: AsyncMultimodalDataTracker,
 ) -> List[ConversationMessage]:
-    content = []
-    mm_content_dict = {}
-
+    """Parse multiple parts of a chat message."""
+    content_parts = []
     for part in parts:
-        parse_res, mm_content = parse_chat_message_content_part(part, )
+        parse_res = parse_chat_message_content_part(part, mm_data_tracker)
         if parse_res:
-            content.append(parse_res)
+            content_parts.append(parse_res)
+
+    text_prompt = "\n".join(content_parts)
+    mm_placeholder_counts = mm_data_tracker.mm_data_counts()
 
-        # Collect multimodal content
-        if mm_content:
-            for media_type, media_value in mm_content.items():
-                if media_type not in mm_content_dict:
-                    mm_content_dict[media_type] = []
-                mm_content_dict[media_type].append(media_value)
+    if mm_placeholder_counts:
+        text_prompt = add_multimodal_placeholders(text_prompt,
+                                                  mm_placeholder_counts)
 
-    text_prompt = "\n".join(content)
-    if mm_content_dict:
-        text_prompt = add_multimodal_placeholders(text_prompt, mm_content_dict,
-                                                  model_config)
-    return [ConversationMessage(role=role,
-                                content=text_prompt)], mm_content_dict
+    return [ConversationMessage(role=role, content=text_prompt)]
 
 
 def parse_chat_message_content(
     message: ChatCompletionMessageParam,
-    model_config: AutoConfig,
-) -> Tuple[List[ConversationMessage], Optional[Dict[str, List[Any]]]]:
+    mm_data_tracker: AsyncMultimodalDataTracker,
+) -> List[ConversationMessage]:
+    """Parse the content of a chat message."""
     role = message["role"]
     content = message.get("content")
 
@@ -184,12 +215,28 @@ def parse_chat_message_content(
             ChatCompletionContentPartTextParam(type="text", text=content)
         ]
 
-    result, mm_data = parse_chat_message_content_parts(
+    result = parse_chat_message_content_parts(
         role,
         content,
-        model_config,
+        mm_data_tracker,
     )
-    return result, mm_data
+    return result
+
+
+def parse_chat_messages_coroutines(
+    messages: List[ChatCompletionMessageParam],
+    model_config: AutoConfig,
+) -> Tuple[List[ConversationMessage], Optional[Coroutine[
+        Any, Any, Optional[Dict[str, List[Any]]]]]]:
+    """Parse multiple chat messages and return conversation and coroutine."""
+    conversation = []
+    mm_data_tracker = AsyncMultimodalDataTracker(model_config)
+
+    for msg in messages:
+        sub_messages = parse_chat_message_content(msg, mm_data_tracker)
+        conversation.extend(sub_messages)
+
+    return conversation, mm_data_tracker.retrieve_all_mm_data()
 
 
 def resolve_hf_chat_template(
@@ -197,7 +244,8 @@ def resolve_hf_chat_template(
     processor: ProcessorMixin,
     chat_template: Optional[str],
     tools: Optional[list[dict[str, Any]]],
-) -> str:
+) -> Optional[str]:
+    """Resolve the appropriate chat template to use."""
 
     # 1. If chat_template is not None, return it
     if chat_template is not None:
@@ -213,7 +261,6 @@ def resolve_hf_chat_template(
     except Exception:
         logger.debug("Failed to load AutoTokenizer chat template for %s",
                      tokenizer.name_or_path)
-
     return None
 
 
@@ -229,6 +276,7 @@ def apply_chat_template(
     chat_template_kwargs: Optional[dict[str, Any]] = None,
     **kwargs: Any,
 ) -> str:
+    """Apply chat template to the conversation."""
     hf_chat_template = resolve_hf_chat_template(tokenizer, processor,
                                                 chat_template, tools)
 
diff --git a/tensorrt_llm/serve/openai_server.py b/tensorrt_llm/serve/openai_server.py
index 8b09603ce663..fb8ef8e083ee 100644
--- a/tensorrt_llm/serve/openai_server.py
+++ b/tensorrt_llm/serve/openai_server.py
@@ -22,7 +22,7 @@
 from tensorrt_llm.logger import logger
 from tensorrt_llm.serve.chat_utils import (ConversationMessage,
                                            apply_chat_template,
-                                           parse_chat_message_content)
+                                           parse_chat_messages_coroutines)
 from tensorrt_llm.serve.openai_protocol import (ChatCompletionRequest,
                                                 ChatCompletionResponse,
                                                 CompletionRequest,
@@ -102,6 +102,7 @@ def create_error_response(
         return JSONResponse(content=error_response.model_dump(),
                             status_code=error_response.code)
 
+
     def register_routes(self):
         self.app.add_api_route("/health", self.health, methods=["GET"])
         self.app.add_api_route("/version", self.version, methods=["GET"])
@@ -180,10 +181,9 @@ async def create_chat_response(
             ]
             sampling_params = request.to_sampling_params()
             postproc_args = ChatPostprocArgs.from_request(request)
+            disaggregated_params = to_llm_disaggregated_params(request.disaggregated_params)
 
-            for msg in request.messages:
-                conv_messages, mm_data = parse_chat_message_content(msg, self.model_config)
-                conversation.extend(conv_messages)
+            conversation, mm_coroutines = parse_chat_messages_coroutines(request.messages, self.model_config)
 
             prompt: str = apply_chat_template(
                 tokenizer=self.tokenizer,
@@ -196,18 +196,11 @@ async def create_chat_response(
                 chat_template=request.chat_template,
                 **(request.chat_template_kwargs or {}),
             )
-            sampling_params = request.to_sampling_params()
-            disaggregated_params = to_llm_disaggregated_params(request.disaggregated_params)
-            postproc_args = ChatPostprocArgs.from_request(request)
             prompt = prompt_inputs(prompt)
 
-            if mm_data:
-                if "multi_modal_data" not in prompt:
-                    prompt["multi_modal_data"] = {}
-                for media_type, media_values in mm_data.items():
-                    if media_type not in prompt["multi_modal_data"]:
-                        prompt["multi_modal_data"][media_type] = []
-                    prompt["multi_modal_data"][media_type].extend(media_values)
+            mm_data = await mm_coroutines
+            if mm_data is not None:
+                prompt["multi_modal_data"] = mm_data
 
             postproc_args.reasoning_parser = self.llm.args.reasoning_parser
             if conversation and conversation[-1].get(
diff --git a/tests/integration/defs/stress_test/stress_test.py b/tests/integration/defs/stress_test/stress_test.py
index da268ef7d09b..ad9b39a5a883 100644
--- a/tests/integration/defs/stress_test/stress_test.py
+++ b/tests/integration/defs/stress_test/stress_test.py
@@ -705,8 +705,6 @@ def create_genai_perf_command(model_name,
         model_name,
         "--tokenizer",
         model_path,
-        "--service-kind",
-        "openai",
         "--endpoint-type",
         "completions",
         "--random-seed",
diff --git a/tests/unittest/llmapi/apps/_test_openai_chat_multimodal.py b/tests/unittest/llmapi/apps/_test_openai_chat_multimodal.py
index 4644e01e5350..66ff8d59e176 100644
--- a/tests/unittest/llmapi/apps/_test_openai_chat_multimodal.py
+++ b/tests/unittest/llmapi/apps/_test_openai_chat_multimodal.py
@@ -6,15 +6,17 @@
 import pytest
 import yaml
 
+from tensorrt_llm.inputs import encode_base64_content_from_url
+
 from ..test_llm import get_model_path
 from .openai_server import RemoteOpenAIServer
 
 pytestmark = pytest.mark.threadleak(enabled=False)
 
 
-@pytest.fixture(scope="module", ids=["Qwen2-VL-7B-Instruct"])
+@pytest.fixture(scope="module", ids=["Qwen2.5-VL-3B-Instruct"])
 def model_name():
-    return "Qwen2-VL-7B-Instruct"
+    return "Qwen2.5-VL-3B-Instruct"
 
 
 @pytest.fixture(scope="module")
@@ -25,6 +27,10 @@ def temp_extra_llm_api_options_file(request):
         extra_llm_api_options_dict = {
             "kv_cache_config": {
                 "enable_block_reuse": False,
+                "free_gpu_memory_fraction": 0.6,
+            },
+            "build_config": {
+                "max_num_tokens": 16384,
             }
         }
 
@@ -111,6 +117,174 @@ def test_single_chat_session_image(client: openai.OpenAI, model_name: str):
         == chat_completion.choices[0].message.content
 
 
+@pytest.mark.asyncio(loop_scope="module")
+def test_single_chat_session_multi_image(client: openai.OpenAI,
+                                         model_name: str):
+    content_text = "Tell me the difference between two images"
+    image_url1 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
+    image_url2 = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
+    messages = [{
+        "role":
+        "user",
+        "content": [{
+            "type": "text",
+            "text": content_text
+        }, {
+            "type": "image_url",
+            "image_url": {
+                "url": image_url1
+            }
+        }, {
+            "type": "image_url",
+            "image_url": {
+                "url": image_url2
+            }
+        }],
+    }]
+
+    max_completion_tokens = 10
+    # test single completion
+    chat_completion = client.chat.completions.create(
+        model=model_name,
+        messages=messages,
+        max_completion_tokens=max_completion_tokens,
+        temperature=0.0,
+        logprobs=False)
+    assert chat_completion.id is not None
+    assert len(chat_completion.choices) == 1
+    message = chat_completion.choices[0].message
+    assert message.content is not None
+    assert message.role == "assistant"
+    # test finish_reason
+    finish_reason = chat_completion.choices[0].finish_reason
+    completion_tokens = chat_completion.usage.completion_tokens
+    if finish_reason == "length":
+        assert completion_tokens == 10
+    elif finish_reason == "stop":
+        assert completion_tokens <= 10
+    else:
+        raise RuntimeError(
+            f"finish_reason {finish_reason} not in [length, stop]")
+    # test max_tokens
+    legacy = client.chat.completions.create(
+        model=model_name,
+        messages=messages,
+        max_tokens=10,
+        temperature=0.0,
+        logprobs=False,
+    )
+    assert legacy.choices[0].message.content \
+        == chat_completion.choices[0].message.content
+
+
+@pytest.mark.asyncio(loop_scope="module")
+def test_single_chat_session_video(client: openai.OpenAI, model_name: str):
+    content_text = "Tell me what you see in the video briefly."
+    video_url = "https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4"
+    messages = [{
+        "role":
+        "user",
+        "content": [{
+            "type": "text",
+            "text": content_text
+        }, {
+            "type": "image_url",
+            "video_url": {
+                "url": video_url
+            }
+        }],
+    }]
+
+    max_completion_tokens = 10
+    # test single completion
+    chat_completion = client.chat.completions.create(
+        model=model_name,
+        messages=messages,
+        max_completion_tokens=max_completion_tokens,
+        temperature=0.0,
+        logprobs=False)
+    assert chat_completion.id is not None
+    assert len(chat_completion.choices) == 1
+    message = chat_completion.choices[0].message
+    assert message.content is not None
+    assert message.role == "assistant"
+    # test finish_reason
+    finish_reason = chat_completion.choices[0].finish_reason
+    completion_tokens = chat_completion.usage.completion_tokens
+    if finish_reason == "length":
+        assert completion_tokens == 10
+    elif finish_reason == "stop":
+        assert completion_tokens <= 10
+    else:
+        raise RuntimeError(
+            f"finish_reason {finish_reason} not in [length, stop]")
+    # test max_tokens
+    legacy = client.chat.completions.create(
+        model=model_name,
+        messages=messages,
+        max_tokens=10,
+        temperature=0.0,
+        logprobs=False,
+    )
+    assert legacy.choices[0].message.content \
+        == chat_completion.choices[0].message.content
+
+
+@pytest.mark.asyncio(loop_scope="module")
+def test_single_chat_session_image_embed(client: openai.OpenAI,
+                                         model_name: str):
+    content_text = "Describe the natural environment in the image."
+    image_url = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
+    image64 = encode_base64_content_from_url(image_url)
+    messages = [{
+        "role":
+        "user",
+        "content": [{
+            "type": "text",
+            "text": content_text
+        }, {
+            "type": "image_url",
+            "image_url": {
+                "url": "data:image/png;base64," + image64
+            }
+        }],
+    }]
+
+    max_completion_tokens = 10
+    # test single completion
+    chat_completion = client.chat.completions.create(
+        model=model_name,
+        messages=messages,
+        max_completion_tokens=max_completion_tokens,
+        temperature=0.0,
+        logprobs=False)
+    assert chat_completion.id is not None
+    assert len(chat_completion.choices) == 1
+    message = chat_completion.choices[0].message
+    assert message.content is not None
+    assert message.role == "assistant"
+    # test finish_reason
+    finish_reason = chat_completion.choices[0].finish_reason
+    completion_tokens = chat_completion.usage.completion_tokens
+    if finish_reason == "length":
+        assert completion_tokens == 10
+    elif finish_reason == "stop":
+        assert completion_tokens <= 10
+    else:
+        raise RuntimeError(
+            f"finish_reason {finish_reason} not in [length, stop]")
+    # test max_tokens
+    legacy = client.chat.completions.create(
+        model=model_name,
+        messages=messages,
+        max_tokens=10,
+        temperature=0.0,
+        logprobs=False,
+    )
+    assert legacy.choices[0].message.content \
+        == chat_completion.choices[0].message.content
+
+
 @pytest.mark.asyncio(loop_scope="module")
 async def test_chat_streaming_image(async_client: openai.AsyncOpenAI,
                                     model_name: str):
diff --git a/tests/unittest/llmapi/apps/_test_trtllm_serve_multimodal_example.py b/tests/unittest/llmapi/apps/_test_trtllm_serve_multimodal_example.py
index 5478238f9a71..5efa0171a199 100644
--- a/tests/unittest/llmapi/apps/_test_trtllm_serve_multimodal_example.py
+++ b/tests/unittest/llmapi/apps/_test_trtllm_serve_multimodal_example.py
@@ -12,9 +12,9 @@
 from test_llm import get_model_path
 
 
-@pytest.fixture(scope="module", ids=["Qwen2-VL-7B-Instruct"])
+@pytest.fixture(scope="module", ids=["Qwen2.5-VL-3B-Instruct"])
 def model_name():
-    return "Qwen2-VL-7B-Instruct"
+    return "Qwen2.5-VL-3B-Instruct"
 
 
 @pytest.fixture(scope="module")
@@ -25,6 +25,11 @@ def temp_extra_llm_api_options_file(request):
         extra_llm_api_options_dict = {
             "kv_cache_config": {
                 "enable_block_reuse": False,
+                "free_gpu_memory_fraction": 0.6,
+            },
+            # NOTE: This is for video support.
+            "build_config": {
+                "max_num_tokens": 16384,
             }
         }
 
@@ -57,7 +62,8 @@ def example_root():
 
 @pytest.mark.parametrize("exe, script",
                          [("python3", "openai_chat_client_for_multimodal.py"),
-                          ("bash", "curl_chat_client_for_multimodal.sh")])
+                          ("bash", "curl_chat_client_for_multimodal.sh"),
+                          ("bash", "genai_perf_client_for_multimodal.sh")])
 def test_trtllm_serve_examples(exe: str, script: str,
                                server: RemoteOpenAIServer, example_root: str):
     client_script = os.path.join(example_root, script)


Reviews:
user: login: yechank-nvidia
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-23T01:30:24Z
user: login: LinPoly
body: Would it be better to have a test file for `chat_utils.py`, like vLLM does https://github.com/vllm-project/vllm/blob/main/tests/entrypoints/test_chat_utils.py ?
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-24T14:01:16Z
user: login: symphonylyh
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-26T00:12:42Z
user: login: yechank-nvidia
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-30T13:03:26Z
user: login: yechank-nvidia
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-30T13:03:37Z
user: login: yechank-nvidia
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-30T13:03:50Z
user: login: yechank-nvidia
body: Need to test e2e performance on language model that the 'await mm_coroutines' is affecting negatively.

Will do E2E on genai-perf and will update here.

+ @LinPoly , I will add the chat_utils test on later PR.

UPDATE: Did E2E and concluded that async is not negatively impacting the perf.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-30T13:07:07Z
user: login: symphonylyh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-01T04:32:41Z
user: login: Funatiq
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T13:13:30Z
user: login: yechank-nvidia
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-22T00:19:08Z

Review comments:
diff_hunk: @@ -318,12 +318,15 @@ def __call__(
         mm_processor_kwargs['do_rescale'] = False
         processed_inputs = self._preprocess(text_prompt, mm_data,
                                             mm_processor_kwargs).to(self.device)
-
-        mm_features = self._process(
-            processed_inputs.get('pixel_values', None),
-            processed_inputs.get('pixel_values_videos', None),
-            processed_inputs.get('image_grid_thw', None),
-            processed_inputs.get('video_grid_thw', None))
+        if mm_data:
+            mm_features = self._process(
+                processed_inputs.get('pixel_values', None),
+                processed_inputs.get('pixel_values_videos', None),
+                processed_inputs.get('image_grid_thw', None),
+                processed_inputs.get('video_grid_thw', None))
+            prompt_tuning_config = [mm_features, None, None]
+        else:
+            prompt_tuning_config = None
path: tensorrt_llm/_torch/models/modeling_qwen2vl.py
user: login: yechank-nvidia
body: Hi @symphonylyh, I changed the logic here to do the inference even when no mm_data is given.
created_at: 2025-04-23T01:30:24Z
author_association: COLLABORATOR
diff_hunk: @@ -318,12 +318,15 @@ def __call__(
         mm_processor_kwargs['do_rescale'] = False
         processed_inputs = self._preprocess(text_prompt, mm_data,
                                             mm_processor_kwargs).to(self.device)
-
-        mm_features = self._process(
-            processed_inputs.get('pixel_values', None),
-            processed_inputs.get('pixel_values_videos', None),
-            processed_inputs.get('image_grid_thw', None),
-            processed_inputs.get('video_grid_thw', None))
+        if mm_data:
+            mm_features = self._process(
+                processed_inputs.get('pixel_values', None),
+                processed_inputs.get('pixel_values_videos', None),
+                processed_inputs.get('image_grid_thw', None),
+                processed_inputs.get('video_grid_thw', None))
+            prompt_tuning_config = [mm_features, None, None]
+        else:
+            prompt_tuning_config = None
path: tensorrt_llm/_torch/models/modeling_qwen2vl.py
user: login: symphonylyh
body: FYI, we're working on deprecate the prompt tuning idea from multimodal, where just passing mm_features as a field in llmrequest is enough https://github.com/NVIDIA/TensorRT-LLM/pull/3855/files#diff-29ee5c5f987a12d0fc5f359ddb3e6e3b411f019624e57458d71fe9810251be2fR150
in this text-only mode, I think it's also just setting "mm_embedding": None
created_at: 2025-04-26T00:06:05Z
author_association: COLLABORATOR
diff_hunk: @@ -224,3 +231,112 @@ def default_video_loader(prompts, videos, image_data_format="pt", num_frames=8):
     "qwen2_vl": format_qwen2_vl_input,
     "qwen2_5_vl": format_qwen2_vl_input,
 }
+
+
+async def async_load_image(
+        image: str,
+        format: str = "pt",
+        device: str = "cuda") -> Union[Image.Image, torch.Tensor]:
+    # print('[INFO] async_load_image is called')
path: tensorrt_llm/inputs/utils.py
user: login: symphonylyh
body: cleanup reminder
created_at: 2025-04-26T00:10:19Z
author_association: COLLABORATOR
diff_hunk: @@ -224,3 +231,112 @@ def default_video_loader(prompts, videos, image_data_format="pt", num_frames=8):
     "qwen2_vl": format_qwen2_vl_input,
     "qwen2_5_vl": format_qwen2_vl_input,
 }
+
+
+async def async_load_image(
+        image: str,
+        format: str = "pt",
+        device: str = "cuda") -> Union[Image.Image, torch.Tensor]:
+    # print('[INFO] async_load_image is called')
+    assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
+
+    parsed_url = urlparse(image)
+
+    def load_and_conver_image(image):
path: tensorrt_llm/inputs/utils.py
user: login: symphonylyh
body: nit: conver-> convert?
created_at: 2025-04-26T00:10:34Z
author_association: COLLABORATOR
diff_hunk: @@ -224,3 +231,112 @@ def default_video_loader(prompts, videos, image_data_format="pt", num_frames=8):
     "qwen2_vl": format_qwen2_vl_input,
     "qwen2_5_vl": format_qwen2_vl_input,
 }
+
+
+async def async_load_image(
+        image: str,
+        format: str = "pt",
+        device: str = "cuda") -> Union[Image.Image, torch.Tensor]:
+    # print('[INFO] async_load_image is called')
+    assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
+
+    parsed_url = urlparse(image)
+
+    def load_and_conver_image(image):
+        image = Image.open(image)
+        image.load()
+        return image.convert("RGB")
+
+    if parsed_url.scheme in ["http", "https"]:
+        async with aiohttp.ClientSession() as session:
+            async with session.get(image) as response:
+                content = await response.read()
+                image = load_and_conver_image(BytesIO(content))
+    elif parsed_url.scheme == "data":
+        data_spec, data = parsed_url.path.split(",", 1)
+        media_type, data_type = data_spec.split(";", 1)
+
+        if data_type != "base64":
+            msg = "Only base64 data URLs are supported for now."
+            raise NotImplementedError(msg)
+
+        content = base64.b64decode(data)
+        image = load_and_conver_image(BytesIO(content))
+    else:
+        filepath = Path(parsed_url.path)
+        image = load_and_conver_image(filepath)
+
+    # print('[INFO] async_load_image is finished')
+    if format == "pt":
+        return ToTensor()(image).to(device=device)
+    else:
+        return image
+
+
+async def async_load_video(
+        video: str,
+        num_frames: int = 10,
+        format: str = "pt",
+        device: str = "cuda") -> Union[List[Image.Image], List[torch.Tensor]]:
+    assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
+
+    # Load video frames from a video file
+    if video.startswith("http://") or video.startswith("https://"):
+        async with aiohttp.ClientSession() as session:
+            async with session.get(video) as response:
+                with tempfile.NamedTemporaryFile(delete=False,
+                                                 suffix='.mp4') as tmp:
+                    tmp.write(await response.content.read())
+                    video_path = tmp.name
+    else:
+        video_path = video
+
+    vidcap = cv2.VideoCapture(video_path)
+
+    if not vidcap.isOpened():
+        raise ValueError(
+            f"Video '{video}' could not be opened. Make sure opencv is installed with video support."
+        )
+
+    # Find the last frame as frame count might not be accurate
+    frame_count = int(vidcap.get(cv2.CAP_PROP_FRAME_COUNT))
+    while frame_count > 0:
+        vidcap.set(cv2.CAP_PROP_POS_FRAMES, frame_count - 1)
+        if vidcap.grab():
+            break
+        frame_count -= 1
+    else:
+        raise ValueError(f"Video '{video}' has no frames.")
+
+    # Extract frames uniformly
+    indices = np.round(np.linspace(0, frame_count - 1, num_frames)).astype(int)
+    frames = {}
+    for index in indices:
+        if index in frames:
+            continue
+        vidcap.set(cv2.CAP_PROP_POS_FRAMES, index)
+        success, frame = vidcap.read()
+        if not success:
+            continue
+        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+        frames[index] = Image.fromarray(frame)
+
+    if video.startswith("http://") or video.startswith("https://"):
+        os.unlink(video_path)
+
+    return [
+        ToTensor()(frames[index]).to(
+            device=device) if format == "pt" else frames[index]
+        for index in indices if index in frames
+    ]
path: tensorrt_llm/inputs/utils.py
user: login: symphonylyh
body: can this part just call `load_video()`?
created_at: 2025-04-26T00:11:27Z
author_association: COLLABORATOR
diff_hunk: @@ -224,3 +231,112 @@ def default_video_loader(prompts, videos, image_data_format="pt", num_frames=8):
     "qwen2_vl": format_qwen2_vl_input,
     "qwen2_5_vl": format_qwen2_vl_input,
 }
+
+
+async def async_load_image(
+        image: str,
+        format: str = "pt",
+        device: str = "cuda") -> Union[Image.Image, torch.Tensor]:
+    # print('[INFO] async_load_image is called')
+    assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
+
+    parsed_url = urlparse(image)
+
+    def load_and_conver_image(image):
+        image = Image.open(image)
+        image.load()
+        return image.convert("RGB")
+
+    if parsed_url.scheme in ["http", "https"]:
+        async with aiohttp.ClientSession() as session:
+            async with session.get(image) as response:
+                content = await response.read()
+                image = load_and_conver_image(BytesIO(content))
+    elif parsed_url.scheme == "data":
+        data_spec, data = parsed_url.path.split(",", 1)
+        media_type, data_type = data_spec.split(";", 1)
+
+        if data_type != "base64":
+            msg = "Only base64 data URLs are supported for now."
+            raise NotImplementedError(msg)
+
+        content = base64.b64decode(data)
+        image = load_and_conver_image(BytesIO(content))
+    else:
+        filepath = Path(parsed_url.path)
+        image = load_and_conver_image(filepath)
+
+    # print('[INFO] async_load_image is finished')
+    if format == "pt":
+        return ToTensor()(image).to(device=device)
+    else:
+        return image
+
+
+async def async_load_video(
+        video: str,
+        num_frames: int = 10,
+        format: str = "pt",
+        device: str = "cuda") -> Union[List[Image.Image], List[torch.Tensor]]:
+    assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
+
+    # Load video frames from a video file
+    if video.startswith("http://") or video.startswith("https://"):
+        async with aiohttp.ClientSession() as session:
+            async with session.get(video) as response:
+                with tempfile.NamedTemporaryFile(delete=False,
+                                                 suffix='.mp4') as tmp:
+                    tmp.write(await response.content.read())
+                    video_path = tmp.name
+    else:
+        video_path = video
+
+    vidcap = cv2.VideoCapture(video_path)
+
+    if not vidcap.isOpened():
+        raise ValueError(
+            f"Video '{video}' could not be opened. Make sure opencv is installed with video support."
+        )
+
+    # Find the last frame as frame count might not be accurate
+    frame_count = int(vidcap.get(cv2.CAP_PROP_FRAME_COUNT))
+    while frame_count > 0:
+        vidcap.set(cv2.CAP_PROP_POS_FRAMES, frame_count - 1)
+        if vidcap.grab():
+            break
+        frame_count -= 1
+    else:
+        raise ValueError(f"Video '{video}' has no frames.")
+
+    # Extract frames uniformly
+    indices = np.round(np.linspace(0, frame_count - 1, num_frames)).astype(int)
+    frames = {}
+    for index in indices:
+        if index in frames:
+            continue
+        vidcap.set(cv2.CAP_PROP_POS_FRAMES, index)
+        success, frame = vidcap.read()
+        if not success:
+            continue
+        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+        frames[index] = Image.fromarray(frame)
+
+    if video.startswith("http://") or video.startswith("https://"):
+        os.unlink(video_path)
+
+    return [
+        ToTensor()(frames[index]).to(
+            device=device) if format == "pt" else frames[index]
+        for index in indices if index in frames
+    ]
path: tensorrt_llm/inputs/utils.py
user: login: yechank-nvidia
body: Changed. Thx
created_at: 2025-04-30T13:03:25Z
author_association: COLLABORATOR
diff_hunk: @@ -224,3 +231,112 @@ def default_video_loader(prompts, videos, image_data_format="pt", num_frames=8):
     "qwen2_vl": format_qwen2_vl_input,
     "qwen2_5_vl": format_qwen2_vl_input,
 }
+
+
+async def async_load_image(
+        image: str,
+        format: str = "pt",
+        device: str = "cuda") -> Union[Image.Image, torch.Tensor]:
+    # print('[INFO] async_load_image is called')
+    assert format in ["pt", "pil"], "format must be either Pytorch or PIL"
+
+    parsed_url = urlparse(image)
+
+    def load_and_conver_image(image):
path: tensorrt_llm/inputs/utils.py
user: login: yechank-nvidia
body: Fixed. Thx.
created_at: 2025-04-30T13:03:37Z
author_association: COLLABORATOR
diff_hunk: @@ -224,3 +231,112 @@ def default_video_loader(prompts, videos, image_data_format="pt", num_frames=8):
     "qwen2_vl": format_qwen2_vl_input,
     "qwen2_5_vl": format_qwen2_vl_input,
 }
+
+
+async def async_load_image(
+        image: str,
+        format: str = "pt",
+        device: str = "cuda") -> Union[Image.Image, torch.Tensor]:
+    # print('[INFO] async_load_image is called')
path: tensorrt_llm/inputs/utils.py
user: login: yechank-nvidia
body: Removed. Thx.
created_at: 2025-04-30T13:03:50Z
author_association: COLLABORATOR
diff_hunk: @@ -318,12 +318,15 @@ def __call__(
         mm_processor_kwargs['do_rescale'] = False
         processed_inputs = self._preprocess(text_prompt, mm_data,
                                             mm_processor_kwargs).to(self.device)
-
-        mm_features = self._process(
-            processed_inputs.get('pixel_values', None),
-            processed_inputs.get('pixel_values_videos', None),
-            processed_inputs.get('image_grid_thw', None),
-            processed_inputs.get('video_grid_thw', None))
+        if mm_data:
+            mm_features = self._process(
+                processed_inputs.get('pixel_values', None),
+                processed_inputs.get('pixel_values_videos', None),
+                processed_inputs.get('image_grid_thw', None),
+                processed_inputs.get('video_grid_thw', None))
+            prompt_tuning_config = [mm_features, None, None]
+        else:
+            prompt_tuning_config = None
path: tensorrt_llm/_torch/models/modeling_qwen2vl.py
user: login: symphonylyh
body: just merged. you can rebase to the new simpler usage now
created_at: 2025-05-01T04:32:41Z
author_association: COLLABORATOR
diff_hunk: @@ -29,7 +87,6 @@ def load_video(
         num_frames: int = 10,
         format: str = "pt",
         device: str = "cuda") -> Union[List[Image.Image], List[torch.Tensor]]:
-    import cv2
path: tensorrt_llm/inputs/utils.py
user: login: Funatiq
body: Please keep this import local! I will revert this change in #4541.
created_at: 2025-05-21T13:13:29Z
author_association: COLLABORATOR
diff_hunk: @@ -29,7 +87,6 @@ def load_video(
         num_frames: int = 10,
         format: str = "pt",
         device: str = "cuda") -> Union[List[Image.Image], List[torch.Tensor]]:
-    import cv2
path: tensorrt_llm/inputs/utils.py
user: login: yechank-nvidia
body: Can I ask why this should be local?
created_at: 2025-05-22T00:19:08Z
author_association: COLLABORATOR

PR comments:
user: login: yechank-nvidia
created_at: 2025-04-23T01:26:00Z
body: /bot run
author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-07T05:35:54Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-07T05:41:51Z
body: [PR_Github #4324](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4324/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-07T08:00:47Z
body: [PR_Github #4324](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4324/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3100](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3100/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-08T10:20:41Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-08T10:26:42Z
body: [PR_Github #4541](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4541/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-08T14:35:50Z
body: [PR_Github #4541](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4541/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3263](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3263/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-09T00:32:48Z
body: /bot help
author_association: COLLABORATOR
user: login: github-actions[bot]
created_at: 2025-05-09T00:32:57Z
body: ## GitHub Bot Help

`/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...`

Provide a user friendly way for developers to interact with a Jenkins server.

Run `/bot [-h|--help]` to print this help message.

See details below for each supported subcommand.

<details>

`run  [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]`

Launch build/test pipelines. All previously running jobs will be killed.

`--disable-fail-fast ` *(OPTIONAL)* : Disable fail fast on build/tests/infra failures.

`--skip-test ` *(OPTIONAL)* : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does **NOT** update GitHub check status.

`--stage-list "A10-1, xxx"` *(OPTIONAL)* : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does **NOT** update GitHub check status.

`--gpu-type "A30, H100_PCIe"` *(OPTIONAL)* : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does **NOT** update GitHub check status.

`--only-multi-gpu-test ` *(OPTIONAL)* : Only run the multi-GPU tests. Note: Does **NOT** update GitHub check status.

`--disable-multi-gpu-test ` *(OPTIONAL)* : Disable the multi-GPU tests. Note: Does **NOT** update GitHub check status.

`--add-multi-gpu-test ` *(OPTIONAL)* : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

`--extra-stage "H100_PCIe-[Post-Merge]-1, xxx"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

### kill

`kill  `

Kill all running builds associated with pull request.

### skip

`skip --comment COMMENT `

Skip testing for latest commit on pull request. `--comment "Reason for skipping build/test"` is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

### reuse-pipeline

`reuse-pipeline `

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

</details>
author_association: NONE
user: login: yechank-nvidia
created_at: 2025-05-09T00:35:38Z
body: /bot run --reuse-test 3263
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-09T00:41:02Z
body: [PR_Github #4610](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4610/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-09T07:31:57Z
body: [PR_Github #4610](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4610/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3322](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3322/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-09T07:39:35Z
body: /bot run --reuse-test 3322
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-09T07:45:23Z
body: [PR_Github #4679](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4679/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-09T15:31:29Z
body: [PR_Github #4679](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4679/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3374](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3374/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-11T12:52:46Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-11T12:58:50Z
body: [PR_Github #4782](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4782/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-11T14:37:52Z
body: [PR_Github #4782](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4782/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3457](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3457/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-12T00:29:16Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-12T00:34:48Z
body: [PR_Github #4793](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4793/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-12T02:33:48Z
body: [PR_Github #4793](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4793/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3466](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3466/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-12T05:47:59Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-12T05:53:26Z
body: [PR_Github #4833](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4833/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-12T12:35:34Z
body: [PR_Github #4833](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4833/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3499](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3499/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-13T02:22:36Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-13T02:28:31Z
body: [PR_Github #4934](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4934/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-13T04:33:09Z
body: [PR_Github #4934](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/4934/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3579](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3579/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-13T15:11:26Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-13T15:19:18Z
body: [PR_Github #5037](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5037/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-13T18:41:39Z
body: [PR_Github #5037](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5037/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3665](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3665/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-14T06:02:00Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T06:09:22Z
body: [PR_Github #5114](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5114/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T09:26:34Z
body: [PR_Github #5114](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5114/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3726](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3726/) completed with status: 'FAILURE'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-14T10:13:54Z
body: /bot run
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T10:20:54Z
body: [PR_Github #5162](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5162/) [ run ] triggered by Bot
author_association: COLLABORATOR
user: login: tensorrt-cicd
created_at: 2025-05-14T18:14:02Z
body: [PR_Github #5162](https://nv/trt-llm-cicd/job/helpers/job/PR_Github/5162/) [ run ] completed with state `SUCCESS`
[/LLM/main/L0_MergeRequest_PR pipeline #3764](https://nv/trt-llm-cicd/job/main/job/L0_MergeRequest_PR/3764/) completed with status: 'SUCCESS'

author_association: COLLABORATOR
user: login: yechank-nvidia
created_at: 2025-05-15T06:36:04Z
body: @symphonylyh Hi Haohang, can you merge this into the maiN?
author_association: COLLABORATOR