Repository: vllm-project/vllm
Pull request #14275: [VLM] Support Pixtral-HF on V1
Author: login: lk-chen
Labels: name: documentation
description: Improvements or additions to documentation
name: ready
description: ONLY add when PR is ready to merge/full CI is needed
name: v1
Created: 2025-03-05T10:19:34Z | Merged: 2025-03-06T08:58:41Z

Original work done in #11409. This PR addressed the comment in https://github.com/vllm-project/vllm/pull/13080#issuecomment-2653051780

-----

Tested with following cases:
* on main branch: `VLLM_USE_V1=0 python examples/offline_inference/vision_language.py --model pixtral_hf --num-prompts=2`
* on current branch: `VLLM_USE_V1=0 python examples/offline_inference/vision_language.py --model pixtral_hf --num-prompts=2`
* on current branch: `VLLM_USE_V1=1 python examples/offline_inference/vision_language.py --model pixtral_hf --num-prompts=2`

all generating the same results:
```
The image features a prominent structure partially obscured by cherry blossom branches in the foreground. The structure appears to be a tall, modern tower with a distinctive design, possibly a telecommunications or observation tower. The cherry blossoms are in full bloom, creating a picturesque and serene scene. The sky is clear and blue,
The image depicts a scene featuring a tall, modern tower with a distinctive architectural design, partially obscured by cherry blossom trees in full bloom. The cherry blossoms, with their delicate pink flowers, dominate the foreground, creating a beautiful contrast against the clear blue sky. The tower appears to be a notable landmark, characterized by
```

Also tested multi-image cases:
* `VLLM_USE_V1=0 python examples/offline_inference/vision_language_multi_image.py --model pixtral_hf`
* `VLLM_USE_V1=1 python examples/offline_inference/vision_language_multi_image.py --model pixtral_hf`

cc @DarkLight1337 @ywang96


Diff:
diff --git a/docs/source/models/supported_models.md b/docs/source/models/supported_models.md
index fc363585b0e7..6cae2971ab26 100644
--- a/docs/source/models/supported_models.md
+++ b/docs/source/models/supported_models.md
@@ -866,7 +866,7 @@ See [this page](#generative-models) for more information on how to use generativ
 - * `PixtralForConditionalGeneration`
   * Pixtral
   * T + I<sup>+</sup>
-  * `mistralai/Pixtral-12B-2409`, `mistral-community/pixtral-12b` (see note), etc.
+  * `mistralai/Pixtral-12B-2409`, `mistral-community/pixtral-12b`, etc.
   *
   * ✅︎
   * ✅︎
@@ -930,10 +930,6 @@ For more details, please see: <gh-pr:4087#issuecomment-2250397630>
 Currently the PaliGemma model series is implemented without PrefixLM attention mask. This model series may be deprecated in a future release.
 :::
 
-:::{note}
-`mistral-community/pixtral-12b` does not support V1 yet.
-:::
-
 :::{note}
 To use Qwen2.5-VL series models, you have to install Hugging Face Transformers library from source via `pip install git+https://github.com/huggingface/transformers`.
 :::
diff --git a/vllm/model_executor/models/llava.py b/vllm/model_executor/models/llava.py
index 66b79f809bc9..e83dfd320bb6 100644
--- a/vllm/model_executor/models/llava.py
+++ b/vllm/model_executor/models/llava.py
@@ -4,7 +4,7 @@
 from collections.abc import Iterable, Mapping, Sequence
 from functools import cached_property
 from typing import (Final, List, Literal, Optional, Protocol, Set, Tuple,
-                    TypedDict, TypeVar, Union)
+                    TypedDict, TypeVar, Union, cast)
 
 import torch
 import torch.nn as nn
@@ -35,6 +35,7 @@
                                         PromptReplacement, PromptUpdate)
 from vllm.multimodal.profiling import BaseDummyInputsBuilder, ProcessorInputs
 from vllm.sequence import IntermediateTensors
+from vllm.utils import JSONTree, flatten_2d_lists, json_map_leaves
 
 from .clip import CLIPVisionModel
 from .interfaces import SupportsMultiModal, SupportsPP
@@ -56,6 +57,25 @@ class LlavaImagePixelInputs(TypedDict):
     in which case the data is passed as a list instead of a batched tensor.
     """
 
+    feat_is_patch: Union[torch.Tensor, List[torch.Tensor]]
+    """
+    A boolean mask indicating which image features correspond
+    to patch tokens.
+
+    Shape: `(batch_size, num_crops, num_patch)`
+    """
+
+    embed_is_patch: Union[torch.Tensor, List[torch.Tensor]]
+    """
+    A boolean mask indicating which image embeddings correspond
+    to patch tokens.
+    
+    Shape: `(batch_size, num_embeds)`
+    """
+
+    num_crops: torch.Tensor
+    """Shape: `(batch_size, num_images)`"""
+
 
 class LlavaImageEmbeddingInputs(TypedDict):
     type: Literal["image_embeds"]
@@ -65,6 +85,25 @@ class LlavaImageEmbeddingInputs(TypedDict):
     `hidden_size` must match the hidden size of language model backbone.
     """
 
+    feat_is_patch: Union[torch.Tensor, List[torch.Tensor]]
+    """
+    A boolean mask indicating which image features correspond
+    to patch tokens.
+
+    Shape: `(batch_size, num_crops, num_patch)`
+    """
+
+    embed_is_patch: Union[torch.Tensor, List[torch.Tensor]]
+    """
+    A boolean mask indicating which image embeddings correspond
+    to patch tokens.
+    
+    Shape: `(batch_size, num_embeds)`
+    """
+
+    num_crops: torch.Tensor
+    """Shape: `(batch_size, num_images)`"""
+
 
 LlavaImageInputs = Union[LlavaImagePixelInputs, LlavaImageEmbeddingInputs]
 
@@ -317,6 +356,26 @@ def _call_hf_processor(
                     for p, (h, w) in zip(pixel_values, image_sizes)
                 ]
 
+            hf_config = self.info.get_hf_config()
+
+            tile_sizes = [
+                get_pixtral_hf_image_feature_grid_size(
+                    hf_config.vision_config,
+                    image_width=pixel_value.shape[-1],
+                    image_height=pixel_value.shape[-2])
+                for pixel_value in processed_outputs["pixel_values"]
+            ]
+            num_crops = torch.tensor([(ncols + 1) * nrows
+                                      for ncols, nrows in tile_sizes])
+            # Each image may result to masks of different sizes, so we need to
+            # flatten the list and later use `num_crops` to get per-image masks.
+            embed_is_patch = torch.tensor(
+                flatten_2d_lists([([True] * ncols + [False]) * nrows
+                                  for ncols, nrows in tile_sizes]))
+            processed_outputs["num_crops"] = num_crops
+            processed_outputs["embed_is_patch"] = embed_is_patch
+            processed_outputs["feat_is_patch"] = embed_is_patch
+
         return processed_outputs
 
     def _get_mm_fields_config(
@@ -324,7 +383,13 @@ def _get_mm_fields_config(
         hf_inputs: BatchFeature,
         hf_processor_mm_kwargs: Mapping[str, object],
     ) -> Mapping[str, MultiModalFieldConfig]:
+        num_crops = hf_inputs.get("num_crops", torch.empty(0)).view(-1)
         return dict(
+            feat_is_patch=MultiModalFieldConfig.flat_from_sizes(
+                "image", num_crops),
+            embed_is_patch=MultiModalFieldConfig.flat_from_sizes(
+                "image", num_crops),
+            num_crops=MultiModalFieldConfig.batched("image"),
             pixel_values=MultiModalFieldConfig.batched("image"),
             image_embeds=MultiModalFieldConfig.batched("image"),
         )
@@ -562,6 +627,23 @@ def _parse_and_validate_image_input(
         if pixel_values is None and image_embeds is None:
             return None
 
+        feat_is_patch = kwargs.pop("feat_is_patch", None)
+        if feat_is_patch is not None and not isinstance(
+                feat_is_patch, (torch.Tensor, list)):
+            raise ValueError("Incorrect type of feat_is_patch. "
+                             f"Got type: {type(feat_is_patch)}")
+
+        embed_is_patch = kwargs.pop("embed_is_patch", None)
+        if embed_is_patch is not None and not isinstance(
+                embed_is_patch, (torch.Tensor, list)):
+            raise ValueError("Incorrect type of embed_is_patch. "
+                             f"Got type: {type(embed_is_patch)}")
+
+        num_crops = kwargs.pop("num_crops", None)
+        if num_crops is not None and not isinstance(num_crops, torch.Tensor):
+            raise ValueError("Incorrect type of num_crops. "
+                             f"Got type: {type(num_crops)}")
+
         if pixel_values is not None:
             if not isinstance(pixel_values, (torch.Tensor, list)):
                 raise ValueError("Incorrect type of pixel values. "
@@ -571,12 +653,18 @@ def _parse_and_validate_image_input(
                 return LlavaImagePixelInputs(
                     type="pixel_values",
                     data=flatten_bn(pixel_values),
+                    feat_is_patch=feat_is_patch,
+                    embed_is_patch=embed_is_patch,
+                    num_crops=num_crops,
                 )
 
             return LlavaImagePixelInputs(
                 type="pixel_values",
                 data=self._validate_pixel_values(
                     flatten_bn(pixel_values, concat=True)),
+                feat_is_patch=feat_is_patch,
+                embed_is_patch=embed_is_patch,
+                num_crops=num_crops,
             )
 
         if image_embeds is not None:
@@ -587,6 +675,9 @@ def _parse_and_validate_image_input(
             return LlavaImageEmbeddingInputs(
                 type="image_embeds",
                 data=flatten_bn(image_embeds, concat=True),
+                feat_is_patch=feat_is_patch,
+                embed_is_patch=embed_is_patch,
+                num_crops=num_crops,
             )
 
         raise AssertionError("This line should be unreachable.")
@@ -633,16 +724,74 @@ def _process_image_input(self,
 
         assert self.vision_tower is not None
         image_features = self._process_image_pixels(image_input)
-        return self.multi_modal_projector(image_features)
 
-    def get_multimodal_embeddings(
-        self, **kwargs
-    ) -> Union[list[torch.Tensor], torch.Tensor, tuple[torch.Tensor, ...]]:
+        if isinstance(image_features, torch.Tensor):
+            return self.multi_modal_projector(image_features)
+
+        feature_sizes = [
+            image_feature.shape[0] for image_feature in image_features
+        ]
+
+        image_embeds = self.multi_modal_projector(torch.cat(image_features))
+        image_embeds = torch.split(image_embeds, feature_sizes)
+        return image_embeds
+
+    def _get_mm_embeds(
+            self,
+            features: torch.Tensor,  # Shape: (num_crop, num_patch, d)
+            feat_is_patch: torch.Tensor,  # Shape: (num_crop, num_patch)
+            num_crops: torch.Tensor,  # Shape: (num_images,)
+            embed_is_patch: torch.Tensor,  # Shape: (num_embeds,)
+    ) -> list[torch.Tensor]:
+        """Scatter the patch features into a contiguous tensor that corresponds
+        to the embedding tokens defined by the multimodal processor.
+
+        Mostly copied from `Molmo._get_mm_embeds`. See following fixme comment.
+        """
+
+        # Insert columns of nan values according to `feat_is_patch`. This work
+        # ideally should be done in `_process_image_input`, but
+        # `_process_image_input` is used in both V0 and V1 path. It's safer to
+        # put the logic here.
+        # FIXME: Move this logic to `_process_image_input` when v0 is
+        # deprecated. Merge this function with `Molmo._get_mm_embeds`.
+        feat_is_patch = feat_is_patch.view(-1)
+        embed_is_patch = embed_is_patch.view(-1)
+        expanded_embedding = torch.full(
+            (sum(num_crops), *features.shape[1:]),
+            torch.nan,
+            dtype=features.dtype).to(features.device)
+        expanded_embedding[feat_is_patch] = features
+
+        num_crops_per_image = num_crops.tolist()
+        feats_per_image = expanded_embedding.split(num_crops_per_image)
+        f_is_patch_per_image = feat_is_patch.split(num_crops_per_image)
+
+        embed_dim = expanded_embedding.shape[-1]
+        num_embeds = embed_is_patch.shape[0]
+
+        embeds_in_batch = list[torch.Tensor]()
+        for feats, f_is_patch in zip(feats_per_image, f_is_patch_per_image):
+            embeds = feats.new_full((num_embeds, embed_dim), torch.nan)
+            embeds[embed_is_patch] = feats[f_is_patch]
+            embeds_in_batch.append(embeds)
+
+        return embeds_in_batch
+
+    def get_multimodal_embeddings(self, **kwargs) -> Optional[NestedTensors]:
         image_input = self._parse_and_validate_image_input(**kwargs)
         if image_input is None:
             return None
         vision_embeddings = self._process_image_input(image_input)
-        return vision_embeddings
+        if kwargs.get("v0_path", False):
+            return vision_embeddings
+        else:
+            nested_emb = [
+                self._get_mm_embeds(*args) for args in zip(
+                    vision_embeddings, image_input["feat_is_patch"],
+                    image_input["num_crops"], image_input["embed_is_patch"])
+            ]
+            return flatten_2d_lists(nested_emb)
 
     def get_input_embeddings(
         self,
@@ -651,8 +800,15 @@ def get_input_embeddings(
     ) -> torch.Tensor:
         inputs_embeds = self.language_model.get_input_embeddings(input_ids)
         if multimodal_embeddings is not None:
+            # Extract the patch tokens
+            patch_embeddings = json_map_leaves(
+                lambda x: x[~x.isnan()].view(-1, *x.shape[1:]),
+                cast(JSONTree[torch.Tensor], multimodal_embeddings),
+            )
+
             inputs_embeds = merge_multimodal_embeddings(
-                input_ids, inputs_embeds, multimodal_embeddings,
+                input_ids, inputs_embeds, cast(NestedTensors,
+                                               patch_embeddings),
                 self.config.image_token_index)
         return inputs_embeds
 
@@ -705,6 +861,7 @@ def forward(
         # NOTE: In v1, inputs_embeds is always generated at model runner, this
         # condition is for v0 compatibility.
         elif inputs_embeds is None:
+            kwargs.update({"v0_path": True})
             vision_embeddings = self.get_multimodal_embeddings(**kwargs)
             inputs_embeds = self.get_input_embeddings(input_ids,
                                                       vision_embeddings)
diff --git a/vllm/model_executor/models/molmo.py b/vllm/model_executor/models/molmo.py
index cc571bc24bac..554080533059 100644
--- a/vllm/model_executor/models/molmo.py
+++ b/vllm/model_executor/models/molmo.py
@@ -1484,8 +1484,8 @@ def _parse_and_validate_image_input(
 
         img_patch_id = kwargs.pop("img_patch_id", None)
         if not isinstance(img_patch_id, torch.Tensor):
-            raise ValueError("Incorrect type of num_crops. "
-                             f"Got type: {type(num_crops)}")
+            raise ValueError("Incorrect type of img_patch_id. "
+                             f"Got type: {type(img_patch_id)}")
         self.img_patch_id = img_patch_id.flatten().unique().item()
 
         return MolmoImageInputs(
diff --git a/vllm/model_executor/models/pixtral.py b/vllm/model_executor/models/pixtral.py
index d2388dda3f4a..8acc07ac353a 100644
--- a/vllm/model_executor/models/pixtral.py
+++ b/vllm/model_executor/models/pixtral.py
@@ -1042,9 +1042,13 @@ def forward(
             for img in pixel_values
         ]
 
+        patch_embeds = [
+            p.flatten(2).permute(0, 2, 1) for p in patch_embeds_list
+        ]
+        embed_sizes = [p.shape[1] for p in patch_embeds]
+
         # flatten to a single sequence
-        patch_embeds = torch.cat(
-            [p.flatten(2).permute(0, 2, 1) for p in patch_embeds_list], dim=1)
+        patch_embeds = torch.cat(patch_embeds, dim=1)
         patch_embeds = self.ln_pre(patch_embeds)
 
         # positional embeddings
@@ -1075,6 +1079,8 @@ def forward(
         out = resolve_visual_encoder_outputs(out, feature_sample_layers, None,
                                              self.config.num_hidden_layers)
 
+        # squeeze dim 0 and split into separate tensors for each image
+        out = torch.split(torch.squeeze(out), embed_sizes)
         return out
 
     # (TODO) Add prefix argument for filtering out weights to be loaded


Reviews:
user: login: DarkLight1337
body: Overall the code looks good, I'll test it out later tonight. Thanks again for working on this!
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T10:45:29Z
user: login: DarkLight1337
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T06:19:50Z
user: login: lk-chen
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T06:49:39Z
user: login: DarkLight1337
body: LGTM now, thanks!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-06T06:56:32Z

Review comments:
diff_hunk: @@ -317,14 +356,39 @@ def _call_hf_processor(
                     for p, (h, w) in zip(pixel_values, image_sizes)
                 ]
 
+            hf_config = self.info.get_hf_config()
+
+            tile_sizes = [
+                get_pixtral_hf_image_feature_grid_size(
+                    hf_config.vision_config,
+                    image_width=pixel_value.shape[-1],
+                    image_height=pixel_value.shape[-2])
+                for pixel_value in processed_outputs["pixel_values"]
+            ]
+            num_crops = torch.tensor([(ncols + 1) * nrows
+                                      for ncols, nrows in tile_sizes])
+            embed_is_patch = [([True] * ncols + [False]) * nrows
+                              for ncols, nrows in tile_sizes]
+            processed_outputs["num_crops"] = num_crops
+            processed_outputs["embed_is_patch"] = embed_is_patch
+            processed_outputs["feat_is_patch"] = embed_is_patch
+
         return processed_outputs
 
     def _get_mm_fields_config(
         self,
         hf_inputs: BatchFeature,
         hf_processor_mm_kwargs: Mapping[str, object],
     ) -> Mapping[str, MultiModalFieldConfig]:
+
path: vllm/model_executor/models/llava.py
user: login: DarkLight1337
body: ```suggestion
```
No need for this empty line
created_at: 2025-03-05T10:44:37Z
author_association: MEMBER
diff_hunk: @@ -778,8 +934,7 @@ def apply(
             image_height=-1,
         )
 
-        result = super().apply(prompt, mm_data, hf_processor_mm_kwargs,
-                               return_mm_hashes)
+        result = super().apply(prompt, mm_data, hf_processor_mm_kwargs)
path: vllm/model_executor/models/llava.py
user: login: DarkLight1337
body: Can you revert this change to `apply()`? `return_mm_hashes` is a new parameter
created_at: 2025-03-06T06:19:50Z
author_association: MEMBER
diff_hunk: @@ -778,8 +934,7 @@ def apply(
             image_height=-1,
         )
 
-        result = super().apply(prompt, mm_data, hf_processor_mm_kwargs,
-                               return_mm_hashes)
+        result = super().apply(prompt, mm_data, hf_processor_mm_kwargs)
path: vllm/model_executor/models/llava.py
user: login: lk-chen
body: Done.
created_at: 2025-03-06T06:49:39Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-05T10:19:45Z
body: 👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀
author_association: NONE
user: login: mergify[bot]
created_at: 2025-03-05T10:24:31Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lk-chen.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: DarkLight1337
created_at: 2025-03-05T10:24:38Z
body: I think you need to rebase your PR...
author_association: MEMBER
user: login: DarkLight1337
created_at: 2025-03-05T13:43:02Z
body: This seems to break V0 usage. When I run `python examples/offline_inference/vision_language_multi_image.py -m pixtral_hf`, I get this error:


```
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/cyrus/vllm/examples/offline_inference/vision_language_multi_image.py", line 552, in <module>
[rank0]:     main(args)
[rank0]:   File "/home/cyrus/vllm/examples/offline_inference/vision_language_multi_image.py", line 527, in main
[rank0]:     run_generate(model, QUESTION, IMAGE_URLS)
[rank0]:   File "/home/cyrus/vllm/examples/offline_inference/vision_language_multi_image.py", line 476, in run_generate
[rank0]:     outputs = req_data.llm.generate(
[rank0]:   File "/home/cyrus/vllm/vllm/utils.py", line 1084, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/home/cyrus/vllm/vllm/entrypoints/llm.py", line 472, in generate
[rank0]:     outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]:   File "/home/cyrus/vllm/vllm/entrypoints/llm.py", line 1379, in _run_engine
[rank0]:     step_outputs = self.llm_engine.step()
[rank0]:   File "/home/cyrus/vllm/vllm/engine/llm_engine.py", line 1401, in step
[rank0]:     outputs = self.model_executor.execute_model(
[rank0]:   File "/home/cyrus/vllm/vllm/executor/executor_base.py", line 284, in execute_model
[rank0]:     driver_outputs = self._driver_execute_model(execute_model_req)
[rank0]:   File "/home/cyrus/vllm/vllm/executor/mp_distributed_executor.py", line 144, in _driver_execute_model
[rank0]:     return self.driver_worker.execute_model(execute_model_req)
[rank0]:   File "/home/cyrus/vllm/vllm/worker/worker_base.py", line 420, in execute_model
[rank0]:     output = self.model_runner.execute_model(
[rank0]:   File "/home/cyrus/miniconda3/envs/vllm/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/cyrus/vllm/vllm/worker/model_runner.py", line 1742, in execute_model
[rank0]:     hidden_or_intermediate_states = model_executable(
[rank0]:   File "/home/cyrus/miniconda3/envs/vllm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/cyrus/miniconda3/envs/vllm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/cyrus/vllm/vllm/model_executor/models/llava.py", line 862, in forward
[rank0]:     vision_embeddings = self.get_multimodal_embeddings(**kwargs)
[rank0]:   File "/home/cyrus/vllm/vllm/model_executor/models/llava.py", line 787, in get_multimodal_embeddings
[rank0]:     nested_emb = [
[rank0]:   File "/home/cyrus/vllm/vllm/model_executor/models/llava.py", line 788, in <listcomp>
[rank0]:     self._get_mm_embeds(*args) for args in zip(
[rank0]:   File "/home/cyrus/vllm/vllm/model_executor/models/llava.py", line 756, in _get_mm_embeds
[rank0]:     feat_is_patch = feat_is_patch.view(-1)
[rank0]: AttributeError: 'list' object has no attribute 'view'
```
author_association: MEMBER
user: login: lk-chen
created_at: 2025-03-05T21:26:16Z
body: I used a brute force way to fix v0/v1, i.e. explicitly set "v0_path" as kwargs in v0 path.

Tested {v0, v1} X {vision_language.py, vision_language_multi_image.py} combinations. Please let me know if above fix is acceptable.
author_association: COLLABORATOR
user: login: DarkLight1337
created_at: 2025-03-06T06:21:08Z
body: I've tested the examples locally and they work on my end as well.
author_association: MEMBER
user: login: DarkLight1337
created_at: 2025-03-06T06:27:56Z
body: Can you update the Supported Models page to remove the note about Pixtral-HF not being supported in V1?
author_association: MEMBER
user: login: lk-chen
created_at: 2025-03-06T06:52:02Z
body: Thanks for testing my change! I've updated the markdown, PTAL
author_association: COLLABORATOR
user: login: mgoin
created_at: 2025-03-06T14:15:49Z
body: Have we run a full evaluation to make sure accuracy is good? I can do this today
author_association: MEMBER
user: login: DarkLight1337
created_at: 2025-03-06T14:24:05Z
body: Not yet, I only ran the example script.
author_association: MEMBER
user: login: mgoin
created_at: 2025-03-06T15:48:41Z
body: I ran chartqa with mistral-evals and it seems good

```
VLLM_USE_V1=1 vllm serve nm-testing/pixtral-12b-FP8-dynamic --max-model-len 30000 --port 9000
```

```
python -m eval.run eval_vllm --model_name nm-testing/pixtral-12b-FP8-dynamic --url http://0.0.0.0:9000 --output_dir output/ --eval_name "chartqa"
Waiting for VLLM server to come online at http://0.0.0.0:9000/health ...
Timeout is 120s
Waiting for server (0s) ...
Server is up!
Loading lmms-lab/ChartQA [test]: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:11<00:00, 212.12it/s]
Querying model: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [05:41<00:00,  7.33it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:00<00:00, 25420.89it/s]
================================================================================
Metrics:
{
    "explicit_prompt_relaxed_correctness": 0.8208,
    "anywhere_in_answer_relaxed_correctness": 0.8208
}
================================================================================
```
author_association: MEMBER