MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUs

## Abstract

Abstract Figure 1: Benchmark performance of MindVL and its counterparts.

## 1 Introduction

Multimodal Large Language Models (MLLMs) (Chen et al., 2023a; Zhang et al., 2024a) represent a significant advancement in artificial intelligence, demonstrating remarkable capabilities in understanding and generating content across vision and language modalities. Despite rapid progress, the field faces two major challenges that limit open research. First, the training of top-tier models (Chen et al., 2023b; Guo et al., 2025; Chen et al., 2024; Bai et al., 2025; Team et al., 2025) is predominantly dependent on a specific hardware ecosystem (e.g., NVIDIA GPUs), creating a perception that alternative platforms like Ascend are incapable of efficient full-stage MLLM training. This perception restricts hardware choices for researchers. Second, while the composition of training data is widely acknowledged as a critical factor for performance, most leading models (Bai et al., 2025; Guo et al., 2025) only offer high-level descriptions of their data; the exact recipes, cleaning pipelines, and mixing strategies are often treated as proprietary secrets. This lack of transparency severely impedes reproducibility and hinders community progress.

To address these challenges, this work advocates for a more open and efficient research paradigm for MLLMs. Our primary contribution is MindSpeed-MLLM, an optimized training framework that demonstrates the full capability of Ascend hardware for stable and efficient training of both dense and large-scale Mixture-of-Experts (MoE) models from pre-training to supervised fine-tuning (SFT). This provides researchers with a crucial and performant hardware alternative.

Second, we aim to demystify the ”black box” of leading MLLM data. We provide a comprehensive and open description of our data production methodology, including detailed data collection, cleaning, processing pipelines, and—most importantly—the mixing ratios used for each training stage. We believe this detailed data recipe offers a valuable blueprint for the community.

Third, we present MindVL, a data-efficient multimodal large language model trained end-to-end on Ascend NPUs. MindVL undergoes a three-phase training pipeline: warm-up, multitask training, and supervised instruction tuning, to incrementally enhance its multimodal capabilities. Starting with basic visual and cross-modal pre-training, the pipeline progresses to large-scale instruction adjustment, aligning the model with real-world use cases. Additionally, we integrate multimodal data packaging and hybrid parallelism techniques to significantly boost end-to-end training speed. To further optimize performance, we introduce two key strategies: test-time resolution search (to dynamically select optimal image resolutions for inference) and model weight averaging (to stabilize and improve final performance).

Extensive experiments validate the effectiveness of our approach. Our models achieve performance comparable to state-of-the-art models (e.g., the Qwen2.5VL series (Bai et al., 2025)) while utilizing orders of magnitude less training data. This result underscores the high quality of our data recipe and the robust training capability of the MindSpeed-MLLM framework on Ascend hardware. The contributions of this paper are as follows:

- •
We introduce MindSpeed-MLLM, a framework that enables efficient full-stage training of both Dense and MoE MLLMs on Ascend hardware, challenging existing perceptions.
- •
We provide a detailed and open data recipe for all training stages, promoting transparency and reproducibility in MLLM research.
- •
We introduce two enhancement techniques: multimodal model weight averaging and test-time resolution search, both of which contribute to improved pure text and multimodal performance.
- •
Experimental results show that MindVL achieves performance comparable to state-of-the-art models. Specifically, MindVL-8B matches the performance of Qwen2.5VL-7B using only 10% of its training data, while MindVL-671B-A37B, matches Qwen2.5VL-72B using only 3% of the Qwen2.5VL training data and and achieves comparable performance with other leading multimodal MoE models..

## 2 Related Work

### 2.1 Training of Multimodal Large Language Models

The training of Multimodal Large Language Models (MLLMs) faces significant challenges due to model heterogeneity (e.g., integrating vision encoders with LLMs) and data heterogeneity (e.g., processing images, videos, and text). These challenges necessitate specialized training frameworks and hardware optimizations to achieve efficiency and scalability. MLLM training relied on adapting text-centric frameworks like Megatron-LM (Shoeybi et al., 2019) and DeepSpeed (Rasley et al., 2020) to handle multimodal data by treating visual modules as additional layers within a unified parallelism strategy (e.g., combining Tensor, Pipeline, and Data Parallelism). Although numerous studies have successfully trained state-of-the-art MLLMs using these frameworks, their development has primarily focused on optimization for NVIDIA GPUs. In contrast, exploration of multimodal large-scale model training on Ascend NPUs remains limited, and a comprehensive, full-stage methodology for such environments has yet to be established.

### 2.2 Data Curation of Multimodal Large Language Models

Data curation is a cornerstone of multimodal large language model (MLLM) performance, as high-quality, well-structured multimodal data directly enables effective vision-language alignment and task adaptability. Existing literature and MLLM technical reports generally acknowledge the significance of data curation, outlining broad frameworks that typically categorize data by task and emphasizing core curation goals. Leading MLLMs, including Qwen2.5-VL (Bai et al., 2025) and Seed-VL 1.5 (Guo et al., 2025), provide only generalized descriptions of their data curation pipelines, lacking critical granular details that are essential for reproducibility and comparative analysis. This gap hinders the research community from fully dissecting how data curation choices impact MLLM capabilities and limits the development of future MLLMs.

## 3 MindSpeed-MLLM: Training Infrastructure on Ascend NPUs

Due to substantial hardware and software discrepancies, training frameworks widely used on NVIDIA GPUs—such as Megatron-LM (Shoeybi et al., 2019)—and common acceleration libraries (e.g., FlashAttention (Dao et al., 2022; Dao, 2024), Transformer-Engine (NVIDIA, 2023)) cannot be directly deployed on Ascend devices. (^1^11A detailed analysis of NVIDIA-Ascend hardware and software differences is provided in Appendix [A](https://arxiv.org/html/2509.11662v3#A1).)
Thus, developing a robust distributed training framework for the Ascend ecosystem is essential.
To this end, we introduce MindSpeed-MLLM: a distributed multimodal training library tailored for Ascend NPUs.

### 3.1 MindSpeed-MLLM

#### 3.1.1 MindSpeed Series Libraries

MindSpeed is a high-performance acceleration library tailored for the Ascend platform, encompassing three core components to support large-model training: MindSpeed-Core (Ascend, 2023), MindSpeed-LLM (Ascend, 2024a) MindSpeed-MM (Ascend, 2024b) and MindSpeed-RL (Feng et al., 2025).

MindSpeed-Core, built on and optimized for Ascend hardware based on Megatron-LM, delivers multi-dimensional optimizations in computing, memory, communication, and parallelism, enabling accelerated training for scenarios like long sequences and MoE.
MindSpeed-LLM provides a rich set of LLM with extensive training optimization features, while MindSpeed-MM realizes mainstream Vision-Language Models.

Despite their individual strengths, the existing components present integration and functionality gaps for end-to-end multi-modal large language model (MLLM) training.
Specifically, MindSpeed-MM lacks robust support for critical multi-modal data processing functionalities, including distributed data loading, data packing, and training resumption.
Furthermore, its optimizations for the language backbone are not as comprehensive or mature as those provided by MindSpeed-LLM.
To address these gaps and leverage the strengths of existing components, MindSpeed-MLLM is developed with targeted optimizations.

#### 3.1.2 MindSpeed-MLLM Framework

Figure: Figure 2: The Overall Architecture of MindSpeed-MLLM and Its Relationship with Other MindSpeed Frameworks.
Refer to caption: figures/infra/mllm_figure2.jpg

As depicted in Figure [2](https://arxiv.org/html/2509.11662v3#S3.F2), the MindSpeed-MLLM framework is constructed with a hierarchical architecture. It builds upon the foundational optimizations from MindSpeed-Core and integrates partial modules from MindSpeed-LLM and MindSpeed-MM.
Beyond this integration, the core efforts of MindSpeed-MLLM lie in enhancing multi-modal data processing, operator fusion replacement, and system-level scheduling optimization, which will be introduced separately in the following sections.

#### 3.1.3 Multi-Modal Data Loader

We have developed a multi-modal data loader with the following features:

Distributed Multi-Modal Data Loader:
It supports distributed data loading, where each data parallel group only reads the data within that group, effectively avoiding bottlenecks caused by reading the same data during large-scale training redundantly.

Online Packing:
It enables online packing of multi-modal data, which combines data of different lengths into a specified length and fills in valid data content as much as possible Ma et al. (2025); Wang et al. (2025a).
Thus, each Pack dataset has almost the same length, reducing the number of samples during training and improving training efficiency.
Meanwhile, it controls the number of visual tokens to avoid load imbalance between pipeline parallel stages caused by uneven quantities of different modal data.

Consumed Data Tracker:
It supports tracking of consumed data, which facilitates the location of data breakpoints during checkpoint-based recovery training, eliminates redundant data retraining, and ensures accurate resumption of training tasks.

#### 3.1.4 Operator Fusion Replacement

Choosing hardware-friendly operators effectively boosts training efficiency in model development.
MindSpeed already supports fusion operator replacements, such as RMSNorm, SoftMax, MoE Token Permute, Unpermute, and Adamw, etc.
Beyond these pre-provided fusion operators, MindSpeed-MLLM further optimizes as follows:

Common Attention Patch:
MindSpeed-LLM and MindSpeed-MM replaced attention operators for language and visual components respectively, swapping the flash attention interface with npu fusion attention.
Since both components require this operator, we patched attention in the shared transformer block to a unified fusion operator call.
Different mask types are passed to specific components depending on the module ids.

Mask Compression:
When training native resolutions ViTs, we first concatenate the hidden states of distinct images along the sequence dimension, followed by invoking the attention operator through a variable-length way.
This design simultaneously reduces memory overhead and enhances computational efficiency.
Combined with the sparse mode parameter of the npu fusion attention operator, passing the compressed attention mask further reduces the memory usage.

Operation Replacement:
Profiling showed low computational efficiency of Conv operators in CANN 8.0.
Thus, we replaced Conv2d and Conv3d operations with Matmul operation equivalently.
The supporting checkpoint convert toolkit also added corresponding support for this replacement.

#### 3.1.5 System-Level Scheduling Optimization

Within the Ascend ecosystem, system-level scheduling optimizations deliver performance enhancements through multiple mechanisms.
Fine-grained core binding minimizes cross-NUMA node memory accesses, reducing both task scheduling overhead and inter-core switching costs.
Concurrently, operator deployment queue optimizations partition deployment tasks across multi-stage pipelines operating in parallel. This approach enables partial overlap between execution and submission processes, reducing overall latency and improving end-to-end performance.

## 4 Data Curation

The MindVL training corpus contains 447 billion diverse and high-quality tokens used for three training stages. The data is categorized according to target capabilities, and the curation process for each category is detailed in the following subsections. Due to space limitations, more detailed processing steps and data ratios are provided in the Appendix [B](https://arxiv.org/html/2509.11662v3#A2).

As shown in Figure [3](https://arxiv.org/html/2509.11662v3#S4.F3), the training data of MindVL is open-sourced into two main categories: image-text pairs and visual instructions. The image-text pair data is further divided into eight subcategories. Figure [3](https://arxiv.org/html/2509.11662v3#S4.F3) illustrates the core processing methods for each category, along with fundamental filtering techniques and some models used during data annotation.

Figure: Figure 3: Data curation process of MindVL training data.
Refer to caption: x2.png

### 4.1 Warm-up Data

The MindVL warm-up corpus consists of 256 billion diverse, high-quality tokens. Warm-up data span six categories, including image caption, OCR, Visual Grounding, Table&Chart, GUI, STEM, with key processing steps described in Appendix [B.1](https://arxiv.org/html/2509.11662v3#A2.SS1).

### 4.2 Multitask Training Data

Multitask training data consists of interleaved image-text (Table [12](https://arxiv.org/html/2509.11662v3#A2.T12)), visual instruction (Table LABEL:tab:multitask, about 80B tokens), web2code (Yun et al., 2024) and text instruction (Table [13](https://arxiv.org/html/2509.11662v3#A2.T13)), totally 179B tokens. We add textual data to maintain the MindVL’s linguistic capabilities. Details are described in Appendix [B.2](https://arxiv.org/html/2509.11662v3#A2.SS2)

### 4.3 Supervised Fine-tuning Data

High-quality instruction data is sampled from open-source multitask datasets (classified by model), with low-quality answers re-annotated (via model and human verification), totally 12B tokens. Textual data is incorporated at a multimodal-to-language ratio of 1:1 to preserve the MLLM’s linguistic performance. Details are described in Appendix [B.3](https://arxiv.org/html/2509.11662v3#A2.SS3)

## 5 MindVL

### 5.1 Architecture

The architecture of MindVL bears resemblance to that of Qwen2.5-VL, comprising three core components: a vision encoder, an MLP projector, and a large language model. The vision encoder natively supports dynamic image resolutions and adopts 2D RoPE (Wang et al., 2024b) for positional encoding, enabling flexible adaptation to images of arbitrary dimensions.

For model initialization, we utilize Qwen2.5ViT (^2^22The weights of Qwen2.5ViT are derived from the Qwen2.5-VL 72B model.) as visual encoder and Qwen3 LLM (Yang et al., 2025) / DeepSeek-V3-0324 (Liu et al., 2024a) as language backbones, choices that ensure robust baseline performance. In contrast, the MLP projector is initialized randomly.

### 5.2 Training Recipe

As shown in Table [1](https://arxiv.org/html/2509.11662v3#S5.T1), the training process of MindVL is divided into three distinct phases, each employing different data
configurations and training strategies to progressively enhance the model’s capabilities.

**Table 1: Training setup and hyper-parameters in three training stages.**
| Stages | Warm-up<br>8B / 671B | Multitask training<br>8B / 671B | SFT<br>8B / 671B |
| --- | --- | --- | --- |
| Training budget (tokens) | 256B/16B | 179B/80B | 12B |
| Sequence length | 8192 | 8192 | 2048/4096/8192 |
| Trainable components | MLP adaptor | All | All |
| Batch sizes | 1024 | 1024 | 512 |
| LR warm-up ratio | 0.1 | 0.1 | 0.1 |
| Maximum LR | 1e-3/2e-4 | 2e-5/5e-6 | 1e-5/2e-6 |
| Minimum LR | 0/1e-5 | 1e-5/5e-7 | 0/2e-7 |

During the warm-up phase, only the MLP adapter is trained to align the vision transformer and the language model. This process relies on carefully curated data spanning image captions, visual grounding, OCR, and STEM content to establish foundational multimodal alignment.

In the multitask training phase, all parameters are optimized with diverse multimodal data such as interleaved image-text, VQA, math reasoning, agent tasks, video, and pure text. This enhances complex visual-language reasoning while preserving linguistic abilities.

Finally, supervised fine-tuning unlocks all parameters and focuses on instruction-aware optimization to adapt pretrained representations to downstream tasks.

### 5.3 Model Merging with Different Training Sequence Lengths

Merging weights of deep models (Li et al., 2023) has been verified to be an efficient way in various applications, including multi-task learning, federated learning, model compression, and continual learning.
We also explore the weight averaging strategies with the models trained under different settings of input sequence length and image resolution.
Specifically, for the SFT phase, we train the model with sequence length of 2K, 4K and 8K with max_pixels of 1280*28*28, 3072*28*28 and 4096*28*28 respectively to enhance the model’s adaptability to different context inputs and image resolutions.
Finally, the MindVL model is created by averaging weights of models with different training sequence length.

### 5.4 Test-Time Resolution Search

The evaluation model is merged by model weights trained with different sequence lengths as well as different max_pixels threshold of training images.
Such search strategy is generally effective as the original resolution of the test images maybe be out of distribution relative to that of the training images.
Therefore, we conduct a grid search about up-scaling small images to surpass a specified min_pixels threshold and down-scaling high-resolution image to be lower than a specified max_pixels threshold.
Specifically, we set the grid search space of the min_pixels as {4, 16, 32, 64}*28*28 and max_pixels as {1280, 2048, 2560,3072,4096,8192}*28 *28.
The analysis results are presented in Section [6.3](https://arxiv.org/html/2509.11662v3#S6.SS3).

## 6 Experiments

### 6.1 Performance of MindVL-8B

As shown in Table [2](https://arxiv.org/html/2509.11662v3#S6.T2), we evaluate the overall performance of MindVL-8B on MMBench (Liu et al., 2024c), MME (Chaoyou et al., 2023), OCRBench (Liu et al., 2024d), DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), InfoVQA (Mathew et al., 2022). Overall, MindVL-8B outperforms several leading models, including Qwen2.5-VL-7B, GLM-4.1V-9B, Keya-VL-8B, and InternVL3.5-8B. Notably, MindVL-8B achieves this superior performance using only 447B tokens of training data—roughly one-tenth of the data used by Qwen2.5-VL-7B. Furthermore, compared to models trained with trillion-scale tokens such as GLM-4.1V-9B and Keye-VL-8B, MindVL delivers significantly stronger results, outperforming them by margins of 6.6 and 5.8 points, respectively. When compared to InternVL3.5-8B, which has a similar pre-training scale, MindVL-8B maintains a lead of 2.7 points.
These outcomes highlight the effectiveness of our training methodology and data efficiency, demonstrating the capability to develop high-performing multimodal large language models on Ascend NPUs.

**Table 2: Performance of MindVL-8B and comparison models on multimodal benchmarks. ”+” indicates that there is a portion of data with unlabeled quantities. The results of the comparative models are referenced from (Wang et al., 2025b).**
| Model | #Tokens | MME | MMBench | OCRBench | DocVQA | ChartQA | InfoVQA | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2.5-VL-7B (Bai et al., 2025) | 4.1T+ | 83.8 | 82.6 | 86.4 | 95.7 | 87.3 | 82.6 | 86.4 |
| Keye-VL-8B (Team, 2025) | 1T+ | 87.3 | 85.8 | 82.3 | 87.0 | 72.5 | 63.0 | 79.9 |
| GLM-4.1V-9B (Team et al., 2025) | 2T+ | 79.1 | 76.3 | 85.3 | 93.3 | 70.0 | 80.3 | 80.7 |
| InternVL3.5-8B (Wang et al., 2025b) | 380B+ | 85.0 | 79.5 | 83.2 | 92.3 | 86.7 | 76.2 | 83.8 |
| MindVL-8B | 447B | 84.1 | 84.3 | 87.6 | 94.7 | 87.2 | 81.1 | 86.5 |

### 6.2 Performance of MindVL-671B-A37B

**Table 3: Comparison of OCR, chart, document and general understanding performance.**
| Model | #Tokens | MME | MMBench | OCR<br>Bench | DocVQA | ChartQA | InfoVQA | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4V (OpenAI, 2023) | – | 68.8 | 80.0 | 64.5 | 88.4 | 78.5 | 75.1 | 70.0 |
| GPT-4o-20240513 (Hurst et al., 2024) | – | – | 83.1 | 73.6 | 92.8 | 85.7 | 79.2 | – |
| Claude-3-Opus (Anthropic, 2024a) | – | 56.7 | 60.1 | 69.4 | 89.3 | 80.8 | 55.6 | 67.3 |
| Claude-3.5-Sonnet (Anthropic, 2024b) | – | – | 80.9 | 78.8 | 95.2 | 90.8 | 74.3 | – |
| Gemini-1.5-Pro (Team et al., 2024) | – | – | 74.6 | 75.4 | 93.1 | 87.2 | 81.0 | – |
| Step3V (StepFun, 2025) | – | – | 81.1 | 83.7 | – | – | – | – |
| GLM-4.5V (Team et al., 2025) | 2T+ | – | 86.7 | 87.2 | 94.5 | 86.6 | 84.1 | 75.8 |
| Qwen2-VL-72B (Wang et al., 2024b) | 1.4T+ | 88.7 | 85.9 | 87.7 | 96.5 | 88.3 | 84.5 | 88.6 |
| Qwen2.5-VL-72B (Bai et al., 2025) | 4.1T+ | 87.4 | 88.4 | 88.5 | 96.4 | 89.5 | 87.3 | 89.6 |
| InternVL3-78B (Zhu et al., 2025) | 200B+ | 91.1 | 87.7 | 90.6 | 95.4 | 89.7 | 86.5 | 90.2 |
| InternVL3.5-241B-A28B (Wang et al., 2025b) | 380B+ | – | 87.4 | 90.7 | 94.9 | 88.0 | 82.0 | – |
| MindVL-671B-A37B | 106B | 91.1 | 90.8 | 90.0 | 96.0 | 89.0 | 88.9 | 91.0 |

**Table 4: Comparison of multi-modal Hallusion/STEM performance.**
| Model | #Tokens | Hallusion<br>Bench | AI2D | MMVet | MathVista | MathVision | MMMU pro | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GLM-4.5V (Team et al., 2025) | 2T+ | 65.4 | 86.6 | 75.2 | 78.2 | 52.5 | 59.8 | 69.6 |
| Qwen2-VL-72B (Wang et al., 2024b) | 1.4T+ | 58.1 | 88.1 | 74.0 | 70.5 | 25.9 | 46.2 | 60.5 |
| Qwen2.5-VL-72B (Bai et al., 2025) | 4.1T+ | 55.2 | 88.7 | 76.2 | 74.2 | 38.1 | 51.1 | 63.9 |
| InternVL3-78B (Zhu et al., 2025) | 200B+ | 59.1 | 89.7 | 81.3 | 79.0 | 43.1 | – | – |
| InternVL3.5-241B-A28B (Wang et al., 2025b) | 380B+ | 57.3 | 87.3 | 81.2 | 82.7 | 63.9 | – | – |
| MindVL-671B-A37B | 106B | 68.6 | 85.2 | 67.9 | 72.9 | 34.1 | 49.5 | 63.0 |

**Table 5: Comparison of model performance across diverse language benchmarks. All results are uniformly evaluated by the internal testing platform.**
| Model | AIME2024 | AIME2025 | GPQA-D | IFEval | ArenaHard | C-SimpleQA | C-Eval | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2.5VL-32B (Bai et al., 2025) | 30.0 | 16.7 | 51.5 | 64.3 | 92.2 | 44.9 | 81.3 | 54.1 |
| Qwen2.5VL-72B (Bai et al., 2025) | 26.7 | 16.7 | 50.0 | 85.6 | 71.2 | 49.8 | 86.1 | 55.2 |
| DeepSeek-V3-0324 (Liu et al., 2024a) | 60.0 | 43.3 | 69.2 | 81.9 | 94.8 | 73.3 | 89.6 | 73.2 |
| MindVL-671B-A37B | 63.3 | 46.7 | 68.7 | 84.7 | 97.0 | 72.9 | 88.2 | 74.5 |

As shown in Table [3](https://arxiv.org/html/2509.11662v3#S6.T3),
we compare our MindVL-671B-A37B with open source models of approximate parameter size and closed source excellent models on OCR, chart, and document understanding benchmarks.
Overall, our MindVL-671B-A37B, trained with 106B multimodal tokens, achieves best average score across above benchmarks.
These results show that our data and training recipes are effective in Vision-Language Alignment for large language models with different parameter scales, i.e., Qwen3-8B and DeepSeek-V3.

In addition, the comparison results of MindVL-671B-A37B and other models on multi-modal Hallusion and STEM benchmarks, including HallusionBench (Guan et al., 2024), AI2D (Kembhavi et al., 2016), MMVet (Yu et al., 2023), MathVista (Lu et al., 2023), MathVision (Wang et al., 2024a) and MMMU pro (Yue et al., 2024), are shown in Table [4](https://arxiv.org/html/2509.11662v3#S6.T4).
MindVL-671B-A37B gets the best evaluation results on HallusionBench.
Although MindVL-671B-A37B achieves competitive performance against Qwen2-VL-72B and Qwen2.5-VL-72B on overall score,
some of the model’s multimodal reasoning capabilities are not sufficiently activated.
For example, there is still a certain gap between our model and the state-of-the-art models on MMVet, MathVision and MMMU pro.
The reason is that MindVL-671B-A37B is trained with only 106B multimodal tokens.
In the future, we will add rich interleaved image-text data to boost model reasoning ability with broad domain knowledge.

Moreover, as shown in Table [5](https://arxiv.org/html/2509.11662v3#S6.T5), we report the pure text evaluation results of MindVL-671B-A37B cross diverse language benchmarks, including AIME2024 (MAA, 2024), AIME2025 (MAA, 2025), GPQA-D (Rein et al., 2024), IFEval (Zhou et al., 2023), ArenaHard (Li et al., 2024), C-SimpleQA (He et al., 2024) and C-Eval (Huang et al., 2023).
MindVL-671B-A37B outperforms the original DeepSeek-V3 in Overall evaluation result.
Specifically, MindVL-671B-A37B achieves significant improvements on AIME2024, AIME2025, IFEval and AreanaHard datasets, and maintained the pure text capabilities of DeepSeek-V3 on GPQA-D, C-SimpleQA and C-Eval datasets.
In addition, MindVL-671B-A37B also obtains better evaluation results than Qwen2.5VL-32B and very competitive results compared to Qwen2.5VL-72B across these pure text benchmarks.

**Table 6: Studies of model merging strategy of MindVL-8B on multimodal benchmarks.**
| Model | Sequence<br>Length | Maximum<br>Pixels | MME | MMBench | OCRBench | DocVQA | ChartQA | InfoVQA | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MindVL-8B-2K | 2K | 1280 | 81.6 | 82.4 | 87.0 | 94.2 | 86.8 | 79.6 | 85.3 |
| MindVL-8B-4K | 4K | 3072 | 84.3 | 82.3 | 87.3 | 94.5 | 86.6 | 79.8 | 85.8 |
| MindVL-8B-8K | 8K | 4096 | 82.5 | 82.5 | 86.0 | 94.6 | 86.4 | 79.9 | 85.3 |
| MindVL-8B | - | - | 84.1 | 84.3 | 87.6 | 94.7 | 87.2 | 81.1 | 86.5 |

Figure: Figure 4: Box plots of accuracies with varying input images resolutions for different models. Maximum value, minimum value, median and quartiles are plotted.
Refer to caption: x3.png

### 6.3 Effectiveness of Model Merging with Different Training Sequence Lengths

Table [6](https://arxiv.org/html/2509.11662v3#S6.T6) validate the effectiveness of our model merging strategy, which is implemented by averaging the weights of models trained on sequence lengths of 2K, 4K, and 8K—denoted as MindVL-8B-2K, MindVL-8B-4K, and MindVL-8B-8K, respectively. The merged model is denoted as MindVL-8B. Our MindVL-8B model achieves an average benchmark performance of 86.5%, which are better than the scores of MindVL-8B-2K, MindVL-8B-4K, and MindVL-8B-8K. The proposed strategy significantly surpasses the perceptual capabilities of MindVL-8B, especially on MMBench, ChartQA and InfoVQA datasets.

Figures [4](https://arxiv.org/html/2509.11662v3#S6.F4) shows the box plots of the evaluation results using test-time image resolution search strategy as mentioned in Section [5.4](https://arxiv.org/html/2509.11662v3#S5.SS4).
On OCRBench, DocVQA and InfoVQA, MindVL-8B outperforms other models not only in terms of the highest accuracy scores across different image resolutions, but also shows improvements in both the lowest scores and median scores. On MME dataset, MindVL-8B outerperforms MindVL-8B-2K and MindVL-8B-8K.
The above results demonstrate that the model weight merging strategy can enhance the model’s robustness against variations in input images of different resolutions.
For specific numerical results, please refer to Appendix [C.1](https://arxiv.org/html/2509.11662v3#A3.SS1).

**Table 7: Studies of model merging strategy of MindVL-671B-A37B on language benchmarks.**
| Model | AIME2024 | AIME2025 | GPQA-D | IFEval | ArenaHard | C-SimpleQA | C-Eval | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepSeek-V3-0324 (Liu et al., 2024a) | 60.0 | 43.3 | 69.2 | 81.9 | 94.8 | 73.3 | 89.6 | 73.2 |
| MindVL-671B-A37B-4K | 20.0 | 16.7 | 58.6 | 81.5 | 91.0 | 70.8 | 84.4 | 60.4 |
| MindVL-671B-A37B | 63.3 | 46.7 | 68.7 | 84.7 | 97.0 | 72.9 | 88.2 | 74.5 |

In addition, we verified the effectiveness of the model merging strategy for large-scale language model, i.e. DeepSeek-V3, in maintaining the pure text ability after Vision-Language alignment. As shown in Table [7](https://arxiv.org/html/2509.11662v3#S6.T7),
MindVL-671B-A37B is obtained by merging the language models of DeepSeek-V3 and MindVL-671B-A37B-4K, i.e., the SFT model trained with 4K sequence length.
MindVL-671B-A37B not only outperforms MindVL-671B-A37B-4K on each benchmark, but also outperforms the original language model DeepSeek-V3 on AIME2024, AIME2025, IFEval, ArenaHard and overall average score.
For multimodal performance, MindVL-671B-A37B achieves the evaluation results similar to MindVL-671B-A37B-4K, as shown in Table [8](https://arxiv.org/html/2509.11662v3#S6.T8).

**Table 8: Studies of model merging strategy of MindVL-671B-A37B on multimodal benchmarks. All results are uniformly evaluated by the internal testing platform (Correctness Evaluation).**
| Model | MM-<br>Bench | OCR-<br>Bench | DocVQA<br>Val | ChartQA | InfoVQA<br>Val | AI2D | MMVet | Math-<br>Vista | Math-<br>Vision | MMMU<br>pro | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MindVL-671B-A37B-4K | 90.8 | 90.0 | 96.0 | 79.6 | 83.0 | 85.2 | 67.9 | 72.9 | 33.5 | 45.0 | 74.4 |
| MindVL-671B-A37B | 90.7 | 89.4 | 95.8 | 78.7 | 84.2 | 83.2 | 70.2 | 70.7 | 43.0 | 49.1 | 75.5 |

## 7 Conclusion

In this paper, our proposed MindVL, trained on MindSpeed-MLLM framework with Ascend NPUs. We highlight the effectiveness of our data recipe and the robust training capability of the framework on Ascend hardware. Key contributions include the introduction of an efficient full-stage training framework for both Dense and MoE MLLMs, a transparent data recipe, and novel enhancement techniques. Notably, MindVL-8B and MindVL-671B-A37B achieve competitive results on both multimodal benchmark and language benchmark using only a fraction of the training data required by comparable models. Moreover, our proposed model merging method has effectively enhanced the model’s performance. We find that model merging is an effective and low-cost approach to improve model performance. In future work, we will further explore the methodologies of model merging and the underlying principles behind it.

## 8 Reproducibility Statement

We have provided a detailed description in the main text and appendix of the training framework MindSpeed-MLLM (Section [3](https://arxiv.org/html/2509.11662v3#S3) and Appendix [A](https://arxiv.org/html/2509.11662v3#A1)) used by MindVL, the data and data ratios for each stage (Section [4](https://arxiv.org/html/2509.11662v3#S4) and Appendix [B](https://arxiv.org/html/2509.11662v3#A2)), and the training hyper-parameters of the model at each stage (Section [5](https://arxiv.org/html/2509.11662v3#S5)), all to facilitate researchers in referencing and reproducing our work. Furthermore, following internal review, we will make the MindSpeed-MLLM code open-source to advance research on multimodal large language models on Ascend NPUs.

## References

- Anthropic (2024a)
Anthropic.
Claude 3 sonnet.
2024a.
[https://www.anthropic.com/news/claude-3-sonnet](https://www.anthropic.com/news/claude-3-sonnet).
- Anthropic (2024b)
Anthropic.
Claude 3.5 sonnet.
2024b.
[https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet).
- Ascend (2023)
Ascend.
Mindspeed-core.
2023.
[https://gitcode.com/Ascend/MindSpeed](https://gitcode.com/Ascend/MindSpeed).
- Ascend (2024a)
Ascend.
Mindspeed-llm.
2024a.
[https://gitcode.com/ascend/MindSpeed-LLM](https://gitcode.com/ascend/MindSpeed-LLM).
- Ascend (2024b)
Ascend.
Mindspeed-mm.
2024b.
[https://gitcode.com/ascend/MindSpeed-MM](https://gitcode.com/ascend/MindSpeed-MM).
- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Chaoyou et al. (2023)
Fu Chaoyou, Chen Peixian, Shen Yunhang, Qin Yulei, Zhang Mengdan, Lin Xu, Yang Jinrui, Zheng Xiawu, Li Ke, Sun Xing, et al.
Mme: A comprehensive evaluation benchmark for multimodal large language models.
*arXiv preprint arXiv:2306.13394*, 3, 2023.
- Chen et al. (2023a)
Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu.
Vlp: A survey on vision-language pre-training.
*Machine Intelligence Research*, 20(1):38–56, 2023a.
- Chen et al. (2023b)
Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu.
X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages.
*arXiv preprint arXiv:2305.04160*, 2023b.
- Chen et al. (2024)
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.
*arXiv preprint arXiv:2412.05271*, 2024.
- Cloud (Year)
Alibaba Cloud.
Pai-megatron-patch.
https://github.com/alibaba/Pai-Megatron-Patch, Year.
Accessed: 2023 - 10 - 08.
- Cui et al. (2025)
Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, et al.
Paddleocr 3.0 technical report.
*arXiv preprint arXiv:2507.05595*, 2025.
- Dao (2024)
Tri Dao.
Flashattention-2: Faster attention with better parallelism and work partitioning.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=mZn2Xyh9Ec](https://openreview.net/forum?id=mZn2Xyh9Ec).
- Dao et al. (2022)
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.
Flashattention: Fast and memory-efficient exact attention with io-awareness.
In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), *Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022*, 2022.
URL [http://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html).
- Feng et al. (2025)
Laingjun Feng, Chenyi Pan, Xinjie Guo, Fei Mei, Benzhe Ning, Jianxiang Zhang, Xinyang Liu, Beirong Zhou, Zeng Shu, Chang Liu, Guang Yang, Zhenyu Han, Jiangben Wang, and Bo Wang.
Mindspeed rl: Distributed dataflow for scalable and efficient rl training on ascend npu cluster, 2025.
URL [https://arxiv.org/abs/2507.19017](https://arxiv.org/abs/2507.19017).
- GLM et al. (2024)
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang.
Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024.
- Guan et al. (2024)
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al.
Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 14375–14385, 2024.
- Guo et al. (2025)
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al.
Seed1. 5-vl technical report.
*arXiv preprint arXiv:2505.07062*, 2025.
- He et al. (2024)
Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al.
Chinese simpleqa: A chinese factuality evaluation for large language models.
*arXiv preprint arXiv:2411.07140*, 2024.
- Huang et al. (2023)
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al.
C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.
*Advances in Neural Information Processing Systems*, 36:62991–63010, 2023.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Kembhavi et al. (2016)
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi.
A diagram is worth a dozen images.
In *European conference on computer vision*, pp. 235–251. Springer, 2016.
- Kirillov et al. (2023)
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.
Segment anything.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 4015–4026, 2023.
- Li et al. (2024)
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica.
From live data to high-quality benchmarks: The arena-hard pipeline.
*Blog post.[Accessed 07-02-2025]*, 2024.
- Li et al. (2023)
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen.
Deep model fusion: A survey.
*arXiv preprint arXiv:2309.15698*, 2023.
- Lin et al. (2014)
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.
Microsoft COCO: common objects in context.
In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), *Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V*, volume 8693 of *Lecture Notes in Computer Science*, pp. 740–755. Springer, 2014.
doi: 10.1007/978-3-319-10602-1“˙48.
URL [https://doi.org/10.1007/978-3-319-10602-1_48](https://doi.org/10.1007/978-3-319-10602-1_48).
- Liu et al. (2024a)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024a.
- Liu et al. (2024b)
Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang.
Focus anywhere for fine-grained multi-page document understanding.
*arXiv preprint arXiv:2405.14295*, 2024b.
- Liu et al. (2023)
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu.
Mmc: Advancing multimodal chart understanding with large-scale instruction tuning.
*arXiv preprint arXiv:2311.10774*, 2023.
- Liu et al. (2024c)
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.
Mmbench: Is your multi-modal model an all-around player?
In *European conference on computer vision*, pp. 216–233. Springer, 2024c.
- Liu et al. (2024d)
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai.
Ocrbench: on the hidden mystery of ocr in large multimodal models.
*Science China Information Sciences*, 67(12):220102, 2024d.
- Long et al. (2025)
Rujiao Long, Hangdi Xing, Zhibo Yang, Qi Zheng, Zhi Yu, Fei Huang, and Cong Yao.
Lore++: Logical location regression network for table structure recognition with pre-training.
*Pattern Recognition*, 157:110816, 2025.
- Lu et al. (2023)
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
*arXiv preprint arXiv:2310.02255*, 2023.
- Ma et al. (2025)
Dongyang Ma, Yan Wang, and Tian Lan.
Block-attention for efficient prefilling.
In *The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025*. OpenReview.net, 2025.
URL [https://openreview.net/forum?id=7zNYY1E2fq](https://openreview.net/forum?id=7zNYY1E2fq).
- MAA (2024)
MAA.
American invitational mathematics examination - aime.
2024.
[https://maa.org/maa-invitational-competitions](https://maa.org/maa-invitational-competitions).
- MAA (2025)
MAA.
American invitational mathematics examination - aime.
2025.
[https://maa.org/maa-invitational-competitions](https://maa.org/maa-invitational-competitions).
- Masry et al. (2022)
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.
Chartqa: A benchmark for question answering about charts with visual and logical reasoning.
*arXiv preprint arXiv:2203.10244*, 2022.
- Mathew et al. (2021)
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.
Docvqa: A dataset for vqa on document images.
In *Proceedings of the IEEE/CVF winter conference on applications of computer vision*, pp. 2200–2209, 2021.
- Mathew et al. (2022)
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar.
Infographicvqa.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pp. 1697–1706, 2022.
- Meng et al. (2024)
Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo.
Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning.
*arXiv preprint arXiv:2401.02384*, 2024.
- NVIDIA (2023)
NVIDIA.
TransformerEngine.
2023.
[https://github.com/NVIDIA/TransformerEngine](https://github.com/NVIDIA/TransformerEngine).
- OpenAI (2023)
OpenAI.
Gpt-4v(ision) system card.
2023.
[https://cdn.openai.com/papers/GPTV_System_Card.pdf](https://cdn.openai.com/papers/GPTV_System_Card.pdf).
- Peng et al. (2023)
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.
Kosmos-2: Grounding multimodal large language models to the world.
*ArXiv*, abs/2306.14824, 2023.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pp. 8748–8763. PmLR, 2021.
- Rasley et al. (2020)
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.
Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.
In *Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining*, pp. 3505–3506, 2020.
- Rein et al. (2024)
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman.
Gpqa: A graduate-level google-proof q&a benchmark.
In *First Conference on Language Modeling*, 2024.
- Shi et al. (2024)
Bowen Shi, Peisen Zhao, Zichen Wang, Yuhang Zhang, Yaoming Wang, Jin Li, Wenrui Dai, Junni Zou, Hongkai Xiong, Qi Tian, et al.
Umg-clip: A unified multi-granularity vision generalist for open-world understanding.
In *European Conference on Computer Vision*, pp. 259–277. Springer, 2024.
- Shoeybi et al. (2019)
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.
Megatron-lm: Training multi-billion parameter language models using model parallelism.
*arXiv preprint arXiv:1909.08053*, 2019.
- Smock et al. (2022)
Brandon Smock, Rohith Pesala, and Robin Abraham.
Pubtables-1m: Towards comprehensive table extraction from unstructured documents.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 4634–4642, 2022.
- StepFun (2025)
StepFun.
Step3 v.
2025.
[https://stepfun.ai/research/zh/step3](https://stepfun.ai/research/zh/step3).
- Team et al. (2024)
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
*arXiv preprint arXiv:2403.05530*, 2024.
- Team (2025)
Kwai Keye Team.
Kwai keye-vl technical report, 2025.
URL [https://arxiv.org/abs/2507.01949](https://arxiv.org/abs/2507.01949).
- (53)
RapidTable Team.
RapidTable.
[https://github.com/RapidAI/RapidTable](https://github.com/RapidAI/RapidTable).
- Team et al. (2025)
V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li,
Minlie Huang, Yuxiao Dong, and Jie Tang.
Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025.
URL [https://arxiv.org/abs/2507.01006](https://arxiv.org/abs/2507.01006).
- Wang et al. (2024a)
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li.
Measuring multimodal mathematical reasoning with math-vision dataset.
In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024a.
URL [https://openreview.net/forum?id=QWTCcxMpPA](https://openreview.net/forum?id=QWTCcxMpPA).
- Wang et al. (2024b)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024b.
- Wang et al. (2025a)
Shuhe Wang, Guoyin Wang, Yizhong Wang, Jiwei Li, Eduard H. Hovy, and Chen Guo.
Packing analysis: Packing is more appropriate for large models or datasets in supervised fine-tuning.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025*, pp. 4953–4967. Association for Computational Linguistics, 2025a.
URL [https://aclanthology.org/2025.findings-acl.256/](https://aclanthology.org/2025.findings-acl.256/).
- Wang et al. (2025b)
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al.
Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.
*arXiv preprint arXiv:2508.18265*, 2025b.
- Xiao et al. (2024)
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan.
Florence-2: Advancing a unified representation for a variety of vision tasks.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 4818–4829, 2024.
- Xing et al. (2023)
Hangdi Xing, Feiyu Gao, Rujiao Long, Jiajun Bu, Qi Zheng, Liangcheng Li, Cong Yao, and Zhi Yu.
Lore: Logical location regression network for table structure recognition.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, pp. 2992–3000, 2023.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yu et al. (2023)
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.
Mm-vet: Evaluating large multimodal models for integrated capabilities.
*arXiv preprint arXiv:2308.02490*, 2023.
- Yue et al. (2024)
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 9556–9567, 2024.
- Yun et al. (2024)
Sukmin Yun, Rusiru Thushara, Mohammad Bhat, Yongxin Wang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, et al.
Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms.
*Advances in neural information processing systems*, 37:112134–112157, 2024.
- Zhang et al. (2024a)
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu.
Mm-llms: Recent advances in multimodal large language models.
*arXiv preprint arXiv:2401.13601*, 2024a.
- Zhang et al. (2024b)
Jiwen Zhang, Yaqi Yu, Minghui Liao, Wentao Li, Jihao Wu, and Zhongyu Wei.
Ui-hawk: Unleashing the screen stream understanding for gui agents.
2024b.
- Zheng et al. (2024)
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma.
Llamafactory: Unified efficient fine-tuning of 100+ language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, Bangkok, Thailand, 2024. Association for Computational Linguistics.
URL [http://arxiv.org/abs/2403.13372](http://arxiv.org/abs/2403.13372).
- Zhou et al. (2023)
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou.
Instruction-following evaluation for large language models.
*arXiv preprint arXiv:2311.07911*, 2023.
- Zhu et al. (2025)
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.
*arXiv preprint arXiv:2504.10479*, 2025.

## Appendix A MindSpeed-MLLM

### A.1 Ascend v.s. Nvidia

**Table 9: Comparison Between NVIDIA CUDA and Ascend CANN.**
| Component | CUDA | CANN |
| --- | --- | --- |
| GE Graph Engine | TensorRT plugins & parser | Graph Engine |
| Collective Communication Library | NV NCCL | HCCL |
| Library/Template | NV CUTLASS | Ascend C High-Level API |
| General Programming | NV CUDA-C | Ascend C Low-Level API |
| Operator Acceleration Library | NV cuDNN | Ascend aclNN |
| Runtime | NV Runtime | Ascend Runtime |
| Driver | NV Driver | Ascend Driver |

The ecosystem serves as the cornerstone of distributed pre-training frameworks, with NVIDIA’s CUDA and Huawei’s Ascend CANN being the two mainstream ecosystems.
While both possess dedicated driver and runtime layers for OS-AI accelerator communication as illustrated in Table [9](https://arxiv.org/html/2509.11662v3#A1.T9), they differ significantly in middle-level acceleration libraries, upper-level tools, and ecological maturity.
NVIDIA’s cuDNN and NCCL are highly optimized, and with CUTLASS and TensorRT providing flexible programming templates and graph optimization, it has formed a mature ”hardware-software-tools-community” loop, supported by in-depth adaptation of open source communities like those behind PyTorch and Megatron-LM.

In contrast, Ascend’s aclNN, HCCL, and two-tier Ascend C API system have realized core functions but lack coverage of long-tail operators, third-party tools, and community resources.
This gap limits Ascend’s training framework selection—mainstream CUDA-based frameworks cannot directly leverage Ascend’s hardware capabilities, and simple adaptation incurs high costs and performance losses, highlighting the necessity of developing a dedicated framework for Ascend to integrate with CANN components seamlessly.

**Table 10: Comparison of Differences Between Huawei Ascend and NVIDIA Computing Cards.**
| XPU | A100 | A800 | H800 | 910B1 | 910B2 |
| --- | --- | --- | --- | --- | --- |
| TF32 (Tensor Core) | 156 | 156 | 495 | 200 | 188 |
| FP32 (Tensor Core) | NA | NA | NA | 100 | 94 |
| BF16 (Tensor Core) | 312 | 312 | 989 | 400 | 376 |
| FP16 (Tensor Core) | 312 | 312 | 989 | 400 | 376 |
| Int8 (Tensor Core) | 624 | 624 | 1979 | 800 | 752 |
| HBM (GB) | 80 | 80 | 80 | 64 | 64 |
| HBM Bandwidth (GB/s) | 2039 | 2039 | 3350 | 1800 | 1800 |
| NVLink Bandwidth (GB/s) | 600 | 400 | 600 | NA | NA |
| PCIe Bandwidth (GB/s) | 64 | 64 | 128 | 64 | 64 |

Beyond underlying chip architecture differences, Ascend and NVIDIA computing cards vary distinctly in core specifications that impact large-model pre-training efficiency and stability as illustrated in [10](https://arxiv.org/html/2509.11662v3#A1.T10).
In terms of computing power, Ascend 910B shows competitiveness at mainstream precisions and unique advantages in FP32 Tensor Core support.
However, Ascend has obvious shortcomings: 64 GB HBM risks training interruptions for ultra-large models, 1800 GB/s HBM bandwidth is lower than NVIDIA’s, and without NVLink, it relies on 64 GB/s PCIe for multi-card communication, increasing latency.

To mitigate these, hardware-friendly strategies are needed, which further drives Ascend-specific framework optimizations—such as developing fused operators, dynamic scheduling, and communication optimization to offset shortcomings and leverage its computing power.

### A.2 Precision Alignment

To verify the forward and backward precision of the developed framework, we conducted comparisons not only with the original MindSpeed-MM but also with the following mainstream training frameworks within the NVIDIA ecosystem:

Llama-FactoryZheng et al. (2024): an easy-to-use open-source training framework.
It is mainly built on native PyTorch and the HuggingFace ecosystem, and supports distributed engines such as DeepSpeed and FSDP.
Currently, its support for the Ascend ecosystem is becoming increasingly comprehensive.

Pai-Megatron-PatchCloud (Year): an open-source training tool developed by Alibaba.
Similar to MindSpeed-LLM/MM, it includes adaptation and optimization based on Megatron for mainstream open-source LLMs.

Figure: Figure 5: Comparison of loss values among various training tools and platforms on the COCO dataset.
Refer to caption: figures/infra/coco_loss.png

Two datasets were employed to validate the accuracy of the training framework: the public COCO dataset(Lin et al., 2014) and an in-house Slow Thinking dataset.
In all training experiments, consistent training hyper-parameters were maintained.
Specifically, the training length was set to 4K for the COCO dataset, whereas a 12K training length was adopted for the Slow Thinking dataset.

Firstly, we checked the forward precision of the MindSpeed-MLLM.
By manually feeding hundreds of identical inputs into MindSpeed-MLLM and the transformer-based forward code, the Mean Absolute Error (MAE) and Mean Relative Error (MRE) are both within five thousandths.

Next, we conducted precision checks through training tasks.
In the COCO data set, as illustrated in Figure [5](https://arxiv.org/html/2509.11662v3#A1.F5), the loss curves of all frameworks exhibited very similar trends, although with minor discrepancies. These differences are deemed acceptable, as the frameworks utilize distinct data loading modules and workflows.

To further validate consistency, we adapted our custom data loader to MindSpeed-MM, ensuring full alignment of the data load logic, and conducted comparative experiments on the in-house Slow Thinking dataset.
As illustrated in Figures [6](https://arxiv.org/html/2509.11662v3#A1.F6) and [7](https://arxiv.org/html/2509.11662v3#A1.F7), after standardizing the data loading process, the loss values of MindSpeed-MLLM and MindSpeed-MM showed a near-perfect overlap, with the loss difference confined to within a percentile range.
This marginal discrepancy is considered acceptable, given the inherent different associated with the fused operators employed, as well as the randomness with computation and communication processes.

In the final benchmark evaluation, under the same training set, the models trained using Llama-Factory and MindSpeed-MLLM respectively achieved comparable results on the general visual and text benchmarks, with an error margin within ±1.5.

Figure: Figure 6: loss decline trend on the in-house slow thinking dataset.
Refer to caption: figures/infra/slowthinking_loss.png

Figure: Figure 7: loss difference between MindSpeed-MLLM and the original MindSpeed-MM.
Refer to caption: figures/infra/slowthinking_loss_compare.png

## Appendix B Data Curation

The MindVL training corpus contains 447 billion diverse and high-quality tokens used for three training stages. The data is categorized according to target capabilities, and the curation process for each category is detailed in the following subsections.

### B.1 warm-up Data

The MindVL warm-up corpus contains 256 billion diverse, high-quality tokens, including Generic Image-Text Pairs, Optical Character Recognition (OCR), Visual Grounding & Counting, Science, Technology, Engineering, and Mathematics (STEM), and Graphical User Interface (GUI).

**Table 11: Warm-up Phase Dataset Composition**
| Category | Dataset | Total Num | Percentage (%) |
| --- | --- | --- | --- |
| Caption | Caption-CN-Rawcaption | 132,900,000 | 21.92 |
| Caption | Caption-EN-Recaption | 102,500,000 | 16.90 |
| Caption | Caption-EN-Rawcaption | 42,041,409 | 6.93 |
| Caption | CapsFusion-en | 35,840,000 | 5.91 |
| Caption | Caption-CN-Recaption | 25,300,000 | 4.17 |
| Caption | HiVision | 1,800,000 | 0.30 |
| Caption Subtotal |  | 338,381,409 | 55.81 |
| Grounding | UMG-en | 42,240,000 | 6.97 |
| Grounding | Grounding | 35,000,000 | 5.77 |
| Grounding | UMG-zh | 14,080,000 | 2.32 |
| Grounding | GRIT-en | 8,448,000 | 1.39 |
| Grounding Subtotal |  | 99,768,000 | 16.45 |
| OCR | WuKong-Text-zh | 33,280,000 | 5.49 |
| OCR | LAION-Text-en | 14,336,000 | 2.36 |
| OCR | Document-OCR | 17,000,000 | 2.80 |
| OCR | Scene-OCR | 17,000,000 | 2.80 |
| OCR | IIT-CDIP-en | 3,840,000 | 0.63 |
| OCR | Rendered-Text-en | 1,920,000 | 0.32 |
| OCR | PDF-en | 2,560,000 | 0.42 |
| OCR | PDF-zh | 2,560,000 | 0.42 |
| OCR | arXiv-en | 2,304,000 | 0.38 |
| OCR | DOCX-en | 960,000 | 0.16 |
| OCR | DOCX-zh | 960,000 | 0.16 |
| OCR | README | 1,280,000 | 0.21 |
| OCR | Scene-Text | 560,000 | 0.09 |
| OCR Subtotal |  | 98,000,000 | 16.16 |
| Table&Chart | Table-Data | 10,000,000 | 1.65 |
| Table&Chart | Chart-SFT-en | 1,440,000 | 0.24 |
| Table&Chart | PubTables-en | 120,000 | 0.02 |
| Table&Chart | PubTables-zh | 120,000 | 0.02 |
| Table&Chart | MMC-Align-en | 420,000 | 0.07 |
| Table&Chart | MMC-Instruct-en | 420,000 | 0.07 |
| Table&Chart Subtotal |  | 12,520,000 | 2.06 |
| STEM | STEM-Caption-0729-en | 2,230,000 | 0.37 |
| STEM | STEM-Caption-0729-zh | 2,230,000 | 0.37 |
| STEM | STEM-Shuffle | 1,000,000 | 0.16 |
| STEM Subtotal |  | 5,460,000 | 0.90 |
| GUI | Screen-UI-zh | 360,000 | 0.06 |
| GUI | Screen-UI-en | 120,000 | 0.02 |
| GUI | Screen-QA-zh | 120,000 | 0.02 |
| GUI | Screen-Ref-zh | 120,000 | 0.02 |
| GUI | Screen-Navi-zh | 120,000 | 0.02 |
| GUI Subtotal |  | 840,000 | 0.14 |
| Total |  | 606,221,409 | 100.00 |

#### B.1.1 Generic Image-Text Pairs

Web-sourced image-text pairs, which include alt texts, captions, and surrounding contextual text, are now available at an unprecedented scale. With billions of examples, they showcase remarkable diversity across visual and textual concepts. However, such data is inherently noisy, often containing texts that are irrelevant or factually inaccurate relative to the corresponding images, and it frequently exhibits class imbalance.

To address these challenges, we have adopted a series of filtering measures, including image-based criteria, text-based criteria, aesthetic scoring, pornography and violence scoring, watermark scoring, CLIP scoring (Radford et al., 2021), and URL filtering. For data with inadequate image-text relevance, we use a model (Bai et al., 2025; Chen et al., 2024) to recapture the images.

Furthermore, to cover more visual concepts and preserve as many types of long-tail visual concepts as possible, we perform clustering on the images within the filtered image-text pairs. We then manually inspect the data in each cluster and map these clusters to the data’s category labels. For clusters that represent overly broad concepts, we conduct re-clustering. Finally, we obtain the results of hierarchical clustering: a larger variance within a cluster indicates greater diversity. We perform data sampling based on variance—clusters with larger variance undergo more extensive sampling, while clusters with smaller variance are sampled less.

#### B.1.2 Optical Character Recognition (OCR)

OCR data consists of two types, document OCR data and scene OCR data, each following a structured processing workflow. For document OCR data, the process starts with acquiring and initially screening source PDF data, then processes PDFs page-by-page via a pipeline to generate image-text pairs and image-text interleaved formats. After extracting these data (with image markers removed from pair data and text-only MD files filtered from interleaved data), rule-based cleaning addresses issues like garbled characters or abnormal spaces. For scene OCR data, pre-processed tar packages are first sent to the Focus OCR model (Liu et al., 2024b) to generate OCR results; the same tar packages are then processed by the PaddleOCR model (Cui et al., 2025), with results from both models intersected to form production-line OCR results. In addition to the aforementioned process, we have also calculated the coverage rate of words and characters in OCR text, and conducted targeted supplementation for data with relatively low coverage. In addition, we have also incorporated data such as small-language content, ancient books, and calligraphy works.

#### B.1.3 Tabel & Chart

For raw tables, we perform layout detection and content recognition separately. Specifically, we use the RapidTable (Team,) and LORE (Xing et al., 2023; Long et al., 2025) models for layout detection and layout judgment, and PaddleOCR for content recognition. Subsequently, we integrate the results and conduct post-processing for HTML formatting. Finally, we convert the table images into HTML format. Alongside the previously mentioned data, we also collect open data to enhance the ability
to interpret tables and charts. For tables, we use the PubTables-1M dataset (Smock et al., 2022), including both its original English version and a translated Chinese version, to gather table recognition data. For charts, we employ chart-to-table conversion and chart-based QA data from existing datasets, including MMC (Liu et al., 2023) and ChartSFT (Meng et al., 2024).

#### B.1.4 Visual Grounding

First, we perform recaptioning on the crawled and open-source images. We then use the Florence-2-large model (Xiao et al., 2024) to conduct caption to phrase grounding processing. After obtaining object bounding boxes and region captions, we employ SAM (Kirillov et al., 2023) to generate point annotations. In addition, we utilize GrIT-20M (Peng et al., 2023), a synthetic caption dataset with additional location labels for major visual elements, to enhance the grounding capability. We incorporate referring and grounding data from UMG-41M (Shi et al., 2024). Specifically, each region is randomly assigned to either a referring task or a grounding task. In the referring task, we provide the model with a bounding box to generate a caption for that specific region, while in the grounding task, we reverse this by using the caption to predict the corresponding bounding box.

#### B.1.5 Science, Technology, Engineering, and Mathematics (STEM)

To enhance the model’s disciplinary knowledge, we incorporated a diverse collection of data across various STEM domains, obtained through both crawling and manual annotation. We referred to the disciplinary classifications of MMMU (Yue et al., 2024) and K12, extracted such data from a vast volume of PDFs, and also collected publicly available data. Additionally, we used a model to re-annotate the sourced disciplinary images. Ultimately, we obtained nearly 5 million data samples.

#### B.1.6 Graphical User Interface (GUI)

For GUI data processing, we first perform pre-annotation via GPT-4V (OpenAI, 2023), which involves tasks such as Referring&Grounding (for icons, text, and coitems), ScreenQA, and Navigation. Next, human verification is conducted by human annotators, focusing on Referring-coitem, QA, and the answers for Navigation. After that, a sequence of data post-processing operations is carried out: first, merging with original annotations; second, using UI-Hawk (Zhang et al., 2024b) and GLM-4V (GLM et al., 2024) to annotate referring-icon, and UI-Hawk to annotate navigation; third, employing GPT-4o (Hurst et al., 2024) to annotate Navigation actions; fourth, merging the results of these three models and performing consistency checks; finally, rewriting the answer style with Qwen2.5-7B-Instruct to ensure consistency and linguistic quality.

The data used during the warm-up phase is shown in the Table [11](https://arxiv.org/html/2509.11662v3#A2.T11). Note that the ”Total num” in the Table [11](https://arxiv.org/html/2509.11662v3#A2.T11) is an approximate value. 606,221,409 samples are about 256B tokens. Apart from CapsFusion, UMG, GRIT, PubTables, ChartSFT, and MMC, all other datasets are either self-constructed or have undergone further processing of open-source data. The 16B tokens of data used in MindVL-671B-A37B are uniformly sampled from all warm-up datasets.

### B.2 Multitask Training Data

Multitask training data consists of internleaved image-text (Table [12](https://arxiv.org/html/2509.11662v3#A2.T12)), visual instruction (Tabel LABEL:tab:multitask, about 80B tokens), web2code and text instruction (Tabel [13](https://arxiv.org/html/2509.11662v3#A2.T13)), totally 179B tokens.

#### B.2.1 Internleaved Image-Text Data

For interleaved image-text data, we primarily crawl Chinese websites, English websites, news websites, and Wikipedia from the internet. We then use model-based methods to filter the crawled data. First, we apply CLIP scores to filter the data according to image-text relevance. Next, we task the model with scoring the image-text interleaved data across multiple dimensions, including question-and-answer quality, relevance between images and questions, complementarity between answers, images and questions, and balance of information density. The model is also required to provide justifications for these scores. Finally, we use these scores to filter the image-text interleaved data.

**Table 12: Data Statistics and Proportion of Interleaved Image-Text Data. “GPT4o” / “QWEN” denotes we use GPT4o / Qwen2.5-VL 72B filter the crawled data.**
| Data Category | Single-image Count | Multi-image Count | Total Count | Proportion (%) |
| --- | --- | --- | --- | --- |
| HTMLENGPT4o | 2994750 | 2994750 | 5989500 | 12.24 |
| HTMLENQWEN | 356942 | 356942 | 713884 | 1.46 |
| HTMLZHGPT4o | 763016 | 763016 | 1526032 | 3.12 |
| HTMLZHQWEN | 8096192 | 8096192 | 16192384 | 33.09 |
| OBELICSGPT4o | 3339440 | 3339440 | 6678880 | 13.65 |
| News Webpages | 4167856 | 4167856 | 8335712 | 17.03 |
| Knowledge | — | — | 7500000 | 15.33 |
| Wikiweb2m | — | — | 2000000 | 4.09 |
| Total | — | — | 48936392 | 100.00 |

For other types of multitask training data, we have collected a wide range of open source data, as presented in Table LABEL:tab:multitask. Within this dataset, we used Qwen2.5VL-72B to reannotate certain portions, such as those whose names begin with ”Qwen”. In addition, we also sampled some data from the warm-up phase and incorporated it into the training process at this stage. 200M web2code data is also used in this stage. Furthermore, we utilized language data produced by DeepSeek R1 to preserve the MLLM’s linguistic abilities as shown in Tabel [13](https://arxiv.org/html/2509.11662v3#A2.T13), with the ratio of multimodal to language data being approximately 8:2 at this stage.

**Table 13: Data Statistics and Proportion of text instruction.**
| Data Source | Number | Proportion (%) |
| --- | --- | --- |
| Competition Mathematics | 345606 | 16.06 |
| Code | 113064 | 5.25 |
| Math | 296500 | 13.78 |
| General QA | 1316361 | 61.18 |
| Tulu Instruction | 19118 | 0.89 |
| Multi-turn Conversation | 61114 | 2.84 |
| Total | 2151763 | 100.00 |

**Table 14: Data Statistics and Proportion of Visual Instruction**
| No. | Data Set | Number | Percentage (%) |
| --- | --- | --- | --- |
| 1 | Cauldron/ai2d | 2434 | 0.0064 |
| 2 | Cauldron/cocoqa | 46287 | 0.1214 |
| 3 | Cauldron/datikz | 47974 | 0.1258 |
| 4 | Cauldron/diagram-image-to-text | 300 | 0.0008 |
| 5 | Cauldron/docvqa | 10189 | 0.0267 |
| 6 | Cauldron/dvqa | 199937 | 0.5242 |
| 7 | Cauldron/figureqa | 100000 | 0.2622 |
| 8 | Cauldron/finqa | 5276 | 0.0138 |
| 9 | Cauldron/geomverse | 9303 | 0.0244 |
| 10 | Cauldron/hateful-memes | 8500 | 0.0223 |
| 11 | Cauldron/hitab | 2500 | 0.0066 |
| 12 | Cauldron/iam | 5663 | 0.0148 |
| 13 | Cauldron/iconqa | 27307 | 0.0716 |
| 14 | Cauldron/infographic-vqa | 2118 | 0.0056 |
| 15 | Cauldron/intergps | 1280 | 0.0034 |
| 16 | Cauldron/localized-narratives | 199998 | 0.5244 |
| 17 | Cauldron/mapqa | 37417 | 0.0981 |
| 18 | Cauldron/mimic-cgd | 70939 | 0.1860 |
| 19 | Cauldron/multihiertt | 7619 | 0.0200 |
| 20 | Cauldron/nlvr2 | 50426 | 0.1322 |
| 21 | Cauldron/plotqa | 157007 | 0.4117 |
| 22 | Cauldron/raven | 42000 | 0.1091 |
| 23 | Cauldron/rendered-text | 10000 | 0.0262 |
| 24 | Cauldron/robut-sqa | 8514 | 0.0223 |
| 25 | Cauldron/robut-wikisql | 74989 | 0.1966 |
| 26 | Cauldron/robut-wtq | 38246 | 0.1003 |
| 27 | Cauldron/scienceqa | 4976 | 0.0130 |
| 28 | Cauldron/screen2words | 15730 | 0.0412 |
| 29 | Cauldron/spot-the-diff | 8566 | 0.0225 |
| 30 | Cauldron/tabmwp | 22722 | 0.0596 |
| 31 | Cauldron/tallyqa | 98680 | 0.2587 |
| 32 | Cauldron/tat-qa | 2199 | 0.0058 |
| 33 | Cauldron/textcaps | 21953 | 0.0576 |
| 34 | Cauldron/textvqa | 21953 | 0.0576 |
| 35 | Cauldron/tqa | 1493 | 0.0039 |
| 36 | Cauldron/vistext | 9969 | 0.0261 |
| 37 | Cauldron/visual7w | 14366 | 0.0377 |
| 38 | Cauldron/visualmrc | 3027 | 0.0079 |
| 39 | Cauldron/vqarad | 313 | 0.0008 |
| 40 | Cauldron/vsr | 2157 | 0.0056 |
| 41 | Cauldron/websight | 10000 | 0.0262 |
| 42 | ChartQA/augmented | 15474 | 0.0406 |
| 43 | ChartQA/cap | 18207 | 0.0477 |
| 44 | ChartQA/cot | 18215 | 0.0478 |
| 45 | ChartQA/gemini-v | 16393 | 0.0430 |
| 46 | ChartQA/gemini-v-cot | 16393 | 0.0430 |
| 47 | ChartQA/human | 3699 | 0.0097 |
| 48 | ChartSFT/unichart | 120554 | 0.3161 |
| 49 | CoSyn-400K/chart | 116807 | 0.3063 |
| 50 | CoSyn-400K/chemical | 8935 | 0.0234 |
| 51 | CoSyn-400K/circuit | 10463 | 0.0274 |
| 52 | CoSyn-400K/diagram | 34956 | 0.0916 |
| 53 | CoSyn-400K/document | 71275 | 0.1869 |
| 54 | CoSyn-400K/graphic | 26961 | 0.0707 |
| 55 | CoSyn-400K/math | 66707 | 0.1749 |
| 56 | CoSyn-400K/music | 11962 | 0.0314 |
| 57 | CoSyn-400K/nutrition | 6924 | 0.0181 |
| 58 | CoSyn-400K/table | 46511 | 0.1220 |
| 59 | CoSyn-point | 19385 | 0.0508 |
| 60 | DocVQA/cap | 9892 | 0.0259 |
| 61 | DocVQA/cot | 11938 | 0.0313 |
| 62 | DocVQA/gemini-v | 11182 | 0.0293 |
| 63 | DocVQA/gemini-v-cot | 11182 | 0.0293 |
| 64 | DocVQA/human | 10194 | 0.0267 |
| 65 | InfoVQA/cap | 4290 | 0.0112 |
| 66 | InfoVQA/cot | 6060 | 0.0159 |
| 67 | InfoVQA/gemini-v | 5187 | 0.0136 |
| 68 | InfoVQA/gemini-v-cot | 5187 | 0.0136 |
| 69 | InfoVQA/human | 4406 | 0.0116 |
| 70 | M4-Instruct/3D-LLM-3-datasets | 49890 | 0.1308 |
| 71 | M4-Instruct/AESOP | 6915 | 0.0181 |
| 72 | M4-Instruct/ALFRED | 22565 | 0.0592 |
| 73 | M4-Instruct/Birds-to-Words | 14281 | 0.0374 |
| 74 | M4-Instruct/CLEVR-Change | 3885 | 0.0102 |
| 75 | M4-Instruct/FlintstonesSV | 22341 | 0.0586 |
| 76 | M4-Instruct/HQ-Edit | 50000 | 0.1311 |
| 77 | M4-Instruct/HQ-Edit-Diff | 7000 | 0.0184 |
| 78 | M4-Instruct/IEdit | 3456 | 0.0091 |
| 79 | M4-Instruct/MIT-States-PropertyCoherence | 1900 | 0.0050 |
| 80 | M4-Instruct/MIT-States-StateCoherence | 1900 | 0.0050 |
| 81 | M4-Instruct/MagicBrush | 14249 | 0.0374 |
| 82 | M4-Instruct/MagicBrush-Diff | 6698 | 0.0176 |
| 83 | M4-Instruct/PororoSV | 12299 | 0.0322 |
| 84 | M4-Instruct/RAVEN | 35000 | 0.0918 |
| 85 | M4-Instruct/RecipeQA-ImageCoherence | 8699 | 0.0228 |
| 86 | M4-Instruct/ScanQA | 25563 | 0.0670 |
| 87 | M4-Instruct/TQA | 8249 | 0.0216 |
| 88 | M4-Instruct/VISION | 9900 | 0.0260 |
| 89 | M4-Instruct/VIST | 26026 | 0.0682 |
| 90 | M4-Instruct/WebQA | 9338 | 0.0245 |
| 91 | M4-Instruct/coinstruct | 50000 | 0.1311 |
| 92 | M4-Instruct/contrastive-caption | 25240 | 0.0662 |
| 93 | M4-Instruct/dreamsim | 15941 | 0.0418 |
| 94 | M4-Instruct/iconqa | 34603 | 0.0907 |
| 95 | M4-Instruct/imagecode | 16594 | 0.0435 |
| 96 | M4-Instruct/multi-vqa | 4993 | 0.0131 |
| 97 | M4-Instruct/nextqa | 3870 | 0.0102 |
| 98 | M4-Instruct/nlvr2 | 86373 | 0.2265 |
| 99 | M4-Instruct/star | 3032 | 0.0079 |
| 100 | M4-Instruct/twitter-post | 5734 | 0.0150 |
| 101 | Markdown/arxiv-v2 | 200000 | 0.5244 |
| 102 | Markdown/docx-en | 200000 | 0.5244 |
| 103 | Markdown/docx-zh | 200000 | 0.5244 |
| 104 | Markdown/pubtables-c | 200000 | 0.5244 |
| 105 | Markdown/pubtables-e | 200000 | 0.5244 |
| 106 | Markdown/pubtables-o | 200000 | 0.5244 |
| 107 | Markdown/readme-v2 | 200000 | 0.5244 |
| 108 | Math-PUMA/Synthesis-train-answer | 120239 | 0.3153 |
| 109 | Math-PUMA/Synthesis-train-instruction | 120239 | 0.3153 |
| 110 | Math-PUMA/VarsityTutors-pure-text | 232959 | 0.6108 |
| 111 | Math-PUMA/VarsityTutors-with-image | 78935 | 0.2070 |
| 112 | Math-PUMA/VisualWebInstruct-train-answer | 263599 | 0.6911 |
| 113 | OCR/iit-cdip-v2-en | 200000 | 0.5244 |
| 114 | OCR/laion-text-en | 50000 | 0.1311 |
| 115 | OCR/laion-text-v2-en | 200000 | 0.5244 |
| 116 | OCR/pdf-en | 200000 | 0.5244 |
| 117 | OCR/pdf-zh | 200000 | 0.5244 |
| 118 | OCR/wukong-text-v2-zh | 200000 | 0.5244 |
| 119 | OCR/wukong-text-zh | 50000 | 0.1311 |
| 120 | OneVision/ai2d-gpt4v | 4833 | 0.0127 |
| 121 | OneVision/ai2d-internvl | 12372 | 0.0324 |
| 122 | OneVision/clevr-math-mathv360k | 5249 | 0.0138 |
| 123 | OneVision/figureqa-mathv360k | 17556 | 0.0460 |
| 124 | OneVision/geo3k | 2060 | 0.0054 |
| 125 | OneVision/geometry3k-mathv360k | 9693 | 0.0254 |
| 126 | OneVision/geoqa-plus-mathv360k | 17131 | 0.0449 |
| 127 | OneVision/geos-mathv360k | 467 | 0.0012 |
| 128 | OneVision/iconqa-mathv360k | 22558 | 0.0591 |
| 129 | OneVision/iiit5k | 1959 | 0.0051 |
| 130 | OneVision/image-textualization-filtered | 99542 | 0.2610 |
| 131 | OneVision/infographic-gpt4v | 1951 | 0.0051 |
| 132 | OneVision/k12-printing | 256605 | 0.6728 |
| 133 | OneVision/lrv-chart | 1745 | 0.0046 |
| 134 | OneVision/lrv-normal-filtered | 10456 | 0.0274 |
| 135 | OneVision/mapqa-mathv360k | 5194 | 0.0136 |
| 136 | OneVision/orand-car-a | 1968 | 0.0052 |
| 137 | OneVision/pmc-vqa-mathv360k | 35917 | 0.0942 |
| 138 | OneVision/scienceqa-nona-context | 19177 | 0.0503 |
| 139 | OneVision/super-clevr-mathv360k | 8611 | 0.0226 |
| 140 | OneVision/tabmwp-mathv360k | 22421 | 0.0588 |
| 141 | OneVision/unigeo-mathv360k | 11918 | 0.0312 |
| 142 | OneVision/vision-flan-filtered | 186029 | 0.4877 |
| 143 | OneVision/vizwiz-mathv360k | 6573 | 0.0172 |
| 144 | PixMo/ask-model-anything | 161737 | 0.4241 |
| 145 | PixMo/ask-model-anything-zh | 140343 | 0.3679 |
| 146 | PixMo/cap | 717042 | 1.8800 |
| 147 | PixMo/cap-qa | 271714 | 0.7124 |
| 148 | PixMo/cap-qa-zh | 194097 | 0.5090 |
| 149 | PixMo/cap-zh | 508134 | 1.3323 |
| 150 | PixMo/count | 36916 | 0.0968 |
| 151 | PixMo/docs-charts | 116814 | 0.3063 |
| 152 | PixMo/docs-diagrams | 16551 | 0.0434 |
| 153 | PixMo/docs-other | 71282 | 0.1869 |
| 154 | PixMo/docs-tables | 46518 | 0.1220 |
| 155 | PixMo/point-explanations | 79551 | 0.2086 |
| 156 | PixMo/points | 2376222 | 6.2301 |
| 157 | QwenCaps/PixMo/cap | 712401 | 1.8680 |
| 158 | QwenCaps/PixMo/cap-zh | 642166 | 1.6837 |
| 159 | QwenCaps/ai2d | 9778 | 0.0256 |
| 160 | QwenCaps/allava-laion | 480858 | 1.2608 |
| 161 | QwenCaps/chart2text | 16605 | 0.0435 |
| 162 | QwenCaps/chart-to-text | 44096 | 0.1156 |
| 163 | QwenCaps/chartqa | 18317 | 0.0480 |
| 164 | QwenCaps/coco | 163119 | 0.4277 |
| 165 | QwenCaps/docvqa | 11166 | 0.0293 |
| 166 | QwenCaps/flickr30k | 31772 | 0.0833 |
| 167 | QwenCaps/funsd | 149 | 0.0004 |
| 168 | QwenCaps/infovqa | 4890 | 0.0128 |
| 169 | QwenCaps/laion-gpt4v | 10962 | 0.0287 |
| 170 | QwenCaps/llava | 557086 | 1.4606 |
| 171 | QwenCaps/llavar-s1 | 47231 | 0.1238 |
| 172 | QwenCaps/ocr-vqa | 208190 | 0.5458 |
| 173 | QwenCaps/opencqa | 7724 | 0.0202 |
| 174 | QwenCaps/poie | 2999 | 0.0079 |
| 175 | QwenCaps/sam | 33533 | 0.0879 |
| 176 | QwenCaps/scienceqa | 4985 | 0.0131 |
| 177 | QwenCaps/screen2words | 15669 | 0.0411 |
| 178 | QwenCaps/sroie | 608 | 0.0016 |
| 179 | QwenCaps/textvqa | 24990 | 0.0655 |
| 180 | QwenCaps/vg | 105345 | 0.2762 |
| 181 | QwenCaps/vision-flan | 181189 | 0.4751 |
| 182 | QwenCaps/vistext | 8816 | 0.0231 |
| 183 | QwenCaps/xfund | 1043 | 0.0027 |
| 184 | QwenQA/Cauldron/cocoqa | 27022 | 0.0708 |
| 185 | QwenQA/Cauldron/diagram-image-to-text | 297 | 0.0008 |
| 186 | QwenQA/Cauldron/hateful-memes | 3729 | 0.0098 |
| 187 | QwenQA/Cauldron/mapqa | 22950 | 0.0602 |
| 188 | QwenQA/Cauldron/tabmwp | 22410 | 0.0588 |
| 189 | QwenQA/Cauldron/tallyqa | 83671 | 0.2194 |
| 190 | QwenQA/Cauldron/textvqa | 16059 | 0.0421 |
| 191 | QwenQA/Cauldron/tqa | 861 | 0.0023 |
| 192 | QwenQA/Cauldron/visual7w | 13006 | 0.0341 |
| 193 | QwenQA/Cauldron/vsr | 1650 | 0.0043 |
| 194 | QwenQA/ChartQA/augmented | 12469 | 0.0327 |
| 195 | QwenQA/ChartQA/human | 3290 | 0.0086 |
| 196 | QwenQA/DocVQA/human | 8924 | 0.0234 |
| 197 | QwenQA/InfoVQA/human | 4092 | 0.0107 |
| 198 | QwenQA/PixMo/cap-qa | 247256 | 0.6483 |
| 199 | QwenQA/PixMo/cap-qa-zh | 218868 | 0.5738 |
| 200 | QwenQA/ai2d | 10593 | 0.0278 |
| 201 | QwenQA/allava-laion | 484257 | 1.2697 |
| 202 | QwenQA/allava-vflan | 179020 | 0.4694 |
| 203 | QwenQA/aokvqa | 39358 | 0.1032 |
| 204 | QwenQA/gqa | 69198 | 0.1814 |
| 205 | QwenQA/okvqa | 3442 | 0.0090 |
| 206 | QwenQA/raw/ChartQA/augmented | 15474 | 0.0406 |
| 207 | QwenQA/raw/ChartQA/human | 3699 | 0.0097 |
| 208 | QwenQA/raw/DocVQA/human | 10194 | 0.0267 |
| 209 | QwenQA/raw/InfoVQA/human | 4394 | 0.0115 |
| 210 | QwenQA/raw/ai2d | 12413 | 0.0325 |
| 211 | QwenQA/raw/aokvqa | 76762 | 0.2013 |
| 212 | QwenQA/raw/gqa | 72140 | 0.1891 |
| 213 | QwenQA/raw/okvqa | 8996 | 0.0236 |
| 214 | QwenQA/raw/scienceqa | 12726 | 0.0334 |
| 215 | QwenQA/raw/stvqa | 18921 | 0.0496 |
| 216 | QwenQA/raw/tabfact | 13127 | 0.0344 |
| 217 | QwenQA/raw/vqav2 | 82783 | 0.2171 |
| 218 | QwenQA/raw/wtq | 1679 | 0.0044 |
| 219 | QwenQA/scienceqa | 11071 | 0.0290 |
| 220 | QwenQA/stvqa | 14381 | 0.0377 |
| 221 | QwenQA/tabfact | 12648 | 0.0332 |
| 222 | QwenQA/vqav2 | 78718 | 0.2064 |
| 223 | QwenQA/wtq | 1588 | 0.0042 |
| 224 | R1-Vision/PixMo/cap-qa | 260144 | 0.6821 |
| 225 | R1-Vision/PixMo/cap-qa-zh | 218962 | 0.5741 |
| 226 | Screen/grd | 212472 | 0.5571 |
| 227 | Screen/grd-small | 212472 | 0.5571 |
| 228 | Screen/nav | 286364 | 0.7508 |
| 229 | Screen/nav-small | 286364 | 0.7508 |
| 230 | Screen/ref | 402742 | 1.0560 |
| 231 | Screen/ref-small | 402742 | 1.0560 |
| 232 | Screen/sqa | 417579 | 1.0948 |
| 233 | Screen/sqa-small | 417579 | 1.0948 |
| 234 | Screen/ui-en | 50000 | 0.1311 |
| 235 | Screen/ui-en-small | 50000 | 0.1311 |
| 236 | Screen/ui-zh | 50000 | 0.1311 |
| 237 | Screen/ui-zh-small | 50000 | 0.1311 |
| 238 | XFUND/zh | 149 | 0.0004 |
| 239 | XFUND/zh-qa | 671 | 0.0018 |
| 240 | Allava-caption-laion | 505588 | 1.3256 |
| 241 | Allava-caption-vflan | 182864 | 0.4794 |
| 242 | Allava-instruct-laion | 505588 | 1.3256 |
| 243 | Allava-instruct-vflan | 183839 | 0.4820 |
| 244 | Aokvqa | 76762 | 0.2013 |
| 245 | Chartsft | 808768 | 2.1205 |
| 246 | Cipc-docvqa | 5243 | 0.0137 |
| 247 | Coavqa-bench | 36019 | 0.0944 |
| 248 | Coavqa-chat | 18004 | 0.0472 |
| 249 | Coco-rec | 130723 | 0.3427 |
| 250 | Coco-reg | 130723 | 0.3427 |
| 251 | Crohme | 9821 | 0.0257 |
| 252 | Datikz-v2 | 94532 | 0.2479 |
| 253 | Dcc | 97306 | 0.2551 |
| 254 | Docmatix | 1249967 | 3.2773 |
| 255 | E2e-ocr | 259511 | 0.6804 |
| 256 | E2e-ocr-0324 | 359996 | 0.9440 |
| 257 | Eikie-qa | 183639 | 0.4815 |
| 258 | Estvqa | 17047 | 0.0447 |
| 259 | Estvqa-ca | 17047 | 0.0447 |
| 260 | Estvqa-grounding | 17047 | 0.0447 |
| 261 | Flickr | 148915 | 0.3904 |
| 262 | Funsd | 149 | 0.0004 |
| 263 | Funsd-qa | 431 | 0.0011 |
| 264 | Geo170k-alignment | 60252 | 0.1580 |
| 265 | Geo170k-qa-tuning | 117205 | 0.3073 |
| 266 | Geo-synth-caption-1108 | 320000 | 0.8390 |
| 267 | Geo-synth-qa-1108 | 320004 | 0.8390 |
| 268 | Gqa | 72140 | 0.1891 |
| 269 | Hme | 74502 | 0.1953 |
| 270 | Kvqa | 24602 | 0.0645 |
| 271 | Laion-gpt4v | 12356 | 0.0324 |
| 272 | Llava-s2 | 157712 | 0.4135 |
| 273 | Llarar-s2 | 19800 | 0.0519 |
| 274 | Lrv-instruction | 180722 | 0.4738 |
| 275 | Lvis-instruct4v | 222711 | 0.5840 |
| 276 | M2e | 99956 | 0.2621 |
| 277 | Mavis-900k | 458042 | 1.2009 |
| 278 | Mavis-caps | 599748 | 1.5725 |
| 279 | Mlhme | 30000 | 0.0786 |
| 280 | Mmc-non-arxiv-2 | 21103 | 0.0553 |
| 281 | Mmc-non-arxiv-3 | 88917 | 0.2331 |
| 282 | Noahcaps-en | 14956 | 0.0392 |
| 283 | Noahcaps-zh | 44394 | 0.1164 |
| 284 | Objects365 | 1737995 | 4.5568 |
| 285 | Ocrvqa | 207572 | 0.5442 |
| 286 | Okvqa | 8996 | 0.0236 |
| 287 | Petal-wiki | 2396596 | 6.2836 |
| 288 | Poie | 2250 | 0.0059 |
| 289 | Pointqa-general | 82254 | 0.2157 |
| 290 | Pointqa-local | 10403 | 0.0273 |
| 291 | Pointqa-looktwice | 22838 | 0.0599 |
| 292 | Sharegpt4o-image | 57289 | 0.1502 |
| 293 | Sharegpt4o-video | 2111 | 0.0055 |
| 294 | Sharegpt4v | 102025 | 0.2675 |
| 295 | Shikra | 5814 | 0.0152 |
| 296 | Slidecaps | 34810 | 0.0913 |
| 297 | Slidevqa | 10341 | 0.0271 |
| 298 | Sroie | 500 | 0.0013 |
| 299 | Stvqa | 18921 | 0.0496 |
| 300 | Svit | 136349 | 0.3575 |
| 301 | Tabfact | 13182 | 0.0346 |
| 302 | Tech4all | 8256 | 0.0216 |
| 303 | Textocr-gpt4v | 25114 | 0.0658 |
| 304 | Tiku/math-4o-zh | 386025 | 1.0121 |
| 305 | Tiku/multidisciplinary-en | 2220078 | 5.8207 |
| 306 | Unimer | 987289 | 2.5885 |
| 307 | VCR | 95036 | 0.2492 |
| 308 | Vflan-4o-zh | 35899 | 0.0941 |
| 309 | VG-rem | 611465 | 1.6032 |
| 310 | Viquae | 1190 | 0.0031 |
| 311 | VQA-cdip-anneal | 753921 | 1.9767 |
| 312 | VQAv2 | 82783 | 0.2171 |
| 313 | WTQ | 1679 | 0.0044 |
|  | Total | 38140772 | 100.0000 |

### B.3 Supervised Fine-tuning Data

During the initial phase of SFT data construction, our objective is to equip the model to handle a broad spectrum of application scenarios. To this end, we develop a model capability taxonomy based on the classification of traditional visual tasks and practical application requirements of vision-language models. Guided by this taxonomy, we sample from open-source multi-task training data to supplement high-quality instruction data. We use a pretrained model to classify the open-source data and extract the required categories. For data where image and question quality are high but answer quality is poor, we re-annotate the answers using a model and perform manual verification. Additionally, based on category guidelines, we generate targeted instruction data for certain categories to further enhance the model’s capabilities. Furthermore, we utilized language data produced by DeepSeek R1 to preserve the MLLM’s linguistic abilities, with the ratio of multimodal to language data being approximately 1:1 at this stage.

**Table 15: Detailed Statistics of SFT Data**
| No. | Dataset Name | Number | Percentage (%) |
| --- | --- | --- | --- |
| 1 | Competition Mathematics | 150000 | 2.4269 |
| 2 | Code | 50000 | 0.8089 |
| 3 | Math | 200000 | 3.2359 |
| 4 | General QA | 300000 | 4.8539 |
| 5 | Tulu Instruction | 19118 | 0.3093 |
| 6 | Multi-turn Conversation | 61114 | 0.9888 |
| 7 | MegaScience | 1250000 | 20.2244 |
| 8 | OpenHermes2-5 | 200000 | 3.2359 |
| 9 | QwenCaps/PixMo/cap | 712401 | 11.5263 |
| 10 | QwenCaps/PixMo/cap-zh | 10000 | 0.1618 |
| 11 | QwenCaps/Allava-Laion | 10000 | 0.1618 |
| 12 | QwenCaps/COCO | 10000 | 0.1618 |
| 13 | QwenCaps/Flickr30k | 10000 | 0.1618 |
| 14 | QwenCaps/Laion-GPT4V | 10000 | 0.1618 |
| 15 | QwenCaps/LLaVA | 10000 | 0.1618 |
| 16 | QwenCaps/LLaVaR-S1 | 10000 | 0.1618 |
| 17 | QwenCaps/OCR-VQA | 10000 | 0.1618 |
| 18 | QwenCaps/SAM | 10000 | 0.1618 |
| 19 | QwenCaps/VG | 10000 | 0.1618 |
| 20 | QwenCaps/Vision-FLAN | 10000 | 0.1618 |
| 21 | DCC | 10000 | 0.1618 |
| 22 | NoahCaps-EN | 10000 | 0.1618 |
| 23 | NoahCaps-ZH | 10000 | 0.1618 |
| 24 | ShareGPT4O-Image | 10000 | 0.1618 |
| 25 | ShareGPT4O-Video | 10000 | 0.1618 |
| 26 | Tech4All | 10000 | 0.1618 |
| 27 | OneVision/CLEVR-Math-MathV360K | 10000 | 0.1618 |
| 28 | OneVision/Super-CLEVR-MathV360K | 10000 | 0.1618 |
| 29 | PixMo/Ask-Model-Anything | 161737 | 2.6168 |
| 30 | PixMo/Ask-Model-Anything-ZH | 10000 | 0.1618 |
| 31 | PixMo/Count | 10000 | 0.1618 |
| 32 | QwenQA/Cauldron/COCOQA | 27022 | 0.4372 |
| 33 | QwenQA/Cauldron/TallyQA | 83671 | 1.3538 |
| 34 | QwenQA/Cauldron/Visual7W | 13006 | 0.2104 |
| 35 | QwenQA/Cauldron/VSR | 1650 | 0.0267 |
| 36 | QwenQA/PixMo/Cap-QA | 247256 | 3.9990 |
| 37 | QwenQA/PixMo/Cap-QA-ZH | 10000 | 0.1618 |
| 38 | QwenQA/Allava-Laion | 484257 | 7.8350 |
| 39 | QwenQA/Allava-VFLAN | 179020 | 2.8965 |
| 40 | QwenQA/AOKVQA | 39358 | 0.6368 |
| 41 | QwenQA/GQA | 69198 | 1.1196 |
| 42 | QwenQA/OKVQA | 3442 | 0.0557 |
| 43 | QwenQA/VQAv2 | 78718 | 1.2736 |
| 44 | QwenQA/Raw/AOKVQA | 10000 | 0.1618 |
| 45 | QwenQA/Raw/GQA | 10000 | 0.1618 |
| 46 | QwenQA/Raw/OKVQA | 10000 | 0.1618 |
| 47 | QwenQA/Raw/VQAv2 | 10000 | 0.1618 |
| 48 | VFLAN-4O-ZH | 10000 | 0.1618 |
| 49 | Cauldron/IconQA | 10000 | 0.1618 |
| 50 | Cauldron/Mimic-CGD | 10000 | 0.1618 |
| 51 | Cauldron/NLVR2 | 10000 | 0.1618 |
| 52 | Cauldron/Raven | 10000 | 0.1618 |
| 53 | Cauldron/Spot-the-Diff | 10000 | 0.1618 |
| 54 | PixMo/Points | 10000 | 0.1618 |
| 55 | COCO-Rec | 130723 | 2.1150 |
| 56 | COCO-Reg | 130723 | 2.1150 |
| 57 | Flickr | 10000 | 0.1618 |
| 58 | Objects365 | 10000 | 0.1618 |
| 59 | Cauldron/IAM | 10000 | 0.1618 |
| 60 | CoSyn-400K/Chart | 10000 | 0.1618 |
| 61 | CoSyn-400K/Chemical | 10000 | 0.1618 |
| 62 | CoSyn-400K/Circuit | 10000 | 0.1618 |
| 63 | CoSyn-400K/Diagram | 10000 | 0.1618 |
| 64 | CoSyn-400K/Document | 10000 | 0.1618 |
| 65 | CoSyn-400K/Graphic | 10000 | 0.1618 |
| 66 | CoSyn-400K/Music | 10000 | 0.1618 |
| 67 | CoSyn-400K/Nutrition | 10000 | 0.1618 |
| 68 | CoSyn-400K/Table | 10000 | 0.1618 |
| 69 | OneVision/IIIT5K | 10000 | 0.1618 |
| 70 | OneVision/K12-Printing | 10000 | 0.1618 |
| 71 | OneVision/LRV-Chart | 10000 | 0.1618 |
| 72 | OneVision/Orand-Car-A | 10000 | 0.1618 |
| 73 | QwenCaps/ChartQA | 18317 | 0.2964 |
| 74 | QwenCaps/Chart2Text | 16605 | 0.2687 |
| 75 | QwenCaps/Chart-to-Text | 44096 | 0.7135 |
| 76 | QwenCaps/DocVQA | 11166 | 0.1807 |
| 77 | QwenCaps/FUNSD | 149 | 0.0024 |
| 78 | QwenCaps/InfoVQA | 4890 | 0.0791 |
| 79 | QwenCaps/OpenCQA | 7724 | 0.1250 |
| 80 | QwenCaps/POIE | 2999 | 0.0485 |
| 81 | QwenCaps/SROIE | 608 | 0.0098 |
| 82 | QwenCaps/TextVQA | 24990 | 0.4043 |
| 83 | QwenCaps/Vistext | 8816 | 0.1426 |
| 84 | QwenCaps/XFUND | 1043 | 0.0169 |
| 85 | QwenQA/Cauldron/Diagram-Image-to-Text | 297 | 0.0048 |
| 86 | QwenQA/Cauldron/Hateful-Memes | 3729 | 0.0603 |
| 87 | QwenQA/Cauldron/TabMWP | 22410 | 0.3626 |
| 88 | QwenQA/Cauldron/TextVQA | 16059 | 0.2598 |
| 89 | QwenQA/ChartQA/Human | 3290 | 0.0532 |
| 90 | QwenQA/ChartQA/Augmented | 12469 | 0.2017 |
| 91 | QwenQA/DocVQA/Human | 8924 | 0.1444 |
| 92 | QwenQA/InfoVQA/Human | 4092 | 0.0662 |
| 93 | QwenQA/STVQA | 14381 | 0.2327 |
| 94 | QwenQA/TabFact | 12648 | 0.2046 |
| 95 | QwenQA/WTQ | 1588 | 0.0257 |
| 96 | QwenQA/Raw/STVQA | 10000 | 0.1618 |
| 97 | QwenQA/Raw/TabFact | 10000 | 0.1618 |
| 98 | QwenQA/Raw/WTQ | 10000 | 0.1618 |
| 99 | XFUND/ZH-QA | 10000 | 0.1618 |
| 100 | CIPC-DocVQA | 10000 | 0.1618 |
| 101 | CoAVQA-Chat | 10000 | 0.1618 |
| 102 | CoAVQA-Bench | 10000 | 0.1618 |
| 103 | E2E-OCR-0324 | 359996 | 5.8245 |
| 104 | ESTVQA | 10000 | 0.1618 |
| 105 | ESTVQA-Grounding | 10000 | 0.1618 |
| 106 | FUNSD-QA | 10000 | 0.1618 |
| 107 | POIE | 10000 | 0.1618 |
| 108 | SlideCaps | 10000 | 0.1618 |
| 109 | SlideVQA | 10000 | 0.1618 |
| 110 | SROIE | 10000 | 0.1618 |
| 111 | VQA-CDIP-SFT | 10000 | 0.1618 |
| 112 | QwenCaps/Screen2Words | 10000 | 0.1618 |
| 113 | Cauldron/VisualMRC | 10000 | 0.1618 |
| 114 | Cauldron/WebSight | 10000 | 0.1618 |
| 115 | CoSyn-Point | 10000 | 0.1618 |
| 116 | Screen/Nav | 10000 | 0.1618 |
| 117 | Screen/Ref | 10000 | 0.1618 |
| 118 | Screen/SQA | 10000 | 0.1618 |
| 119 | Cauldron/GeomVerse | 10000 | 0.1618 |
| 120 | CoSyn-400K/Math | 66707 | 1.0793 |
| 121 | OneVision/Geometry3K-MathV360K | 10000 | 0.1618 |
| 122 | OneVision/GeoQA-Plus-MathV360K | 10000 | 0.1618 |
| 123 | OneVision/Geos-MathV360K | 10000 | 0.1618 |
| 124 | OneVision/IconQA-MathV360K | 10000 | 0.1618 |
| 125 | OneVision/PMC-VQA-MathV360K | 10000 | 0.1618 |
| 126 | OneVision/TabMWP-MathV360K | 10000 | 0.1618 |
| 127 | OneVision/Unigeo-MathV360K | 10000 | 0.1618 |
| 128 | OneVision/Geo3K | 10000 | 0.1618 |
| 129 | OneVision/Mavis-Math-Metagen | 10000 | 0.1618 |
| 130 | OneVision/Mavis-Math-Rule-Geo | 10000 | 0.1618 |
| 131 | CROHME | 10000 | 0.1618 |
| 132 | Geo170K-Alignment | 10000 | 0.1618 |
| 133 | Geo170K-QA-Tuning | 10000 | 0.1618 |
| 134 | HME | 10000 | 0.1618 |
| 135 | M2E | 10000 | 0.1618 |
| 136 | MLHME | 10000 | 0.1618 |
| 137 | Unimer | 10000 | 0.1618 |
| 138 | QwenCaps/AI2D | 9778 | 0.1582 |
| 139 | QwenCaps/ScienceQA | 4985 | 0.0807 |
| 140 | QwenQA/Cauldron/MapQA | 22950 | 0.3713 |
| 141 | QwenQA/Cauldron/TQA | 861 | 0.0139 |
| 142 | QwenQA/AI2D | 10593 | 0.1714 |
| 143 | QwenQA/ScienceQA | 11071 | 0.1791 |
|  | Total | 6180645 | 100.0000 |

## Appendix C Experiments

### C.1 Results of Test-Time Resolution Search

In this Section, we present detail evaluation results with test-time image resolution search strategy.
On MME dataset, as shown in Figure [8](https://arxiv.org/html/2509.11662v3#A3.F8), models trained with different resolution lengths yield consistent evaluation results when min-pixels is less than 16. In addition, down-sampling images on MME results in a significant improvement in model performance.
On OCRBench dataset, as shown in Figure [9](https://arxiv.org/html/2509.11662v3#A3.F9), when min-pixels is set to 16 or 32, the model performs better than when it is set to 4.
The reason is that too small input images are out of distribution in OCR task for our model.
On the other hand, when min-pixels is set to 64, the model performance degrades. The reason is that the image blurring effect caused by up-sampling images has a negative impact on the model’s visual perception.
On DocVQA and InfoVQA datasets, as shown in Figure [10](https://arxiv.org/html/2509.11662v3#A3.F10) and Figure [11](https://arxiv.org/html/2509.11662v3#A3.F11), min-pixels variable has no detectable impact on the evaluation results since most of the images in these two datasets have a resolution higher than min-pixels.
Furthermore, the optimal inference max-pixels on these two datasets falls within the range of 2560–3072.

Figure: Figure 8: Evaluation results on MME of MindVL with different input image resolutions.
Refer to caption: x7.png

Figure: Figure 9: Evaluation results on OCRBench of MindVL with different input image resolutions.
Refer to caption: x11.png

Figure: Figure 10: Evaluation results on DocVQA of MindVL with different input image resolutions.
Refer to caption: x15.png

Figure: Figure 11: Evaluation results on InfoVQA of MindVL with different input image resolutions.
Refer to caption: x19.png

### C.2 Integration of Multitask Data

We also present the experimental results of the model trained with different data recipes. As shown in Table [16](https://arxiv.org/html/2509.11662v3#A3.T16),
MindVL-8B-V1Data model is trained with introduced data of Video, Captions, OCR, GUI, Math, Knowledge, Grounding and Genreal VQA introduced in Section [4.2](https://arxiv.org/html/2509.11662v3#S4.SS2). For MindVL-8B-V2Data model, Internleaved image-text data are added for training.
For MindVL-8B-V3Data model, STEM data and additional Captions, OCR, GUI, Grounding and Genreal VQA data are added for training.
By comparing the results of MindVL-8B-V1Data and MindVL-8B-V2Data, we can observe that adding Internleaved image-text data for training can significantly improve performance over all benchmarks.
It is because that interleaved image-text data not only contains complex logical relationships between text and images, but also covers a wide range of domain knowledge—both of which are highly beneficial for enhancing the multimodal model’s cognitive capabilities.
Moreover, by adding more refined and diverse Captions, OCR, GUI, Grounding and Genreal VQA data to train the model MindVL-8B-V3Data, the performance on multiple benchmark datasets has been further improved, such as MME, MMBench and InfoVQA.

**Table 16: Ablation study results of multitask data.**
| Model | MME | MMBench | OCRBench | DocVQA | ChartQA | InfoVQA | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MindVL-8B-V1Data | 79.8 | 81.5 | 83.3 | 92.1 | 83.7 | 73.2 | 82.3 |
| MindVL-8B-V2Data | 82.4 | 83.4 | 88.2 | 94.7 | 87.2 | 79.9 | 86.0 |
| MindVL-8B-V3Data | 84.1 | 84.3 | 87.6 | 94.7 | 87.2 | 81.1 | 86.5 |

### C.3 Ablation Study on Training Phases

**Table 17: Ablation study results on training phases.**
| Model | MME | MMBench | OCRBench | DocVQA | ChartQA | InfoVQA | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MindVL-8B-4Phase | 80.8 | 82.3 | 83.6 | 92.7 | 86.4 | 76.9 | 83.8 |
| MindVL-8B-2Phase* | 79.4 | 81.8 | 84.9 | 93.5 | 85.2 | 75.8 | 83.4 |
| MindVL-8B-2Phase | 81.6 | 82.4 | 87.0 | 94.2 | 86.8 | 79.6 | 85.3 |
| MindVL-8B-3Phase | 84.1 | 84.3 | 87.6 | 94.7 | 87.2 | 81.1 | 86.5 |

As mentioned in Section [5.2](https://arxiv.org/html/2509.11662v3#S5.SS2), MindVL-8B undergoes a three-phase training pipeline: warm-up, multi-task learning, and SFT.
We denote the model trained on such recipe as MindVL-8B-3Phase.
Table [17](https://arxiv.org/html/2509.11662v3#A3.T17) shows the results of ablation study on training phases.
MindVL-8B-4Phase model is trained on a four-phase training pipeline: warm-up, pretrain, multi-task learning, and SFT.
In the pretrain phase, both ViT and MLP adaptor are trained.
Our ultimate training recipe has only three-phase: warm-up, multi-task learning and SFT, achieves best performance by merging training data in warm-up and pretrain phases of MindVL-8B-4Phase.
By changing the four-phase training recipe to a three-phase training recipe, we have not only improved training efficiency but also enhanced the model performances.
Such results indicate that training a MLP adaptor can also align ViT and LLMs effectively.

We also conducted experiments with two-phase training recipes, MindVL-8B-2Phase* undergoes warm-up and SFT, and MindVL-8B-2Phase undergoes warm-up and multi-task learning with the training data in corresponding phase.
The results show that MindVL-8B-2Phase outperformance MindVL-8B-2Phase*, which indicate that high-quality multi-task data is crucial for enhancing the capabilities of multimodal models.

## Appendix D Examples

We list some samples whose answers are predicted by our MindVL-671B-37B.

[Uncaptioned image]: x23.png

[Uncaptioned image]: x24.png

[Uncaptioned image]: x25.png

[Uncaptioned image]: x26.png

[Uncaptioned image]: x27.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: The paper claims high training efficiency but doesn’t provide concrete hardware utilization statistics (e.g., throughput, FLOPs/s, energy use) to compare with NVIDIA baselines.
soundness: 3
strengths: 1. Efficiency and Transparency: MindVL achieves high performance with far less data while openly detailing its data recipes and training setup, improving reproducibility.

2. Hardware Innovation: It proves that Ascend NPUs can effectively train large multimodal models, expanding hardware options beyond NVIDIA GPUs.

3. Practical Enhancements: Novel methods like model weight averaging and test-time resolution search boost robustness and multimodal alignment at low computational cost.
summary: The paper MindVL introduces an efficient framework (MindSpeed-MLLM) for training multimodal large language models (MLLMs) on Ascend NPUs, challenging the dominance of NVIDIA GPUs. It provides open, reproducible data recipes and introduces two key techniques—model weight averaging and test-time resolution search—to enhance performance. Results show that MindVL-8B matches or exceeds Qwen2.5VL-7B performance while using 10% of its training data, and MindVL-671B-A37B matches Qwen2.5VL-72B performance with only 3% of the data. Both models perform strongly on multimodal benchmarks, demonstrating the viability of Ascend NPUs for large-scale MLLM training.
weaknesses: 1. Limited Architectural Novelty: The model mainly builds on existing designs like Qwen2.5-VL, offering engineering improvements rather than new modeling concepts.

2. Incomplete Reasoning Performance: MindVL underperforms top models on complex multimodal and STEM reasoning tasks.

3. Restricted Accessibility: Dependence on Ascend hardware limits reproducibility and adoption outside that ecosystem.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - Will the training data be open-sourced? While the data recipe is public, the actual data is annotated using closed-source models, making replication challenging for other researchers.

- Could you elaborate on the process for filtering low-quality data and re-annotating it? Specifically, how is the combination of human and model verification utilized? This is a crucial aspect that requires clarification.
soundness: 2
strengths: - This paper provides a training framework that enables the training MLLMs on Ascend NPU hardware.
- The trained MindVL model achieves competitive performance compared to state-of-the-art MLLMs, while using less data.
- The training data recipe is open-sourced.
summary: This paper introduces MindVL, a multimodal large language model trained on Ascend NPUs. Firstly, this work presents MindSpeed-MLLM, a highly efficient training library designed to support the training of MLLMs with both dense and MoE architectures on Ascend NPU hardware. Moreover, the paper introduces MindVL, a data-efficient MLLM, which is further enhanced through multimodal model weight averaging and test-time resolution search techniques. Experimental results demonstrate that MindVL achieves state-of-the-art performance while utilizing less training data.
weaknesses: - Although the data recipe is public, the data itself is annotated using closed-source API models including gemini, gpt, and doubao. This process is costly, making it difficult for other researchers to replicate.

- For data construction, identifying low-quality data and performing re-annotation is crucial. However, the paper lacks a detailed explanation of how this process is specifically conducted.

- The test-time resolution search method lacks comparison with other models. Additionally, it's unclear whether this test-time method can be effectively transferred to other models.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: -	The authors mentioned the data will be open-sourced. However, given the scale of the data, how will it be open-sourced and made publicly accessible? (line 230)
soundness: 2
strengths: -	It seems to be a comprehensive work involving infrastructure, data curation, model tuning and implementation. The technical execution is solid, and the resulted model achieves good performance on multimodal understanding benchmarks.
summary: This paper proposes MindVL, a multimodal LLM trained on Ascend NPUs. To deal with specific challenges on Ascend hardware, they propose an optimized training framework named MindSpeed-MLLM. They provide a new recipe of data construction pipeline for multimodal-LLMs. The resulted model is named MindVL and achieve on-par performance with Qwen2.5-VL.
weaknesses: -	The whole paper looks like a technical report and does not have a core technical contribution. For the training library, the authors provide explanation on data loader, operator fusion, system-level scheduling optimization. However, these explanations are all on high-level, and it seems there is nothing new compared to existing training frameworks. The authors did not explain clearly what part of the Ascend hardware needs special handling for the large-scale training.
-	The data curation part uses proprietary models (qwen, openAI, gemini, doubao) to annotate the data. It is used in multiple stages (filtering, annotation, rewrite), it is hard to know how much contribution of this external knowledge source has on the final performance.  Due to this, it is unfair to directly compare amount of training data with existing models (line 99-101).
-	Some technical details are not clear. 1. What does the author do specifically to support MoE models? 2. For mask compression (line 209-212), how is the memory overhead and computation efficiency improved?

---

confidence: 5
contribution: 1
initial_rating: 2
presentation: 2
questions: None
soundness: 1
strengths: Check summary.
summary: This work presents MindVL, a competitive VLM which is trained with NPUs. The paper presents the data, infra, and training framework for developing this VLM. Additionally, some data statistics and framework features are included within the work.

Overall, you do not need to submit this technical report to ICLR conference. If you would like to make such paper submission, you may need to claim and present your technical contributions. Outstanding VLMs like InternVL and QwenVL are just released on arxiv instead of included in conference proceedings. If you are confident about your work, then publish in tech report to earn your credits. No need to submit it to the conference to earn credits which are not helpful in building your reputation.

To make a valid paper submission, you should at least specify the following details in VLM development: 1) detailed data curation pipeline, which data quality classifier is used, the prompts for model based data curation, the original sources for collecting the data, the human annotator rubric and criteria, the clustering embedding models and sampling algorithm; 2) the online packing implementations and algorithms in Section 3.1.3. Please consider to follow SAIL-VL and OpenQwen2VL to reposition your contributions.
weaknesses: Check summary.