Title: VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision Transformer

Multimodal
Abstract: Conventional Vision Transformer streamlines visual modeling by employing a uniform input resolution, which underestimates the inherent variability of natural visual data and incurs a cost in spatial-contextual fidelity. While preliminary explorations have superficially investigated native resolution modeling, existing works still lack systematic training recipe from the visual representation perspective. To bridge this gap, we introduce Unified Vision Transformer with Native Resolution, i.e. UniViTAR, a family of homogeneous vision foundation models tailored for unified visual modality and native resolution scenario in the era of multimodal. Our framework first conducts architectural upgrades to the vanilla paradigm by integrating multiple advanced components. Building upon these improvements, a progressive training paradigm is introduced, which strategically combines two core mechanisms: (1) resolution curriculum learning, transitioning from fixed-resolution pretraining to native resolution tuning, thereby leveraging ViT’s inherent adaptability to variable-length sequences, and (2) visual modality adaptation via inter-batch image-video switching, which balances computational efficiency with enhanced temporal reasoning. In parallel, a hybrid training framework further synergizes sigmoid-based contrastive loss with feature distillation from a frozen teacher model, thereby accelerating early-stage convergence. Finally, trained exclusively on public accessible image-caption data, our UniViTAR family across multiple model scales from 0.3B to 1B achieves state-of-the-art performance on a wide variety of visual-related tasks. The code and models are available here.

VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution

Abstract

Conventional Vision Transformer streamlines visual modeling by employing a uniform input resolution, which underestimates the inherent variability of natural visual data and incurs a cost in spatial-contextual fidelity. While preliminary explorations have superficially investigated native resolution modeling, existing works still lack systematic training recipe from the visual representation perspective. To bridge this gap, we introduce Uni fied Vi sion T ransformer with N A tive R esolution, i.e. UniViTAR, a family of homogeneous vision foundation models tailored for unified visual modality and native resolution scenario in the era of multimodal. Our framework first conducts architectural upgrades to the vanilla paradigm by integrating multiple advanced components. Building upon these improvements, a progressive training paradigm is introduced, which strategically combines two core mechanisms: (1) resolution curriculum learning, transitioning from fixedresolution pretraining to native resolution tuning, thereby leveraging ViT's inherent adaptability to variable-length sequences, and (2) visual modality adaptation via inter-batch image-video switching, which balances computational efficiency with enhanced temporal reasoning. In parallel, a hybrid training framework further synergizes sigmoid-based contrastive loss with feature distillation from a frozen teacher model, thereby accelerating early-stage convergence. Finally, trained exclusively on public accessible image-caption data, our UniViTAR family across multiple model scales from 0.3B to 1.4B achieves state-of-the-art performance on a wide variety of visual-related tasks. The code and models are available here.

SigLIP-SO400MUniViTAR-0.3BUniViTAR-0.6BUniViTAR-1B

DFN-HAIMv2-HSigLIP-SO400MUniViTAR-0.6B

Image ZeroShot-Classification

Average on All Benchmarks

DocVQA

HRBench4K

InfovQA

OCRBench

OCRVQA

HallusionBench

MMMLI

ImageNet Probing

ScienceQA

AI2D

VizWiz

MathVista

TextVQA

SEEDBench

Image ZeroShot-Retrieval

VideoZeroShot-Retrieval

MME

GOA

1 Introduction

In the era of rapid advancements of multimodal large models, Vision Transformer [1], characterized by its simplicity and scalability, has emerged as a foundational architecture for visual representation learning. Drawing inspiration from transformer-based large language models, conventional ViT usually uniformly converts raw visual data into square aspect ratio and standardized resolution to reduce modeling complexity and simplify the training workflow. While this paradigm simplifies feature extraction and aligns with existing engineering practices, it inherently imposes artificial constraints on real-world visual data by disregarding the inherent variability of natural images.

Recent studies have preliminarily investigated the vision backbone within a native resolution paradigm. FlexViT [2] introduces a flexible ViT architecture featuring dynamical patch size selection in the patch embedding layer, which facilitates smooth variation of token sequence length through parametric scaling. In contrast, NaViT [3] maintains fixed patch size while directly processing native resolution images with varying aspect ratios, where the token sequence length of different images changes dynamically. This approach demonstrates the feasibility and benefits of adopting natural language processing style packing strategies for vision foundational model. Qwen-VL's [4, 5] vision encoder inherits NaViT's core configuration while specifically investigating native resolution impacts from a multimodal large model perspective. While the aforementioned approaches have attracted initial research attention, the field still lacks a comprehensive series of architecture-homologous vision backbones that can simultaneously support native- and fixed-resolution processing, achieve highfidelity feature extraction for both images and videos.

To address this gap, we present the Uni fied Vi sion T ransformer with N A tive R esolution, termed as UniViTAR , a family of vision foundational backbones designed to uniformly process visual modalities (image or video) with native resolution and dynamic aspect ratio. Building upon insights from large language model recent practices and architectural innovations in visual transformers, our approach firstly conduct systematic architectural upgrades to the vanilla ViT paradigm by integrating multiple advanced components: 2D Rotary Position Embedding, SwiGLU activation function, RMSNorm layer, QK-Norm mechanism, and LayerScale module. These modifications collectively establish a more robust architectural foundation compared to conventional implementations. Secondly, we develop a progressive training paradigm with two complementary adaptation strategies: 1) the progressive resolution adaptation strategy employs curriculum learning from fixed low-resolution (e.g., 224) pretraining to native-resolution fine-tuning. Notably, our experiments reveal that the advanced ViT architecture exhibit remarkable adaptability - models pretrained at fixed resolution can efficiently generalize to variable-length visual sequences through limited native resolution tuning. 2) the progressive visual modality adaptation strategy addresses computational challenges in video processing by deferring video data integration to the final training phase. We further demonstrate that alternating image-video training sequences (inter-batch modality switching) significantly outperforms mixed-batch (intra-batch modality mixing) in preserving image understanding capabilities while acquiring temporal reasoning skills. Thirdly, we implement a hybrid training framework combining contrastive learning objectives with distillation techniques. Our primary optimization employs a sigmoid-based contrastive loss [6] for unified image-video representation learning. To accelerate early-stage convergence, we further incorporate feature distillation from a frozen vision teacher model as an auxiliary training objective during initial phases, then gradually phasing out this regularization as the model matures. Finally, through this comprehensive approach trained on public-accessible datasets, we successfully scale a family of vision backbones supporting native resolutions and both visual modalities, with parameter counts ranging from 0.3B to 1.4B. Extensive evaluations demonstrate the effectiveness of our proposed methods.

Specifically, the contributions of our UniViTAR family are summarized as follows:

We introduce a family of homogeneous visual foundation models that support native resolution and unified feature extraction across visual modalities, offering the community a versatile framework for multimodal research.

We develop an efficient and effective progressive training strategy that addresses the computational challenges of native resolution modeling while systematically enhancing the model's image-caption alignment capability.

We train our models with public-accessible datasets, achieve leading performance with limited resources, and observe a trend of performance increasing with parameter scaling.

UniViTAR

RMS-Norm

SwiGLU-FFN

x

Attention Block

1

Attention Block N

…

RMS-Norm

MHA

Q

K

V

RoPE

2

D

RoPE

2

D

Linear

Linear

Linear

RMS

RMS

Masked Multi-Head Self-Attention

Linear

Linear

Linear

RMS-Norm

Linear

Unified Native Patchify

Image

1 ,

Image

2 ,

Video

1

, … ,

Image m

,

Video n

…

2 Method

2.1 UniViTAR: Homologous Visual Foundation Model

2.1.1 Architecture Design

UniViTAR is a Transformer-based encoder model that inherits the original architecture of the conventional Vision Transformer [1] but incorporates the following advanced modifications:

Unified Patchify for Native Image and Video Modality. As illustrated in the Figure 2, given the native input visual data X ∈ R T × C × H × W of any vision modality (image, video), where T = 1 represents image and T > 1 represents video, UniViTAR firstly patchifies X into a series of dynamic length visual patch sequences P = ( N,S ) , where N is the number of patches per image/video and S is the number of pixels per patch. Then a 3 D convolution layer is adopted as the Patch Embedding Layer to consistently convert the above patch sequence into a visual token sequence T = ( N,D ) , where D is the hidden size of the following attention layers.

2D Rotary Position Embedding. Drawing on the architecture designs of language models, the original ViT regards the position information among different visual tokens as a one-dimensional association. In fact, considering that visual data usually has spatial association (row and column) and temporal association (time), the position information between different tokens is usually considered to be multidimensional. Thence we remove the original absolute position encoding and introduce 2 D -RoPE [7] into each subsequent encoder layer to capture the two-dimensional positional information of images. Furthermore, we found that the presence or absence of the class token in the original ViT has almost no effect on model performance. To ensure the consistency of position encoding, we also empirically remove the design of class token.

SwiGLU and RMSNorm. By leveraging the recent advances of LLaMA [8] architecture design for language modeling, UniViTAR incorporates SwiGLU as the feed-forward network (FFN) and replaces all normalization layers with RMSNorm. In addition, we adds an extra RMSNorm to each SwiGLU-FFN for good expressivity and improving the training stability.

Query-Key Normalization. In order to improve the stability of model training, we adopt the QKNorm technique [9, 10] , which applies normalization to the queries and keys before the dot-product attention computation, to directly controls the norm growth of input to the softmax and avoid abnormal attention logits. Note that we still utilize RMSNorm as the norm function to ensure the consistency of the architecture.

2.1.2 Homologous Model Scaling

The UniViTAR family consists of a comprehensive suite of foundational and scratch-train models, encompassing a parameter range from 0.3 to 1.4 billion, i.e. UniViTAR-0.3B/0.6B/1B. The hyperparameters and important information are listed in Table 1 in details.

Stage 1

Stage 2

Stage 3

Stage 4

Train Strategy

DinoV

2

UniViTAR

LLaMA

Distill Loss

Contrastive

UniViTAR

LLaMA

Contrastive

UniViTAR

LLaMA

Contrastive

UniViTAR

LLaMA

Contrastive

Data Modality

Image

Image

Image

Image,

Video

Resolution

224

×

224

224

×

224

Native

Native

Loss Function

Sigmoid, KL

Sigmoid

Sigmoid

Sigmoid

12B

1B

1B

0.6B

2.2 Contrastive Vision-Language Pretrain with UniViTAR

2.2.1 Architecture Design

In general, the acquisition of UniViTAR largely follows the basic training paradigm of CLIP [11]. Specifically, the native-resolution visual input v is encoded into the visual feature space via the UniViTAR encoder to obtain F v ∈ R N v × D v , while the textual input t is projected into the textual feature space through a pretrained LLaMA [12] decoder to obtain F t ∈ R N t × D t . The dynamic-length visual features F v are then uniformly converted into the visual embedding f v ∈ R D v through a global average pooling and the feature corresponding to the <EOS> token in F t is utilized as the textual representation f t ∈ R D t of the input caption. Subsequently, f v and f t are further projected into the same shared semantic space via a linear projection layer respectively. Then a simple pairwise sigmoid loss [6] is employed as the contrastive supervision to align the visual and text modalities semantically.

2.2.2 Optimized Contrastive Training Strategy

To ensure that the model can converge efficiently and the training cost is controllable, we carefully design the training pipeline of UniViTAR into four stages in sequence, as shown in Table 2.

Stage 1: Visual knowledge pre-acquisition with hybrid paradigm training. The primary objective of this phase is to efficiently pretrain a visual foundation model from scratch by integrating two classic learning paradigms: vision-text contrastive learning and visual knowledge distillation. Specifically, the proposed architecture employs a triple-branch parallel design: (1) a xavier -initialized UniViTAR, (2) a frozen pre-trained text encoder, and (3) a frozen pre-trained visual teacher. During training, only the target visual foundation model receives gradient updates, with other branches fixed to minimize computational overhead while preserving knowledge integration. For implementation, we adopt LLaMA [12] and DINOv2-g [13] as default components, though the framework supports substitution with alternative pre-trained foundation models. The composite training objective is defined as:

where L contrastive is the sigmoid loss from SigLIP [6] and L distillation is the KL Divergence [14]. The target visual foundation model functions as a visual knowledge bridge, simultaneously performing image-text alignment and feature distillation. This phase processes 12B samples with all images resized to 224, constituting 82.2% of the total training data (12B/14.6B).

Stage 2: Finetune with full-parameter for superior alignment. The objective of this stage is to further enhance the upper limit of image-text alignment through full-parameter fine-tuning of both vision and text encoders, establishing a unified semantic-visual space. The visual distillation branch is deactivated during optimization. Training employs identical image-caption pairs as Stage 1 at 224 resolution. Considering the high computational cost of full parameter fine-tuning, the training process is conducted on 1B samples, accounting for 6.9% of the total training data.

Stage 3: Unlock the model-capacity of native-resolution. In this stage, our strategy extends the model capability to handle native-resolution, thereby achieving robust image-text alignment for dynamic-resolution inputs. However, enabling native-resolution capacity necessitates addressing two critical challenges: (1) ensuring positional encoding are thoroughly trained across variable sequence lengths, and (2) transfering feature distribution from uniformly-resized patches to native patches through efficient training. In practice, visual data is batched in its native form to preserve original resolutions and aspect ratios. Then the intra-batch images are dynamically scaled (with aspect ratios maintained) to align total sequence lengths L total with a predefined token limit L max . That is to say, when the value of L max /L total is greater than 1, all data will be uniformly enlarged, and vice versa the shape of all data will be reduced, ensuring consistent computational loads across batches. Within attention blocks, each token's receptive field is confined to tokens from the same image via masking, enabling isolated intra-image contextual modeling while preserving inter-sample independence. During training, resolution diversity within batches ensures comprehensive training of positional encoding across varying context lengths, progressively refining the model's ability to generate features aligned with native patch distributions. At inference, inputs are processed directly at their native resolutions without resizing. In this stage, 1B samples (6.85% of the total training) were trained with the text branch frozen throughout the process.

Stage 4: Unifying visual modalities with image-video alternation training. The goal of this stage is to unify image and video input modalities with native-resolution and dynamic video length. Inspired by the InternVideo series [15, 16], we utilize both image-text and video-text pairs to optimize the UniViTAR checkpoint from Stage 3 with an image-video alternating training strategy. This strategy addresses three critical considerations: (1) leveraging image data's superior scale and diversity compensates for video data scarcity while maintaining visual content continuity; (2) joint image-video training preserves cross-modal comprehension capabilities; (3) alternating modalityspecific updates enforce focused parameter adaptation through sequential modality optimization. The alternating training protocol first initiates each epoch with random permutation of imagevideo data to enhance stochasticity. Subsequently, data batches are partitioned into global batch units and alternately sequenced at global batch granularity. Through this structured approach, the configuration effectively maintains modality purity within individual training batches by enforcing strict image-video alternation. To accommodate native-resolution video processing with dynamic lengths, we implement adaptive frame sampling: full temporal retention when frame count F < F max , and uniform subsampling to F max frames when exceeded. With the predefined token constraints ( L min , L max ) and the calculated frame length F , all frames are subsequently resized within these computed bounds while preserving original aspect ratios.

2.3 UniViTAR as a Vision Encoder for MLLMs.

In this section, we introduce a simple strategy for constructing an effective native resolution MLLM based on the UniViTAR series. The common and industry-validated Vision-Language Models (VLMs) paradigm typically combines pretrained visual backbones with large language models, followed by multimodal training on a rich mixture of vision language tasks. To ensure fair comparison and minimize bias, we adhere to this established configuration. Specifically, we employ UniViTAR as the vision encoder and employ Qwen2.5-1.5B [5] as the large language model. Following established practices [17], we implement a three-layer MLP with pre-normalization and a 2 × pixel-unshuffle operation [18] along the width dimension as the vision-language adapter to bridge the visual and linguistic modalities. For native-resolution modeling, we identify two primary challenges. On one hand, due to the varying lengths of input samples, the boundary between vision and language tokens is not fixed. To enhance "modality isolation", we introduce specialized prompts, known as Boundary Markers , such as <image_start> and <image_end> , at the beginning and end of the vision token sequence. On the other hand, 2D-to-1D flattening of vision tokens may compromise the information of the height-width ratio. To mitigate this, we incorporate Line Anchors , such as <lineidx >, into the vision tokens, where idx denotes the corresponding vertical positions in the original patchified image, thereby potentially strengthening positional awareness in compressed tokens. For a vision token sequence of length hw , the original arrangement x 1 , 1 , ..., x 1 ,w , ..., x 2 ,w , ..., x h,w is transformed as:

Notably, these added markers are string-based identifiers rather than special tokens of the tokenizer. To systematically evaluate multimodal comprehension capabilities, we adopt a dual-stage training paradigm motivated by established methodologies in vision-language alignment like [19, 20].

3 Experiments

3.1 Training Recipe

Data Details. We collect public accessible image-text pairs and build our Merged-1B dataset, which is composed of DataComp-1B [21], COYO [22], LAION-2B [23], LAION-400M [24], DFN-2B [22], CC12M [25] and CC3M [26]. Moreover, to further enhance the video feature extraction capabilities of UniViTAR, we meticulously constructed a dataset Merged-65M of roughly 65 million samples by randomly selecting video clips from three public accessible video datasets, i.e. , Panda-70M [27], WebVid-10M [28], and InternVid-10M-FLT [29]. We refer to the combined image and video data mentioned above as Merged-1.1B. The detailed data composition is summarized in the Appendix.

Hyperparameter Details. The detailed hyperparameter configurations for each training stage are presented in the Appendix. As tabulated, we utilize a progressive reduction of the peak learning rate in correlation with increasing visual backbone scale to ensure optimal training stability. Notably, the learning rate of text branch in Stage 2 remains consistently one-tenth of the visual component throughout this phase. To enhance training efficiency, we integrated the DeepSpeed library [30] by employing ZeRO optimizer sharding [31], gradient checkpointing [32], and flash attention [33].

3.2 Results on Zero-shot Image Classification & Retrieval

Evaluation Setup. Our evaluation protocol encompasses both zero-shot classification and crossmodal retrieval tasks. For zero-shot classification, we conduct evaluation on ImageNet [34] and its established variants [35, 36, 37, 38, 39]. Each class is represented by multiple text prompts curated from [11, 40]. The Top-1 accuracy is utilized to evaluate the model performance. For cross-modal retrieval assessment, we adopt the benchmark protocols defined in [41], evaluating on Flickr [42] and MS-COCO [43] using their official partitions. The retrieval paradigm involves bidirectional image-text matching, namely image-to-text retrieval and text-to-image retrieval tasks.

Results Comparison and Analysis. Table 3 demonstrates the exceptional performance of our model at comparable parameter scales. As the model size increases from 0.3B to 1.4B, the average zero-shot classification accuracy across six benchmarks exhibits a progressive improvement trend, rising from 80.5% to 81.9% and further to 83.4%. Notably, all models of varying scales employ identical training samples and strategies, with this performance enhancement attributed to parameter scaling effects-a finding consistent with established scaling laws in transformer-related research. As detailed in the table, our UniViTAR-1B shows superior performance despite utilizing a smaller training corpus, outperforming its counterparts with more parameters, such as InternViT-6B [12] and EVA-8B [44]. We posit that this advantage stems from two key factors: optimized model atchitecture and training strategy, and preservation of native input resolution, which generates higher-quality visual tokens.

3.3 Results on Zero-shot Video Classification & Retrieval

Evaluation Setup. We evaluate the zero-shot video classification performance on three popular benchmarks as K-400 [50], UCF-101 [51] and HMDB51 [52], using the class names as text prompts. Also, we evaluate the zero-shot video-text retrieval performance on ActivityNet [53], MSR-VTT [54] and MSVD [55]. Following [15, 16], for each video in the 1K version of the test split, we sample one sentence from every set of 20 sentences for MSR-VTT. Following [56], we concatenate the multiple descriptions to form a paragraph and perform a paragraph-to-video retrieval on ActivityNet. All videos are sampled with a dynamic frame rate, with each frame dynamically resized to maintain the original aspect ratio while ensuring the total token within the range of 576 to 16,384.

Results Comparison and Analysis. Table 4 shows the performance of our UniViTAR series models on video benchmarks across comparable parameter scales. As the model size scales from 0.3B to 1B, UniViTAR exhibits consistent performance gains on video benchmarks, with average zero-shot classification metrics improving from 68.0 to 69.0. When compared to models trained on imagecaption data under similar parameter scales, UniViTAR achieves notable improvements. These advancements can be attributed to two key design choices: (1) preserving the aspect ratio of each frame to retain the original semantic information of visual content, and (2) employing dynamic video frame sampling to effectively capture detailed temporal information. However, when compared to the models trained exclusively on video-caption data, UniViTAR still has room for improvement compared to some of the latest models [57, 58, 16], as shown in the Table 4 with gray color.

3.4 Results on Image Classification by Linear Probing

Following common prectices [12, 61], we assess the performance of UniViTAR family as off-theshelf backbones on image classifications. Specifically, we train a linear classifier on the last feature layer with a frozen backbone on ImageNet-1K [34] and evaluate the performance on the validation set and other ImageNet variants [62, 35, 36, 37, 38]. In addition, we also report the classification performance with attentive probing setting as used in [61], which adopts a cross-attention layer with random initialized queries. Table 5 represents the downstream classification performance of our models. First, as the model size increases, the average performance across six benchmarks demonstrates consistent improvement. Second, we observe that the attentive probing performance shows stable improvements over linear probing. Furthermore, compared to public methods, our UniViTAR family shows superior performance across various parameter scales.

3.5 Results on Dense prediction.

In this section, we evaluate the dense prediction performance of our UniViTAR family by transferring to semantic segmentation. Following [12, 63], we fine-tune a decoder with freezing backbones under two different structures, i.e. , Linear and UperNet. Linear decoder transforms the dimension of one single layer visual feature to number of semantic classes, while the UperNet decoder employs PPM and FPN to integrates multi-scale features. Experiments are conducted on the ADE20K [64] dataset. In terms of data preprocessing, we employed the same fixed-resolution input and data augmentation strategies as those used in InternViT [12]. Corresponding results are shown in Table 6. We can observe a performance gap between these two types of decoder, this can be understand that UperNet has significantly more trainable parameters than Linear decoder. Taking UniViTAR-0.6B as an example, Linear decoder has a parameter count of 0.2M, whereas UperNet contains approximately 200M parameters. Notably, our UniViTAR Family demonstrates an obvious performance advantage compared with existing state-of-the-art vision encoders. Under the setting of Linear decoder, our UniViTAR-1B achieves a performance of 45 . 4 mIoU, which is +6 . 1 points over OpenCLIP-G [45] and +10 . 8 points over ViT-22B [10]. In the case of UperNet decoder, our UniViTAR-1B reaches 56.2 mIoU, also surpassing larger parameter-scale model like InternViT-6B [12].

3.6 Results on Multimodal Understarding

Evaluation Setup. To assess the potential of multimodal understanding, we employ a dual-stage training paradigm, similar to common practices [19, 20]. In the pretraining stage, we train the projector with a learning rate of 1 e -3 using a merged 2 . 5 Mdataset comprised of LLaVA-CC3MPretrain [17], ALLaVA-Caption [19], ShareGPT4V-PT [20]. In the fine-tuning stage, we unfreeze the whole model, and train it with a learning rate of 1 e -5 , using the high-quality instrution-tuning

dataset LLaVA1.5-Finetune [65]. Note that the native-resolution strategy of Boundary Markers and Line Anchors are only applied in the fine-tuning stage. All evaluations are conducted using VLMEvalKit [66], assessing performance across 16 popular benchmarks, including GQA [67], DocVQA [68], InfoVQA [69], ScienceQA [70], TextVQA [71], VizWiz [72], OCRVQA [73], OCRBench [74], MME [75], MMMU [76], SEEDBench_IMG [77], MathVista_MINI [78], AI2D [79], HallusionBench [80], POPE [81], HRBench4K [82].

Results Comparison and Analysis. As illustrated in Table 7, under exactly the same training data and training strategy, the proposed UniViTAR surpasses various state-of-the-art vision encoders [47, 61, 6] on numerous multimodal understanding benchmarks. Notably, UniViTAR demonstrates exceptional capabilities in scenarios involving dense information, such as document parsing [68], graphic parsing [69], and high-resolution tasks [82]. We argue that the native resolution plays a crucial role in achieving outstanding performance in these areas, ensuring minimal loss of image information. We also assess the effectiveness of the proposed strategy of Boundary Markers and Line Anchors on 0.6B model size, as demonstrated in Table 7, which highlights their impact.

3.7 Ablation Study

3.7.1 Robustness verification of resolution mode.

This section analyzes the influence of three resolution modes: fixed resolution, native aspect ratio, and native resolution in Figure 3. For fixed resolution mode, we resize the shorter edge of each image to a predefined size S and apply CenterCrop to ensure the sequence length strictly equals ( S/ 14) 2 , where 14 represents the patch size. Increasing S proportionally extends the sequence length. In native aspect ratio mode, we scale images while preserving their original aspect-ratio, ensuring that wh/ 14 2 approximates the target sequence length. We evaluate 12 sequence lengths ranging from 256 to 16K tokens with our 0.6B model. The experimental results reveal three key findings: 1) performance initially improves then declines with increasing sequence length under both fixed and native aspect ratio modes, peaking at 1024 ∼ 4096 tokens. 2) native aspect ratio consistently outperforms fixed resolution, indicating that preserving original aspect ratios retains better image information. 3) native aspect ratio occasionally surpasses native resolution performance at certain lengths.

3.7.2 Verification of the effectiveness of training strategies.

As introduced in the method, we categorize the training strategy into four distinct stages, based on resolution modes (fixed or native), visual data modalities (image or video), and model training parameters (trainable or freeze). As shown in Figure 4, for zero-shot image classification (left), we show that S1, S2, and S3 exhibit progressive performance improvements, while S4 maintains comparable accuracy despite incorporating alternating training. In contrast, for zero-shot video classification (right), S1 and S2 show minimal performance variation, with dynamic-resolution

Zero-ShotPerformanceonImageNet-1KwithDifferentResolutionMode

Zero-ShotPerformanceonImageNet-AwithDifferentResolutionMode

8

NativeAspectRatio

FixedSquareSize

90

NativeAspectRatio

Fixed Square Size

82

81

81.281.2

82.0g1.9

81.9

82.3

82.0

82.4

81.8

82.3

81.8

_82.2-==.8.2.2

81.7

81.6

81.9

81.3

81.5

80.8

608

85

83.4 83.3

85.2 85.0

87.1

85.8"*

87.9

84.

88.1

86.2--060-

88.2

87.5

85.2

83.7

84.1

80.9

82.1

80

80.0

80.2

80

77.2

78.6

79.4

79.2

75

73.5

75.4

79

256

576

729

1024

1600

2304

2916

4096

6400

9216

12544

16384

256

576

729

1024

1600

2304

2916

4096

6400

9216

12544

16384

training in S3 significantly boosting video capabilities, followed by further enhancements in S4 through image-video training. This demonstrates that dynamic-resolution training enables model to process more native visual sequences, while the final unified training stage equips the model with generalized capabilities for handling diverse visual modalities. Notably, these findings remain highly consistent across the UniViTAR-0.3B/0.6B/1B model family.

Zero-ShotClassificationPerformanceonImageBenchmarks atDiffenentTrainingStages

Zero-Shot ClassificationPerformance onVideoBenchmarks atDiffenentTrainingStages

70

68.0

68.6

68.8

84

83.683.6

65

82

82.5

82.1

81.8

82.3

60

59.3

80.6 80.6

80.6

55.7

57.6

80

79.7

55

78.7

50.750.3

50.950.5

78

77.8

50

48.648.6

UniViTAR-O.3B

UniViTAR-O.6B

UniViTAR-1B

UniViTAR-O.3B

UniViTAR-O.6B

UniViTAR-1B

S1

S2

S3

S1

S2

S3

3.8 Verification of the effectiveness of hybrid training with DINOv2.

To investigate the performance benefits of incorporating DINOv2 as a distillation branch in Stage-1 training, we performed a comprehensive empirical study using the UniViTAR-0.3B model trained from scratch on 3B image-text pairs. Checkpoints were evaluated at regular intervals throughout training in Table 8. Our experiments revealed that DINOv2 distillation significantly accelerates early-stage convergence, after only 0.1B samples, distillation improved zero-shot ImageNet-1K performance by 17.3 points. Although this gain gradually diminishes as training progresses, it remains observable at later stages. More importantly, the final model trained with DINOv2 distillation achieves an average improvement of 2.1 points across six zero-shot classification benchmarks compared to the baseline without distillation (Table 9). These results demonstrate that DINOv2 distillation not only speeds up early convergence but also enhances the final model performance.

4 Conclusion

In this work, we introduce UniViTAR, a family of homogeneous vision foundation models tailored for unified visual modality and native-resolution scenarios in the era of multimodal. By integrating advanced architectural upgrades, resolution curriculum learning, visual feature distillation, and inter-batch modality adaptation, UniViTAR achieves significant improvements across diverse tasks, spanning image/video zero-shot classification/retrieval, dense prediction accuracy, and vision-language model transfer performance. Notably, all models are trained exclusively on public-accessible datasets, where we observe consistent performance gains with parameter scaling from 0.3B to 1.4B. We hope that our UniViTAR offers the community a versatile framework for advancing multimodal research.

References

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14496-14506, 2023.

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n'pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems , 36:2252-2274, 2023.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11975-11986, 2023.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 , 2024.

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning , pages 74807512. PMLR, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24185-24198, 2024.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Thomas M Cover. Elements of information theory . John Wiley & Sons, 1999.

Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 , 2022.

Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision , pages 396-416. Springer, 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS , 2023.

Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient subpixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684 , 2024.

Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 , 2023.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems , 36:27092-27112, 2023.

Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS , 35:25278-25294, 2022.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , pages 3558-3568, 2021.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL , 2018.

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13320-13331, 2024.

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision , 2021.

Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 , 2023.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In SIGKDD , pages 3505-3506, 2020.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-16. IEEE, 2020.

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memoryefficient exact attention with io-awareness. NeurIPS , 35:16344-16359, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15262-15271, 2021.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV , 2021.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML , pages 5389-5400, 2019.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in neural information processing systems , 32, 2019.

Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. NeurIPS , 32, 2019.

Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR , pages 18123-18133, 2022.

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3128-3137, 2015.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics , 2:67-78, 2014.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755, 2014.

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. Zenodo. Version 0.1. https://doi.org/10.5281/zenodo.5143773 , 2021. DOI: 10.5281/zenodo.5143773.

Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671 , 2023.

Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425 , 2023.

Xianhang Li, Zeyu Wang, and Cihang Xie. Clipa-v2: Scaling clip training with 81.1% zero-shot imagenet accuracy within $10000 budget; an extra $4000 unlocks 81.8% accuracy. arXiv preprint arXiv:2306.15658 , 2023.

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision , pages 2556-2563. IEEE, 2011.

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition , pages 961-970, 2015.

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR , pages 5288-5296, 2016.

David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies , pages 190-200, 2011.

Mamshad Nayeem Rizve, Fan Fei, Jayakrishnan Unnikrishnan, Son Tran, Benjamin Z Yao, Belinda Zeng, Mubarak Shah, and Trishul Chilimbi. Vidla: Video-language alignment at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14043-14055, 2024.

Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979 , 2022.

Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. arXiv preprint arXiv:2402.13217 , 2024.

Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3042-3051, 2022.

Ziyun Zeng, Yixiao Ge, Zhan Tong, Xihui Liu, Shu-Tao Xia, and Ying Shan. Tvtsv2: Learning out-ofthe-box spatiotemporal visual representations at scale. arXiv preprint arXiv:2305.14173 , 2023.

Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal autoregressive pre-training of large vision encoders. arXiv preprint arXiv:2411.14402 , 2024.

Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159 , 2020.

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICML , pages 7480-7512, 2023.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR , pages 633-641, 2017.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 , 2023.

Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multimodality models. In Proceedings of the 32nd ACM International Conference on Multimedia , pages 11198-11201, 2024.

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2200-2209, 2021.

Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV , pages 1697-1706, 2022.

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR , pages 8317-8326, 2019.

Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR , pages 3608-3617, 2018.

Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR , 2019.

Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences , 67(12):220102, 2024.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 , 2023.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9556-9567, 2024.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 , 2023.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 , 2023.

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 235-251. Springer, 2016.

Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14375-14385, 2024.

Kun Zhou Jinpeng Wang Wayne Xin Zhao Yifan Li, Yifan Du and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. arXiv preprint , 2024.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representation learning. arXiv preprint arXiv:2504.01017 , 2025.

Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems , 37:87310-87356, 2024.

Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. Inception transformer. arXiv preprint arXiv:2205.12956 , 2022.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 , 2023.

Qihang Fan, Quanzeng You, Xiaotian Han, Yongfei Liu, Yunzhe Tao, Huaibo Huang, Ran He, and Hongxia Yang. Vitar: Vision transformer with any resolution. arXiv preprint arXiv:2403.18361 , 2024.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PmLR, 2020.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19175-19186. IEEE Computer Society, 2023.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PmLR, 2021.

Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. arXiv preprint arXiv:2209.06430 , 2022.

Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with cross modal sharing. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 1236-1248, 2024.

openai. Language models are unsupervised multitask learners. https://openai.com/index/ hello-gpt-4o/ , 2024.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 , 2024.

Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886 , 2023.

Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 , 2024.

Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision , pages 390-406. Springer, 2024.

Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 , 2024.

Yuan Liu, Le Tian, Xiao Zhou, Xinyu Gao, Kavio Yu, Yang Yu, and Jie Zhou. Points1.5: Building a vision-language model towards real world applications, 2024.

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models, 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes. Our main contributions are also detailed in Sec. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We analyzed the performance gap with the pure video model in the experiment section. In addition, the reduction in inference efficiency caused by native resolution is also mentioned in the article.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Each experiment in the paper provides a detailed description of its setting, referring to Sec. 3 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use publicly-accessable image-caption dataset. Once the blind review period is finished, we'll open-source code and model checkpoints.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training hyperparameters are introduced in Table 11, and the test details are presented along with the experimental results in Sec. 3 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: This topic has not been reported with experimental statistical significance in other works.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the training resources in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully reviewed the guidelines to ensure that our research strictly adheres to ethical standards.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This research primarily focuses on fundamental network structures, with no potential social harm.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not foresee any high risk for misuse of work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have carefully reviewed the guidelines and ensured adheres to the standards.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This research does not involve crowdsourcing, human subjects, or other related risks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This research does not involve crowdsourcing, human subjects, or other related risks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: We have carefully reviewed the guidelines and ensured adheres to the standards.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A The Overview of UniViTAR Pipeline

The proposed UniViTAR processes visual input at its native resolution, and also supports scaling the resolution down or up while maintaining the aspect ratio or resizing to certain square size to accommodate different application scenarios, such as higher computational efficiency or finer-grained visual details. By treating video inputs as temporally extended image sequences, the framework uniformly produces longer variable-length visual token sequences.

Video

Native Resolution

Native Aspect Ratio

Square Resize

UniViTAR Family

Native Resolution

Native Aspect Ratio

zoom out

zoom in

864

Patches

384

Patches

256

Patches

4320

Patches

480

Patches

960

Patches

864

Tokens

384

Tokens

256

Tokens

4320

Tokens

480

Tokens

960

Tokens

336

224

336 504 B More Details of Training Recipe

Data Details. We collect public accessible image-text pairs and build our Merged-1B dataset, which is composed of DataComp-1B [21], COYO [22], LAION-2B [23], LAION-400M [24], DFN-2B [22], CC12M [25] and CC3M [26]. Moreover, to further enhance the video feature extraction capabilities of UniViTAR, we meticulously constructed a dataset Merged-65M of roughly 65 million samples by randomly selecting video clips from three public accessible video datasets, i.e. , Panda-70M [27], WebVid-10M [28], and InternVid-10M-FLT [29]. We refer to the combined image and video data mentioned above as Merged-1.1B. The detailed data composition is summarized in the Table 10.

Hyperparameter Details. The detailed hyperparameter configurations for each training stage are presented in the Table 11. As tabulated, we utilize a progressive reduction of the peak learning rate in correlation with increasing visual backbone scale to ensure optimal training stability. Notably, the learning rate of text branch in Stage 2 remains consistently one-tenth of the visual component throughout this phase. To enhance training efficiency, we integrated the DeepSpeed library [30] by employing ZeRO optimizer sharding [31], gradient checkpointing [32], and flash attention [33]. Note all experiments are conducted on H800 GPUs.

C More Experimental Results & Ablation Study

C.1 Verification of the effectiveness of image-video alternative strategy.

To validate the efficacy of the alternating image-video training strategy, we conducted initial experiments with 100M image-text pairs and 10M video-text pairs. Note that the image-to-video data ratio is approximately 10:1, consistent with the ratio used in stage 4 of the UniViTAR series. We trained a UniViTAR-0.3B model for 3 epochs, comparing mixed training and alternating training strategies. As shown in Table 12, the alternating training strategy outperforms the mixed strategy across key image and video benchmark metrics, demonstrating its effectiveness in enhancing visual representation learning. This performance gain can be attributed to the increased training difficulty arising from the unification of data modalities within each batch.

C.2 Verification of the effectiveness of native resolution for video.

In this section, we conduct an ablation study to explore the role of native resolution in video data processing. We dynamically sample a maximum of 32 frames (denoted as F ) for each video clip. For frames exceeding the sequence length limit, we resize them while preserving their native aspect ratio to a smaller resolution. We evaluate 15 maximum video sequence length, ranging from 1024 to 65,536, and test the zero-shot classification performance of UniViTAR-0.6B on the K400 dataset. Note that the minimum video sequence length is fixed to 576. As shown in Figure 6, the performance initially improves and then stabilizes as the sequence length limit increases, reaching a plateau at length 10,240. We attribute this to the fact that, with 32 sampled frames, a sequence length of 10,240 corresponds to a resolution of 490 × 256 , enabling most frames in K400 to retain their native resolution during data processing. This finding underscores the importance of native resolution in enhancing video understanding capabilities.

C.3 Verification of the effectiveness of data scale.

From an intuitive perspective, data scale has a significant impact on the effectiveness of contrastive learning. In this section, we conduct cold-start experiments on UniViTAR-0.3B to confirm this view. For the experiment setup, seen samples is fixed at 1B. We respectively train the UniViTAR-0.3B for 1 epoch using Merged-1B and for 10 epochs using Merged-100M, which contains 100M image-text pairs that randomly sampled from Merged-1B. Result on zero-shot classification and retrieval is

70

99.73

98'66

99.92

99.93

66°66

66°66

66°66

66'66

100.00

100.00

100

68

66.78

67.51

67.67

67.69

67.68

67.68

67.69

67.69

67.68

67.70

67.69

67.68

67.71

80

66

60

64

64.11

47.87

40

62

60

59.31

11.41

12.37

20

58

0.00

0.00

0

1024

2048

4096

6144

8192

10240

12288

14336

16384

18432

20480

24576

28672

32768

65536

MaximumSequence LengthPer Video

Zero-Shot Classification Accuracy on K400

NativeFrameKept Ratio on TotalVideo Clips

shown in Table 13. There is an observable trend where the performance improves as the dataset scale increases. With larger dataset scale, the model is exposed to a broader range of image-text pairs, facilitating a more comprehensive learning and understanding of the visual and linguistic space, thereby enhancing zero-shot performance.

C.4 Verification of the effectiveness of LLM scale.

To verify the model's effectiveness across language models of varying scales, we integrated UniViTAR-0.6B with progressively larger language backbones, specifically, Qwen2.5-1.5B, 3B, and 7B, within a multimodal large language model (MLLM) framework. As the model scale increases, our models achieve average scores of 48.7, 51.9, and 54.6, respectively, across 16 multimodal benchmarks. The experimental results demonstrate consistent scaling behavior, confirming UniViTAR's strong compatibility and performance potential when combined with larger language models. These findings support the conclusion that the UniViTAR architecture possesses promising scalability.

C.5 Comparative analyses of additional relevant visual foundation models.

To provide a comprehensive comparison with other prominent visual encoders, we conduct a systematic analysis of several relevant visual foundation models. (1) NaViT [3]: Quantitative evaluations in Table 14 on linear-probing classification tasks demonstrate that UniViTAR-0.3B exhibits clear advantages over NaViT. (2) FlexiViT [2]: Introduced in the main text, FlexiViT supports dynamic patch size to handle variable-resolution inputs. We include supplementary linear-probing results on ImageNet variants in Table 14 for direct comparison. (3) Web-DINO [84]: This self-supervised model shows that scaling data and parameters can approach CLIP-level performance; however, a noticeable gap remains relative to CLIP-based paradigms, as indicated by model size (7B vs. 0.3B) and benchmark performance. Preliminary comparisons are provided in the accompanying Table 14. (4) Cambrian-1 [85]: This vision-centric MLLM family uses a Mixture-of-Features (MoF) scheme over multiple visual encoders to reduce information loss. While effective, MoF introduces higher computational costs and integration complexity compared to unified models like UniViTAR.

D Related Work

D.1 Flexible Vision Transformers

Vision Transformers have showcased impressive performance in numerous visual tasks, such as image classification [1], image language pre-training [11], etc. Those methods work only at a single, fixed resolution. Some works [86, 87] attempt to meet the need for fine-grained visual representation by adapting the model to a higher resolution during the fine-tuning stage. However, directly resizing the input to a fixed square resolution still limits their representation capacity in diverse visual scenarios. Recently, there are some works in vision transformers attempting to accommodate images with native resolutions with variable aspect ratios. ViTAR [88] proposes an adaptive token merger module to alleviate the constraints of fixed resolution and adapt to multi-resolution inputs. However, it still limited by a predefied number of tokens that the model ultimately aims to obtain. NaViT [3] introduces sample packing used in language modeling for handling variable sequence length of image patches. Meanwhile, it introduces a factorized positional embedding schema in vanilla ViT to support variable aspect ratios and extrapolate to unseen resolutions. Qwen2.5-VL [5] integrates an NaViT-like approach to support native input resolutions, and employs multiple training phases for adapting it to multimodal large languages models, including CLIP pre-training, vision-language alignment, etc.

D.2 Vision Foundation Models

The development of vision foundation models has progressed through distinct phases, beginning with supervised learning paradigm dominated by landmark architectures like ResNet [89] and ViT [1], which established performance benchmarks through reliance on labeled data. However, the field witnessed a paradigm shift with the rise of self-supervised learning, which circumvented annotation bottlenecks through three principal branches: contrastive learning frameworks like SimCLR [90] and MoCo [91], masked image modeling methods such as BEiT [92] and MAE [93], and self-distillation techniques including BYOL [94] and DINO [95]. Recently, language-supervised contrastive pretraining has emerged as a transformative paradigm, exemplified by CLIP [96], which aligns multimodal embeddings through noise-robust contrastive objectives, enabling zero-shot task generalization. This approach has been further refined in works like SigLIP [6], which employs a more efficient sigmoid-base loss function while preserving cross-modal transfer capabilities. Besides images, a robust visual foundation model with effective video alignment capabilities serves as another critical building block. The existing strategies for training such models can be classified into three main paradigms: training on video-only data [56, 58, 15, 97], utilizing multimodal data encompassing both video and image [15, 87, 4, 5], and incorporating multimodal data that integrate video, images, audio and other modalities [16, 98]. VideoPrism [58] employs a two-stage video-only pretraining strategy: contrastive learning followed by token distillation, yet lacks image understanding. VidLA [56] adapts CLIP [96] via spatio-temporal attention on video-text data. InternVideo [15] combines masked video modeling with alternating video/image-text pretraining, enhanced by cross-modal attention, while InternVideo2 [16] extends this framework with audio/speech modalities for multimodal alignment.

D.3 Multimodal Large Language Models

Recently, multimodal large language models (MLLMs) have witnessed significant advancements and rapid development [99, 17, 100, 87, 5, 4, 12, 101, 102]. As a critical modality in MLLMs, visual input encounters inherent limitations when relying on conventional ViT with fixed resolutions, which may induce shape distortions, content blurring, and suboptimal handling of images/videos with diverse aspect ratios, high resolutions, or dynamic frame rates. To mitigate these challenges, the field has converged on two principal technical directions: 1) The tiling-based paradigm, as adopted by models like [103, 100, 12, 104], decomposes ultra-high-resolution inputs into a varied number of fix-resolution tiles, and each tile is processed by a fixed-resolution vision encoder. As such, it enables MLLMs adaptivity to dynamic-resolution images without padding or shape-distorting resizing. However, the tile limits the model's ability to capture spatial information across different tiles and the primary subjects of the images are often fragmented, leading to the loss of spatial relationships and quantitative information. 2) native-resolution methodology, exemplified by models such as [4, 105, 106], attempts to circumvent the limitations of the tiling-based paradigm by using native resolution input. However, they typically employ a pretrained fixed-resolution vision transformer as vision encoder, which leads to additional costs associated with adapting the ViT's distribution.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes UniViTAR (Unified Vision Transformer with Native Resolution), a family of vision foundation models designed to handle visual inputs in their native resolution and aspect ratio across both image and video modalities. Unlike conventional Vision Transformers that operate on fixed-size, square inputs, UniViTAR aims to preserve more spatial and contextual fidelity by avoiding aggressive resizing.\n \nTo achieve this, the authors make several contributions:\n- Versatile Framework: They introduce a family of homogeneous visual foundation models that support native resolution and unified feature extraction across visual modalities. \n- Progressive Training Paradigm: A four-stage training process is introduced, beginning with fixed-resolution pretraining and gradually adapting to native resolution and video inputs. \n- Empirical Performance: Trained only on public datasets, UniViTAR models (ranging from 0.3B to 1.4B parameters) achieve leading performance on a range of image and video classification/retrieval tasks, as well as strong results on multimodal benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Originality: Currently, there is no complete training scheme for native resolution-based Vision Transformers (ViTs) available in the open-source community.\n \n2. Significance: Native resolution-based ViTs play a crucial role in visual tasks, particularly in applications that are highly sensitive to resolution requirements, like extreme resolution and ratio.\n \nWeaknesses\n \n1. The comparisons lack fairness. The paper states that models are kept at native resolution during inference; however, in Table 6, benchmarks such as DocVQA, InfoVQA, and HRBench4K are known to contain images with extremely high resolutions. Using native resolution results in a much larger number of tokens (several thousand) compared to algorithms based on fixed resolutions (a few hundred tokens), making the comparisons clearly unfair. It would be more appropriate to follow the approach of LLaVA-NeXT—by using tiled sub-images to increase the token count and enhance dynamic resolution awareness for the baseline methods, or by comparing with models that have more parameters but similar computational resource usage to UniViTAR. Only if the superior performance is maintained under these fairer conditions can the effectiveness of the proposed method be truly demonstrated.\n \n2. The paper lacks intuitive case studies and theoretical analysis. Most of the content is devoted to experimental results showing that the native resolution approach outperforms the fixed resolution approach, but it does not analyze the underlying reasons. Does the native resolution method perform better in all scenarios, or only in cases of extreme aspect ratios or resolutions? It is necessary to further analyze the advantages of native resolution on specific cases or benchmarks.\n\nQUESTIONS:\n1. Does the training scheme based on native resolution present efficiency issues? In this paper, images are resized during training to ensure that the final sequence length matches the specified maximum length. Compared to directly maintaining the image’s native resolution, this approach significantly increases computational resource requirements during training. Would a method similar to the Qwen series, which directly uses the native resolution of images for training, be a better solution?\n \n2. Does scaling images to the maximum sequence length during training, while keeping their native resolution during inference, impact inference performance? During inference, the model may encounter very small images, whereas most images during training are enlarged to the maximum sequence length. This leads to a distribution mismatch between training and inference data, which could negatively affect the performance on small images. The poor performance of the Native Resolution approach at a sequence length of 256, as shown in the right graph of Figure 3, precisely illustrates this issue. Since many benchmarks in Table 6 include a large number of small images, it would be valuable to investigate whether appropriately increasing the resolution of these small images can further improve performance. Such an ablation study should be included.\n \n3. How efficient is inference with ViT using native resolution? When the image resolution is very high, a large number of tokens will inevitably be generated. Even though this may lead to improved performance compared to traditional methods, will the inference efficiency decrease significantly as a result?\n\n4. While the model is capable of handling higher-resolution inputs, it was not sufficiently trained across the full range of such resolutions. Did the authors analyze how the distribution of image resolutions and aspect ratios in the training data affects the model’s performance on tasks requiring varying input sizes?\n\nLIMITATIONS:\nThe authors did not discuss the limitations of the paper in a dedicated section. In my opinion, the main limitations of this work are as follows:\n \n1. Lack of theoretical analysis or intuitive case. For example, under what specific circumstances does the native resolution ViT outperform traditional ViT models?\n \n2. Lack of fair comparisons. One of the main applications of ViT is as a visual encoder in MLLMs. However, Table 6 does not successfully demonstrate the superiority of native resolution ViT in this regard. This is my primary concern.\n \n3. Computational efficiency issues. For large images, using native resolution produces a significant number of tokens. It is not desirable to sacrifice a great deal of speed for only a marginal improvement in accuracy.\n \nOverall, this is a meaningful work, but I think there are still some pain points that need to be addressed.I will consider raising my score if the authors can adequately address the question I raised.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UniViTAR, a family of Vision Transformer models designed to process images and videos at their native resolution and aspect ratio, addressing limitations of conventional fixed-resolution ViTs. \nThe architecture integrates modern components such as 2D RoPE, SwiGLU, RMSNorm, and QK-Norm, and employs a progressive training strategy that transitions from fixed to native resolution and from image to video modalities. \nA hybrid loss combining sigmoid-based contrastive learning and feature distillation accelerates training. \nTrained solely on publicly available image-caption data, UniViTAR achieves strong performance across a range of visual and multimodal benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1.This paper proposes a native resolution Vision Transformer that enables effective feature extraction from images of varying resolutions. It adopts a simple yet effective progressive training strategy, and demonstrates strong performance across image classification, video understanding, and multimodal tasks.\n\n2. This paper is well-written and easy-to-follow.\n\n**Weakness**\n\n1.One limitation of the paper is the lack of discussion on related work that tackles similar problems using different architectures. For example, causal convolutions, which are widely used in video generation, have also been applied in encoding tasks for both images and videos, and naturally support variable-length or multi-resolution inputs. Similarly, diffusion transformer (DiT) models for image and video generation share many architectural elements with UniViTAR, such as Transformer backbones with RoPE, QK-Norm, and progressive resolution handling. While these models target generative tasks, they address similar structural and resolution challenges.\n\n2. While positioning UniViTAR as a vision encoder for MLLMs is arguably the most impactful application of this work, the comparison is limited and outdated. The paper mainly contrasts with earlier vision encoders, but omits recent models that specifically support native resolution, such as NaViT and FlexiViT. Furthermore, several recent works focused on multimodal understanding—like Scaling Language-Free Visual Representation Learning and Cambrian-1—propose advanced vision encoders tailored for MLLMs.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNo\nThe paper does not discuss its own limitations, although mentioned in check list. For a model targeting native resolution and multimodal understanding, it would be valuable to acknowledge potential weaknesses such as computational cost, scalability, or video performance gaps.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UniViTAR, a family of unified Vision Transformer backbones that support native resolution and both image/video modalities, aiming to serve as a solid visual encoder for multimodal large models. The key contribution is not in architectural novelty, but in the systematic integration of community-accepted best practices—such as 2D Rotary Position Embedding, dynamic patch tokenization, RMSNorm, QK-Norm, and progressive training strategies including staged curriculum from fixed to native resolution, as well as inter-batch image/video alternation. The authors conduct extensive and rigorous experiments on public datasets, covering zero-shot classification, retrieval, linear probing, and dense prediction tasks, consistently demonstrating strong performance scaling across model sizes.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. The implementation is solid, with carefully staged training, systematic ablation studies, and thorough benchmarking across diverse visual tasks (images, video, multimodal VQA, dense prediction, etc.).\n\n2. The methodology, training details, and experimental setups are clearly described, with extensive tables and analyses covering key baselines, scaling trends, and ablation results.\n\n3. The work serves as a robust “best practices” summary for engineering strong vision backbones for multimodal LLMs, and the systematic experiments further validate the effectiveness of these recipes.\n\n# Weaknesses\n\nThe core architectural design and training strategy closely follow established approaches in the community, especially Qwen2VL and NaViT. The work mainly consolidates community best practices rather than proposing new model components or learning algorithms.\n\nQUESTIONS:\n1. What concrete differences (in architecture, training, or results) set UniViTAR apart from Qwen2VL's vision encoder? A table would help clarify any unique contributions.\n\n2. How much benefit can be gained by using Dinov2 for distillation in Stage 1?\n\n3. On vision-language benchmarks, the performance gain of scaling the vision encoder from 0.6B to 1B seems to be limited. What is the author's perspective on this phenomenon?\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces UniViTAR, a family of homogeneous vision foundation models designed to process native-resolution visual data (images and videos) uniformly, addressing limitations of conventional fixed-resolution Vision Transformers (ViTs). It introduces architectural enhancements (like 2D Rotary Positional Embeddings, SwiGLU, RMSNorm, and QK-Norm) and a progressive training strategy combining curriculum learning, modality alternation, and contrastive learning with feature distillation. Trained on public datasets, UniViTAR scales from 0.3B to 1.4B parameters and achieves strong performance across zero-shot image/video classification, retrieval, and multimodal understanding tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Open-sourcing benefits the community. Training on publicly accessible datasets and releasing models/code promotes reproducibility and encourages further research.\n\n2. UniViTAR achieves competitive or superior results on a broad range of benchmarks across image, video, and multimodal tasks.\n\n3. The integration of advanced components (e.g., 2D-RoPE, SwiGLU, QK-Norm) and the staged curriculum (e.g., resolution and modality progression) shows a well-engineered and scalable framework.\n\nWeaknesses:\n1. While the paper emphasizes native resolutions, existing models like Qwen2.5-VL already adopt native resolution strategies, reducing the novelty of the contribution.\n2. The multimodal performance (Table 6) is relatively modest, and extensions to large-scale MLLMs remain under-explored.\n3.  The paper would benefit from more granular ablation studies on the architectural choices (e.g., LayerScale, RMSNorm) and their impact across both classification and multimodal tasks.\n\nQUESTIONS:\nHow does UniViTAR differentiate itself from prior native-resolution approaches like NaViT and Qwen2.5-VL?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}