VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution
Authors: Limeng Qiao
Yiyang Gan
Bairui Wang
Jie Qin
Shuang Xu
Siqi Yang
Lin Ma
Venue: NeurIPS 2025 poster ()
Keywords: Vision Transformer
Multimodal
Primary area: deep_learning

Abstract: Conventional Vision Transformer streamlines visual modeling by employing a uniform input resolution, which underestimates the inherent variability of natural visual data and incurs a cost in spatial-contextual fidelity. While preliminary explorations have superficially investigated native resolution modeling, existing works still lack systematic training recipe from the visual representation perspective. To bridge this gap, we introduce Unified Vision Transformer with Native Resolution, i.e. UniViTAR, a family of homogeneous vision foundation models tailored for unified visual modality and native resolution scenario in the era of multimodal. Our framework first conducts architectural upgrades to the vanilla paradigm by integrating multiple advanced components. Building upon these improvements, a progressive training paradigm is introduced, which strategically combines two core mechanisms: (1) resolution curriculum learning, transitioning from fixed-resolution pretraining to native resolution tuning, thereby leveraging ViT’s inherent adaptability to variable-length sequences, and (2) visual modality adaptation via inter-batch image-video switching, which balances computational efficiency with enhanced temporal reasoning. In parallel, a hybrid training framework further synergizes sigmoid-based contrastive loss with feature distillation from a frozen teacher model, thereby accelerating early-stage convergence. Finally, trained exclusively on public accessible image-caption data, our UniViTAR family across multiple model scales from 0.3B to 1B achieves state-of-the-art performance on a wide variety of visual-related tasks. The code and models are available here.

VITRIX-UmiViTAR: Unified Vision Transformer with Native Resolution
Limeng Qiao
Yiyang Gan
Bairui Wang
Jie Qin
Shuang Xu
Siqi Yang
Lin Ma"
Meituan Inc.
qiaolmêpku. edu. en, realgumpêtjj.c edu. edu en, (tibairuiwang, jayginllu)@gmail.comm com sxu19970122 com, siqi. yanguuq net au, forest. .innstgmaamaamml........ com
Abstract
Conventional Vision Transformer streamlines visual modeling by employing a uniform input resolution, which underestimates the inherent variability of natural visual data and incurs a cost cpatial-contextual fidelity. While preliminary explo- rations have superficially investigated native resolution modeling, existing works still lack systematic training recipe from the visual representation perspective. To bridge this gap, introduce Unified Vision Transformer with NAtive Resolution, Le. UniViTAR, a family of homogeneous vision foundation models tailored for unified visual modality and native resolution scenario in the era of multimodal. Our framework first conducts architectural upgrades to the vanilla paradigm by integrating multiple advanced components. Building upon these improvements, a progressive training paradigm is introduced, which strategically combines two core mechanisms: (1) resolution curriculum learning. transitioning from fixed- resolution pretraining to native resolution tuning, thereby leveraging ViT's inherent adaptability variable-length sequences, and (2) visual modality adaptation via inter- batch image- video switching, which balances computational efficiency with enhanced temporal reasoning. In parallel, a hybrid training framework further synergizes sigmoid-based contrastive loss with feature distillation from a frozen teacher model, thereby accelerating early-stage convergence. Finally, trained ex- clusively public accessible image-caption data, our UniViTAR family across multiple model scales from 0.3B to 1.4B achieves sate-of-the-aa performance on wide variety of visual-related tasks. The code and models are available here.
SLUP-SDORR CUNITAB--II UUNITAR--EE LLINTAB-I wwe.Zocchicceeeaaaaato
DEN-IE ADM-2441 SOL.P-SORRIEEE HANITAR--AA2
Veeeo Envobbet Cleaffation
wwbeecblltaderennngggg
OCRIGA
Hallater Banch
Iragehut
Sag
ALD
Figure 1: The figure presents: (left) systematic overview of model scaling performance across downstream tasks when increasing parameter size from 0.3B to IB, and (right) a comprehensive comparison of multimodal capabilities against SOTA baselines on diversified benchmarks.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
In the era of rapid advancements of multimodal large models, Vision Transformer [. characterized by its simplicity and scalability, has emerged as a foundational architecture for visual representation learning. Drawing inspiration from transformer-bas based large language models, conventional ViT usually uniformly converts raw visual data into square aspect ratio and standardized resolution to reduce modeling complexity and simplify the training workflow. While this paradigm simplifies feature extraction and aligns with existing engineering practices, it inherently imposes artificial constraints on real-world visual data by disregarding the inherent variability of natural images.
Recent studies have preliminarily investigated the vision backbone within native resolution paradigm. Flex ViT [2] introduces flexible ViT architecture featuring dynamical patch size selection in the patch embedding layer, which facilitates smooth variation of token sequence length through parametric scaling. In contrast, NaViT 3 maintains fixed patch size while directly processing native resolution images with varying aspect ratios, where the token sequence length of different images changes dynamically. This approach demonstrates the feasibility and benefits of adopting natural language processing style packing strategies for vision foundational model. Qwen-VL's 4 [ vision encoder inherits NaViT's core configuration while specifically investigating native resolution impacts from multimodal large model perspective. While the aforementioned approaches have attracted initial research attention, the field still lacks a comprehensive series of backbones that simultaneously support native- and fixed-resolution processing, achieve high- fidelity feature extraction for both images and videos.
vision
To address this gap, we present the Unified Vision Transformer with NAtive Resolution, termed as UniViTAR, a family of vision foundational backbones designed to uniformly process visual modalities (image video) with native resolution and dynamic aspect ratio. Building upon insights from large language model recent practices and architectural innovations in visual transformers, our approach firstly conduct systematic architectural upgrades to the vanilla VIT paradigm by integrating multiple advanced components: 2D Rotary Position Embedding, SwiGLU activation function, RMSNorm layer, QK-Norm mechanism, and LayerScale module. These modifications collectively establish a more robust architectural foundation compared to conventional implementations. Secondly, we develop a progressive training paradigm with two complementary adaptation strategies: 1) the progressive resolution adaptation strategy employs curriculum learning from fixed low-resolution (e.g., 224) pretraining to native-resolution fine-tuning. Notably, our experiments reveal that the advanced ViT architecture exhibit remarkable adaptability models pretrained at fixed resolution can efficiently generalize to variable-length visual sequences through limited native resolution tuning. 2) the progressive visual modality adaptation strategy addresses computational challenges in video processing by deferring video data integration the final training phase. We further demonstrate that alternating image-video training sequences (inter-batch modality switching) significantly outperforms mixed-baie batch (intra-batch modality mixing) in preserving image understanding capabilities while acquiring temporal reasoning skills. Thirdly, we implement hybrid training framework combining contrastive learning objectives with distillation techniques. Our primary optimization employs a sigmoid-based contrastive loss [6] for unified image-video representation learning. To accelerate early-stage convergence, we further incorporate feature distillation from a frozen vision teacher model as auxiliary training objective during initial phases, then gradually phasing out this regularization as the model matures. Finally, through this comprehensive approach trained on accessbecces datasets, we successfully scale family of vision backbones supporting native resolutions and both visual modalities, with parameter counts ranging from 0.3B to 1.4B. Extensive evaluations demonstrate the effectiveness of our proposed methods.
Specifically, the contributions of our ViTVR family are summarized as follows:
We introduce a family of homogeneous visual foundation models that support native res- olution and unified feature extraction across visual modalities, offering the community a versatile framework for multimodal research.
We develop an efficient and effective progressive training strategy that addresses the com- putational challenges of native resolution modeling while systematically enhancing the model's image-caption alignment capability.
We train our models with public-accessible datasets, achieve leading performance with limited resources, and observe trend of performance increasing with parameter scaling.

---

Linear
Layer Scale
RMM-Norm
Attention Block
SwIGLU-FFN
Swissh Layer
:
RMS-Norm
Linear
Attention Block
Layer
Linear
Patch Embedding
Macked Huadi-Head Self-Attentie
RMS
RMS
Unified Native Patchify
RoPE2D
RoPE2D
Image', Image', Video .... Image", Video"
Figure 2: Architecture of UniVITAR family. All visual inputs are uniformly transformed into patch sequences and fed into Vision Transformer. In addition to using the Pre-Norm approach, we also adopt RMS-Norm the normalization layer in both MHA and FFN module.
Method
2.1
UniViTAR: Homologous Visual Foundation Model
2.1.1 Architecture Design
UniViTAR is Transformer-based encoder model that inherits the original architecture of the conventional Vision Transformer [I] but incorporates the following advanced modifications:
2 given the native input visual data X E T TCx XXW xW of any vision modality (image, video), where = represents image and T > represents video, UniViTAR firstly patchifies X into series of dynamic length visual patch sequences = (N, S), where is the number patches per image/video and the number of pixels per patch. Then a 3D convolution layer is adopted the Patch Embedding Layer to consistently convert the above patch sequence into a visual token sequence (N, D), where D the hidden size of the following attention layers.
Unified Patchify for Native Image and Video Modality. As illustrated in the Figure
2D Rotary Position Embedding. Drawing on the architecture designs language models, the original VIT regards the position information among different visual tokens one-dimensional association. In fact, considering that visual data usually has spatial association (row and column) and temporal association (time), the position information between different tokens is usually considered be multi- dimensional. Thence we remove the original absolute position encoding and introduce 2D-RoPE 7 into each subsequent encoder layer to capture the two-dimensional positional information of images. Furthermore, we found that the presence or absence of the class token the original ViT has almost no effect model performance. To ensure the consistency of position encoding, also empirically remove the design of class token.
SwiGLU and RMSNorm. By leveraging the recent advances of LLaMA 8 architecture design for language modeling, UniViTAR incorporates SwiGLU as the feed-forward network (FFN) and replaces all normalization layers with RMSNorm. In addition, adds extra RMSNorm to each SwiGLU-FFN for good expressivity and improving the training stability.
Query-Key Normalization. In order to improve the stability of model training. we adopt the QK- Norm technique 10 which applies normalization to the queries and keys before the dot-product attention computation, to directly controls the norm growth of input the softmax and avoid abnormal attention logits. Note that we still utilize RMSNorm as the norm function ensure the consistency of the architecture.
2.1.2 Homologous Model Scaling
The VniVITAR family consists of a comprehensive suite of foundational and scratch-train models, encompassing parameter range from 0.3 to 1.4 billion, i.e. UniViTAR-0033//0.61 The hyper- parameters and important information are listed in Table I in details.

---

Table 1: Detailed architectural configuration for UniViTAR family.
Model
Hidden-Size
Intermediate-Size Enender-Iayers
Attention-Heade
Parameters (M)
UniViTAR-0.38
1024
4224
24
16
310
UniViTAR-O.66
1280
5184
32
16
637
UniViTAR-IB
1920
7680
32
24
1419
Table 2: Detailed training strategy illustration of UniViTAR family.
Stage
Stage
Stage 3
Stage 4
LLAMA
LLAMA
LLaMA
LLaMA
Train Strutegy
UnIVITAR
Contraction
Contraciio
Confrnntive
UniVITAR
UniVITAR
UniVITAR
Data Modality
Image
Image
Image
Image. Video
Resolution
224
224
224 Sigmoid IB
Native Sigmoid 1B
Native Sigmoid
Loss Function
Sigmoid, KL 12B
Seen Sumples
0.6B
2.2
Contrastive Visian-Language Pretrain with UniViTAR
2.2.1
Architecture Design
In general, the acquisition of UniViTAR largely follows the basic training paradigm of CLIP [[]] Specifically, the native-resolution visual input is encoded into the visual feature space via the UniViTAR encoder to obtain F, E N. while the textual input & is projected into the textual feature space through a pretrained LLaMA 12] decoder to obtain F; E R. Ne-ee D. The dynamic-length visual features F, are then uniformly converted into the visual embedding f. E R. through a global average pooling and the feature corresponding to the <EOS> token in F; is utilized as the textual representation ft E R: of the input caption. Subsequently, fo and fe are further projected into the same shared semantic space via a linear projection layer respectively. Then a simple pairwise sigmoid loss 6 is employed as the contrastive supervision to align the visual and text modalities semantically.
2.2.2 Optimized Contrastive Training Strategy
To ensure that the model can converge efficiently and the training cost is controllable, we carefully design the training pipeline of UniViTAR into four stages sequence, shown in Table 2
Stage I: Visual knowledge pre-acquisition with hybrid paradigm training. The primary objective of this phase is to efficiently pretrain visual foundation model from scratch by integrating two classic learning paradigms: vision-text contrastive learning and visual knowledge distillation. Specifically, the proposed architecture employs riple-branch parallel design: (1) xavierr-iittialized ViTAR, (2) frozen pre-trained text encoder, and (3) a frozen pre-trained visual teacher. During training, only the target visual foundation model receives gradient updates, with other branches fixed to minimize computational overhead while preserving knowledge integration. For implementation, we adopt LLaMA 12 and DINOv2-g [3] as default components, though the framework supports substitution with alternative pre-trained foundation models. The composite training objective is defined as:
Loverall = Lcontrastive (f (FUNVVETAR ff FLLaMA) A Ldistitlation (UUVViTAA f.Dioo
where Leontrastive is the sigmoid loss from SigLIP [6] and Ldsetillttion is the KL Divergence [4) The target visual foundation model functions a visual knowledge bridge, simultaneously performing image-text alignment and feature distillation. This phase processes 12B samples with all images resized to 224, constituting 82.2% of the total training data (12B/14.6B).
Stage 2: Finetune with full-parameter for superior alignment. The objective of this stage is to further enhance the upper limit of image-text alignment through cull-parameter fine-tuning of both vision and text encoders, establishing a unified semantic-visual space. The visual distillation branch is deactivated during optimization. Training employs identical image-caption pairs as Stage 1 at 224 resolution. Considering the high computational cost of full parameter fine-tuning, the training process is conducted on IB samples, accounting for 6.9% of the total training data.

---

Stage 3: Unlock the model-capacity of native-resolution. In this stage, our strategy extends the model capability to handle native-resolution, thereby achieving robust image-text alignment for dynamic-resolution inputs. However, enabling native-resolution capacity necessitates addressing two critical challenges: (1) ensuring positional encoding are thoroughly trained across variable sequence lengths, and (2) transfering feature distribution from uniformly-resized patches to native patches through efficient training. In practice, visual data is batched in its native form to preserve original resolutions and aspect ratios. Then the intra-batch images are dynamically scaled (with aspect ratios maintained) to align total sequence sequence lengths Liotal with a predefined token limit Linaa. That is to say, when the value of / Leotal is greater than 1, all data will be uniformly enlarged, and vice versa the shape of all data will be reduced, ensuring consistent computational loads across batches. Within attention blocks, each token's 's receptive field is confined to tokens from the same image via masking, enabling isolated intra-image contextual modeling while preserving inter-sample independence. During training, resolution diversity within batches ensures comprehensive training of positional encoding across varying context lengths, progressively refining the model's ability to generate features aligned with native patch distributions. At inference, inputs processed directly at their native resolutions without resizing. In this stage, IB samples (6.85% of the total training) were trained with the text branch frozen throughout the process.
Stage 4: Unifying visual modalities with image-video alternation training. The goal of this stage is to unify image and video input modalities with nativerresolution and dynamic video length. Inspired by the Intern Video series [15] [6], we utilize both image-text and video-text pairs to optimize the UniViTAR checkpoint from Stage 3 with image-video alternating training strategy. This strategy addresses three critical considerations: (1) leveraging image data's superior scale and diversity compensates for video data scarcity while maintaining visual content continuity; (2) joint image-video training preserves cross-modal comprehension capabilities; (3) alternating modality- specific updates enforce focused parameter adaptation through sequential modality optimization. The alternating training protocol first initiates each epoch with random permutation of image- video data enhance stochasticity. Subsequently, data batches are partitioned into global batch units and alternately sequenced at global batch granularity. Through this structured approach, the configuration effectively maintains modality purity within individual training batches by enforcing strict image- vvido alternation. To accommodate native-resolutig video processing with dynamic lengths, we implement adaptive frame sampling: full temporal retention when frame count F << Finas- and uniform subsampling to Fnaz frames when exceeded. With the predefined token constraints (Imin: Imaz) ) and the calculated frame length F, all frames are subsequently resized within these computed bounds while preserving ori original aspect ratios.
2.3 UniViTAR as a Vision Encoder for MLLMs.
In this section, we introduce a simple strategy for constructing effective native resolution MLLM based on the UniViTAR series. The common and industry-validated Dision-Languaae Models (VLMs) paradigm typically combines pretrained visual backbones with large language models, followed by multimodal training on a rich mixture of vision language tasks. To ensure fair comparison and minimize bias, we adhere to this established configuration. Specifically, we employ UniViTAR as the vision encoder and employ Qwen2.5-11.5 [5] as the large language model. Following established practices [T7), we implement& three-layer MLP with pre-normalization and a 2x eixel-unshuffle operation [8] along the width dimension as the vision-language adapter to bridge the visual and linguistic modalities. For native-resolution modeling, we identify two primary challenges. On one hand, due to the varying lengths of input samples, the boundary between vision and language tokens not fixed. To enhance "modality isolation", we introduce specialized prompts, known as Boundary Markers, such as <image_stuue sstrrsssrr>aaa and <image_ent -iriigeeendd,___e end>, at the beginning and end of the vision token sequence. On the other hand, 2D- 1D flattening of vision tokens may compromise the information of the height-width ratio. To mitigate this, we incorporate Line Anchors, such <line-idx>, into the vision tokens, where idx denotes the corresponding vertical positions in the original patchified image, thereby potentially strengthening positional awareness in compressed tokens. For vision token sequue
sequence of length lww, the original arrangement x ...,, 11w ....""" 221000 ..... h.w is transformed as:
<inage_slar(>,. 111 1.W, <line-1>,-t 211 h. <line-h>, <image_ends (2)
Notably, these added markers are string- identifiers rather than special tokens of the tokenizer. To systematically evaluate multimodal comprehension capabilities, we adopt a dual--stge training
paradigm
paradigm motivated by established methodologies in vision-language alignment like [19 20).
5

---

3 Experiments
3.1 Training Recipe
Data Details. We collect public accessible image-text pairs and build our Merged-1 B dataset, which is composed of DataComp- IB (21]. COYO [22]. LAION-2B [3]. LAION-400M 24] DFN-2B (22),
CC12M [25] and CC3M 26 Moreover, to further enhance the video feature extraction capabilities of UniViTAR, we meticulously constructed a dataset Merged-65M of roughly 65 million samples by randomly selecting video clips from three public accessible video datasets, i.e. Panda-70M [27).
WebVid-I0M (28), and Intern Vid- 1OM-LTT [29]. We refer to the combined image and video data mentioned above as Merged- I.IB. The detailed data composition is summarized in the Appendix.
Hyperparameter Details. The detailed hyperparameter configurations for each training stage presented in the Appendix. As tabulated, we utilize progressive reduction of the peak learning rate in correlation with increasing visual backhone scale ensure optimal training stability. Notably, the learning rate of text branch in Stage 2 remains consistently one-tenth of the visual component throughout this phase. To enhance training efficiency, we integrated the DeepSpeed library (30 by employing ZeRO optimizer sharding [3] gradient checkpointing [32] and flash attention 33
3.2 Results on Zero-shot Image Classification & Retrieval
Evaluation Setup. Our evaluation protocol encompasses both zero-shot classification and cross- modal retrieval tasks. For zero-shot classification, we conduct evaluation on ImageNet [4] and its established variants [35 36
37 38 39 Each class is represented by multiple text prompts curated from [ [0]. The Top-/ accuracy is utilized to evaluate the model performance. For cross-modal retrieval assessment, we adopt the benchmark protocols defined in [41], evaluating on Flickr [42 43 using their official partitions. The retrieval paradigm involves bidirectional image-text matching, namely image-to-ter retrieval and text-to-image io-imaee retrieval tasks.
and MS-COCO
Results Comparison and Analysis. Table 3 demonstrates the exceptional performance of our model at comparable parameter scales. As the model size increases from 0.3B to 1.4B, the average zero-shot classification accuracy across six benchmarks exhibits a progressive improvement trend, rising from 80.5% to 81.9% and further to 83.4%. Notably, all models of varying scales employ identical training samples and strategies, with this performance enhancement attributed to parameter scaling effects finding consistent with established scaling laws in mansformer-relatee research. As detailed in the table, our UniViTAR- IB shows superior performance despite utilizing a smaller training corpus, outperforming its counterparts with more parameters, such as Intern VTT-6B [2] and EVA-SB (44] We posit that this advantage stems from two key factors: optimized model atchitecture and training strategy, and preservation of native input resolution, which generates higher-quality visual tokens.
3.3 Results on Zero-shot Video Classification & Retrieval
Evaluation Setup. We evaluate the zero-shot vidco classification performance on three popular benchmarks as K-400 [50), UCF-101 [5]] and HMDB51 1 [52), using the class names as text prompts.
Also, we evaluate the zero-shot video-text retrieval performance on ActivityNet 533 MSR- VTT [54]
and MSVD [55] Following [15 16,, for each video in the 1K version of the test split, we sample one
sentence from every set of 20 sentences for MSR- VTT. Following (56), we concatenate the multiple descriptions to form a paragraph and perform paragraph-to-vider retrieval on ActivityNet. All videos are sampled with a dynamic frame rate, with each frame dynamically resized to maintain the original aspect ratio while ensuring the total token within the range of 576 to 16,384.
Results Comparison and Analysis. Table 4 shows the performance of our UniViTAR series models on video benchmarks across comparable parameter scales. As the model size scales from 0.3B to IB, UniViTAR exhibits consistent performance gains on video benchmarks. with average zero-shot classification metrics improving from 68.0 to 69.0. When compared to models trained on image- caption data under similar parameter scales, UniViTAR achieves notable improvements. These advancements can be attributed to two key design choices: (1) preserving the aspect ratio of each frame to retain the original semantic information of visual content, and (2) employing dynamic video frame sampling to effectively capture detailed temporal information. However, when compared to the models trained exclusively on video-caption data, VniVR still has room for improvement compared to some of the latest models [57] (88 [6]. shown in the Table with gray color.

---

Table 3: Evaluation of zero-shot performance on various image benchmarks. The symbol c indicates that the image-caption data used by the corresponding method not publicly available.
Melhod Data Source
Overall
IN-IK
INEA IN-V2 IN-S ON-Net
ImageNet Variants
Overall
Flickr T=N-- (-TTTTT TII T-TT
COCO
CLIP-L 1
WIT400M +
224
72.1
75.5
70.8
87.8
69.8
59.6
68.9
60.8
65.0
85.2
36.5
56.3
OpenC1.P-11
151
DataComp1R
224
75.7
79.2
69.6
90.8
72.1
68.0
74.3
67.9
73.4
89.0
45.7
63.3
MetsCLIP-L.
46
CC-2.5B $
224
76.6
79.2
72.3
92.1
72.6
69.0
74.6
69.5
76.4
90.1
47.1
64.4
DFN.L
DFNSB $
77.1
67.5
75.7
70.4
75.5
89.6
65.6
EVA02-L
14
Merged-2B
336
77.5
79.8
76.2
92.7
73.0
68.1
74.9
69.9
78.0
89.6
47.9
64.2
CLIPAW2-LL
48
DataCompl1
336
78.1
80.3
77.7
93.3
73.5
70.9
73.1
69.5
74.6
90.4
47.2
65.6
SigLIP-L
Web +
384
79.4
82.1
76.6
95.1
75.9
7366
72.8
75.2
81.4
93.7
53.9
71.9
UniViAAR-0.BB
Merged-1BB
Native
880.6"
"T.55"
84.1
93.9
75.7
69.7
79..
76.3
84.0
95.1
54.7
771.2"
OpenCLIP-H
45
LAION2B-cn
224
72.3
78.0
59.4
89.3
70.9
66.6
69.4
68.7
75.5
89.5
46.5
63.4
MetaC1.IP.H
45
CC-2,5B +
75.3
70.5
76.4
78,3
91.8
66.2
CLIPAv2-H
DataCompIB
336
80.8
81.8
82.7
94.4
75.6
72.8
77.4
70.8
76.3
90.3
49.2
67.2
DEN II
++
DFNSB 4
378
80.5
84.4
79.6
93.8
78.3
73.2
73.4
75.9
82.0
94.0
55.6
71.9
SigLJI--SO
We +
384
81.7
83.1
82,5
95.8
77.2
74.5
77.0
76.0
83.0
94.3
54.2
72.4
UniViTAR-0.6B
Merged-1B
Native
82,1
82.3
86.8
76.7
71.6
81.1
766.6
84.1
95.5
71.7
OpenCLIP-g
45
LA1ON28-co
224
73.0
78.5
60.9
90.2
71.6
67.5
69.1
71.1
77.7
91.4
48.8
66.4
OpenCLIP-G
15
LAION2B-ea
224
76.2
80.1
69.3
92.1
73.6
68.9
72.8
72.8
79.6
92.9
51.4
67.4
EVA01-g
Merged-2B
224
76.9
79.3
74.2
92.5
72.1
68.1
74.9
72.3
79.0
91.7
50.3
68.2
EVA02-E 45
Merged-2B
336
80.9
82.0
82.2
94.6
75.6
71.6
79.4
73.2
78.9
94.1
51.1
68.7
CLIPAV2-G
DataComp1 I
336
82.7
83.1
86.0
95.4
77.3
74.5
79.7
72.2
78.3
92.2
50.4
67.8
IntemViT-68
I2
Ln5ernV--BB
224
82.5
83.2
83.8
95.7
77.3
74.3
80.6
75.3
81.7
94.7
54.1
70.6
FVVA-8B
Mergeil-2B
224
82.9
83.5
85.2
95.3
77.7
74.3
81.2
74.9
80.8
95.6
53.0
70.3
UniViTAR-IB IB
Merged-1B
Native
83.5
82.9
89.1
95.7
77.
73.4
82.
76.3
83.5
95.1
55.5
713
Table 4:
Evaluation of zero-shot performance on various video benchmarks. The symbol f
signifies that the reported metrics are based on our own evaluations.
Method
Type
Res.
Frames Overall
Classifictioon K400 UCF HMDB
Overall
ANet V-T
MSR-VTT V
V T-
MSVD
OpenCLIP-LL L.UUIP-L-
45
Image
224
16
58.4
61.5
69.2
44.5
41.0
32.0
34.2
30.1
37.5
63.7
48.5
!DFN-L
Image
224
16
56.4
56.8
67.7
44.8
40.4
31.6
34.1
32.1
35.2
61.9
47.7
1EVA02-L
44
Image
336
64.4
64.4
76.0
52.8
44.7
35.8
37.2
35.4
39.7
51.0
SigLLI-
Image
16
64,8
79.2
50.9
45.3
34.3
35.7
40.0
73.0
53,0
VICLIP-L
29
Video
224
8
61.8
41.2
24.0
15.1
41.3
42.4
75.1
49.1
InterVideo-L.
31.4
39.6
40.7
67.5
43.4
UMT-L 59
Video
224
16
47.7
39.4
41.9
38.6
42.6
74.5
49.0
Image&&
Native
2--32"
68.0
66.0
82.6
55.4
53.9
47.9
49.9
48.0
48.8
77..8
30.7
OpenCLII-H 45
Image
224
16
62.0
61.7
72.5
51.6
43.5
36.1
38.9
34.5
38.9
63.3
49.4
*DEN-H
17
Image
378
16
62.9
63.8
76.7
48.2
46.2
39.7
42.9
36.1
39.6
66.6
52.4
"Sig..I -O
6
Image
384
16
67.3
66.8
83.0
52.1
47.5
36.6
39.3
37.5
41.1
75.5
54.7
TVTSV2.H [
Video
224
12
63.2
78.0
52.1
41,3
UniVFTAR-0.6R Image&Vider Native
T-32""
66.6"
66.66"
82.97
55.22"
69.5
48.1"
"$1.."
"486
50.2
"15.8"
'34.3
'OpenCLIPP
45
224
16
63.1
61.5
76.6
51.1
44.4
36.8
39.8
36.4
39.2
64.3
50.1
1OpenCI JP-G
45
Image
224
16
64.2
63.2
76.2
53.4
46.0
36.7
41.4 A
36.9
41.8
67.5
51.5
EEVAD1-g
Image
224
16
62.8
63.4
72.1
52.9
45.5
37.0
40.1
37.2
40.1
67.6
50.8
IntemVVTT-B
Image
42.4
46.3
- Image& Video Native
2-332"
69.0
68.6
81.0
57.3
54.6
47.8
"49.6"
48.3
47.6'
795.5"
35.2"
VideoCoCa-g
37
Video
224
8
72.4
72.0
86.6
58.7
39.0
33.0
34.5
64.7
34.4
33.0
34.5
VideoPrism-g
Video
288
16
764
50.3
52.7
51.7
52.7
Intern Vidco266
Video
56.5
53.7
55.9
59.3
3.4 Results on Image Classification by Linear Probing
Following common prectices [12 we assess the performance of UniViTAR family as off-the- shelf backbones on image classifications. Specifically, we train a linear classifier on the last feature layer with a frozen backbone on ImageNet-IK (34] and evaluate the performance on the validation 38) In addition, we also report the classification performance with attentive probing setting as used in [61, which adopts a cross-attenticn layer with random initialized queries. Table Birpresett the downstream classification performance of our models. First, as the model size increases, the average performance across six benchmarks demonstraks demonstrates consistent improvement. Second, we observe that the attentive probing performance shows stable improvements over linear probing. Furthermore, compared to public methods, our UniViTAR family shows superior performance across various parameter scales.
set and other ImageNet variants 62 35 36
37
7

---

Table 5:
Evaluation of classification performance on various image benchmarks. The signifies that the reported metrics are based on our own evaluations.
ImageNet Variants
Method
Classifier
Overall
IN-IK
IN-Real
IN-V2
IN-A
IN-R
IN-S
CLIP-L
1
Linear
336
85.3
88.8
75.8
SigLIP-1
6
Attentive
224
86.5
AIMv2-L 61
Attentive
UnIVITAR--EA
Linear
Raiiv
K3.0
87.6
90.3
79.5
84.T
90.5
66.0
UniVITAR-0.38
Attentive
Native
83.3
87.7
90.5
79.8
83.8
91.1
66.8
CLIP-H 11
Lincar
224
84.4
88.4
75.5
IDFN. 47
Linear
378
81.6
87.3
90.4
78.8
74.8
90.3
68.3
SipLIP-S00 100 0
Attentive
384
87.3
AIMv2-11 112221111 61
Attentive
224
87.5
Lincur
Native
84.3
88.2
90.6
80.6
87.I
92.0
UniVITAR-0.6B
Attentive
Native
84.8
88.3
90.7
81.0
87.3
92.5
68.8
OpenCI.IP.G
Linear
77.2
DINOv2-g
13
Linear
224
78.6
86.5
89.6
78.4
75.9
78.8
62.5
EVA01-g
39
Lincar
224
79.1
86.5
89.3
77.4
70.5
87.7
63.1
AIMv2-1B
61
Attentive
224
88.1
VitemVVT--B
P2
Linear
224
82.5
88.2
90.4
79.9
69.1
EVA 8B 49
Linear
224
88.5
UniVITAR-1B
Linear
Native
86.0
88.9
90.8
81.5
90.1
94.0
70.7
UniViTAR-IB
3.5 Results on Dense prediction.
In this section, evaluate the dense prediction performance of UniViTAR family by transferring to semantic segmentation. Following 12 63 we fine-tune a decoder with freezing backbones under two different structures, iee, Linear and UperNet. Linear decoder transforms the dimension of one single layer visual feature to number of semantic classes, while the UperNet decoder employs PPM and FPN integrates multi-scale features. Experiments are conducted on the ADE20K (64] dataset. In terms of data preprocessing, we employed the same fixed-resolution input and data augmentation strategies as those used in Intern ViT [12]. Corresponding results are shown in Table 6 We can observe a performance gap between these two types of decoder, this can be understand that UperNet has significantly more trainable parameters than Linear decoder. Taking UniViTAR-0.6B as an example, Linear decoder has parameter count of 0.2M, whereas UperNet contains approximately 200M parameters. Notably, our UniViTAR Family demonstrates an obvious performance advantage compared with existing state-of-the vision encoders. Under the setting of Linear decoder, our UniViTAR- IB achieves performance of 45.4 mloU, which +6.1 points over OpenCLIP-G (55 and +10.8 points over ViT-22B [10. In the case of UperNet decoder, UniViTAR: BB reaches 56.2 mloU, also surpassing larger parameter-scale model like Intern ViT-6B [12.
Table 6: Evaluation of semantic segmentation on ADE20k dataset with frozen backbones.
Method
CropSize
mioUimmee
mlo UperNer
CL.IP-I. M
39.0
IigLIP-SO
6
40.8
DDNNI11 47
41.3
OpenCLIP-G
45
512²
39.3
Intern VIT-6B
12
5042
47.2
54.9
VIT-22B
[10
5042
34.6
52.7
503
40.7
554.
UniVITAR-0.c6
5042
42.9
55.1
UniVITAR-IB
5042
56.2
3.6 Results on Multimodal Understarding
Evaluation Setup. To assess the potential of multimodal understanding, we employ dual-stage training paradigm, similar to common practices [19 20 the pretraining stage, we train the
projector with learning rate of le using merged 2.5M dataset comprised of LLaVA-CC3M-
Pretrain [77, VAL-aV [19], ShareGPT4V-PP (20). In the fine-tuning stage, we unfreeze
the whole model, and train it with learning rate of Le using the high-quality instrution-tunil

---

dataset LLaVAI.5-Finette [65]. Note that the native-ress resilvtissootionn strategy of Boundary Markers
and Line Anchors are only applied in the fine-tuning stage. All evaluations are conducted using
VLMEval [66]. assessing performance across 16 popular benchmarks, including GQA [67].
DocVQA 688 VQA [69], ScienceQA [70,, TextVQA 711, VizWiz [72] OCRVQA [73], OCR-
Bench (74,, MME 775. MMMU [76]. SEEDBench _
Vista VilthaMMiMM MINI 78 AI2D [79].
HallusionBench 80). POPE 81. HRBench4K [82].
Results Comparison and Analysis. As illustrated in Table under exactly the same training data and training strategy, the proposed UniViTAR surpasses various state-of-the-an vision encoders [47
61 6 on numerous multimodal understanding benchmarks. Notably, Uni demonstrates exceptional capabilities scenarios involving dense information, such as document parsing [68]. graphic parsing [69], and high-rews nigh-resolution tasks [22]. We argue that the native resolution plays crucial role achieving outstanding performance these areas, ensuring minimal loss of image information. We also assess the effectiveness of the proposed strategy of Boundary Markers and Line Anchors on 0.6B model size, as demonstrated in Table which highlights their impact.
Table 7: Evaluation of multimodal understanding on various vision- laigua- benchmarks. Note the superscript represents model with Boundary Markers and Line Anchors.
UnIViTAR
Si
DFN-H
SigLIP-SO
0,3B
0.6B
0.6
IB-
Resolution
Nalive
GQATouee
6I.5
60.6
61.3
61.0
60.3
58.2
60.3
61.2
DocVQAvaL
30.8
25.9
36.2
32.0
47.7
46.3
48.2
47.0
VQAvVQ
ScienceQAvat
63.6
62.7
64.8
66.4
64.5
65.0
63.6
65.3
TeutVQAVAL
48.0
41.7
53.2
50.9
50.8
52.0
50.7
$2.0
VizWiz
30.5
30.8
29.8
OCRVQA
31.2
32.0
31.0
30.9
32.2
31.6
32.2
32.1
OCRBench
35.2
30.6
22.4
36.0
33.6
37.0
36.9
36.4
MME
59.8
60.0
57.9
58.6
59.0
MMMUVAL
35.9
37.1
35.4
36.1
38.7
36.6
37.0
SEEDBench. IMG
70.0
70.3
70.9
71.2
68.2
67.8
68.0
69.3
Vista_MINI
28.6
29.6
27.5
27.9
28.5
AT2D
60.5
6044
60.6
58.0
57.7
58.7
59.3
HallusionBench
56.8
56.6
53.9
54.8
$7.0
55.2
57.8
$4.1
POPE
87.2
85.4
87.7
87.1
88.1
87.9
88.
HRBench4K
39.9
39.5
44.5
45,0
44.1
43.6
46.1
46.4
Average
47.6
46.5
48.0
48.5
48.
49.1
493
49.6
3.7
Ablation Study
3.7.1
Robustness verification of resolution mode.
This section analyzes the influence of three resolution modes: fixed resolution, native aspect ratio, and native resolution in Figure For fixed resolution mode, we resize the shorter edge of each image to predefined size and apply CenterCrop to ensure the sequence length strictly equals ( (S/14)², where represents the patch size. Increasing S proportionally extends the sequence length. In native aspect ratio mode, we scale images while preserving their original aspect-ratio, ensuring that wh/14 114 approximates the target sequence length. We evaluate 12 sequence lengths ranging from 256 to 16K tokens with our 0.6B model. The experimental results reveal three key findings: /) performance initially improves then declines with increasing sequence length under both fixed and native aspect ratio modes, peaking at 1024~-0966 tokens. 2) native aspect ratio consistently outperforms fixed resolution, indicating that preserving original aspect ratios retains better image information. 3) native aspect ratio occasionally surpasses native resolution performance at certain lengths.
3.7.2 Verification of the effectiveness of training strategies.
As introduced in the method, we categorize the training strategy into four distinct stages, based on resolution modes (fixed or native), visual data modalities (image or video), and model training parameters (trainable or freeze). As shown Figure 4 for zero-shot image classificution (left), we show that S1, S2, and S3 exhibit progressive performance improvements, while $4 maintains comparable accuracy despite incorporating alternating training. In contrast, for zero-shot video classification (right), SI and S2 show minimal performance variation, with dynamic-resolution

---

Zaro Shot Perfermane on Iragellessk Defirmett Basckttonn Mode
Zaro Perfermance on Iragehiet Detierezz Bewokitinn Plove
Merto Aupert Bate Hoant
Aesood Rite
FUOLE
Figure 3: Performance comparison of different resolution modes as the length of the vision sequence increases. The black dashed line shows the performance when using native resolution.
training in S3 significantly boosting video capabilities, followed by further enhancements S4 through image-video training. This demonstrates that dynamic-resolution training enables model to process more native visual sequences, while the final unified training stage equips the model with generalized capabilities for handling diverse visual modalities. Notably, these findings remain highly consistent across the Uni R-0.AR---B/OO..O. B model family.
Zero ChastSicaton Performance Jenchmaris Dfffesent Traiting
Clestfictti Perforaace Benchmatis Oifienent B8A
34
50. 23.5
357
70.2
519
71.
15. 500
UniAATIA 046
Figure 4: Average performance improvement illustration across different training stages.
3.8
Verification of the effectiveness of hybrid training with DINOv2.
To investigate the performance benefits of incorporating DINOv2 as distillation branch in Stage-I training, we performed a comprehensive empirical study using the UniViTAR-0.3B model trained from scratch on 3B image-texi pairs. Checkpoints were evaluated at regular intervals throughout training in Table Our experiments revealed that DINOv2 distillation significantly accelerates early-stage convergence, after only 0.1B samples, distillation improved zero-shot ImageNet-1K performance by 17.3 points. Although this gain gradually diminishes as training progresses, remains observable at later stages. More importantly, the final model trained with DINOv2 distillation achieves an average improvement of 2. points across six zero-shot classification benchmarks compared to the baseline without distillation (Table 99). These results demonstrate that DINOv2 distillation not only speeds up early convergence but also enhances the final model performance.
Table 8: The performance gains of hybrid training on ImageNet- 1K as data increases.
Table 9: Zero-shot classification performance of hybrid training with DINOv2.
DIND12
0.5
1.5
DINOK2
O.Ne
26.8
67.00
6975
73333
71.9
UN
18.96
1078
03
44.15
08.15
71.44
73117
7421
76.00
77335
UNVITAR 033
Yes
72.84 311
75335
63.55
30.1
7040
60.65
10.11
19.50
313
56"
135
135"
17
131
250
708"
238
4 Conclusion
In this work, we introduce UniViTAR, family of homogeneous vision foundation models tai- lored for unified visual modality and native-resolution scenarios in the era of multimodal. By integrating advanced architectural upgrades, resolution curriculum learning, visual feature distil- lation, and inter-bat batch modality adaptation, UniViTAR achieves significant improvements across diverse tasks, spanning image/video zero-shot classificatio//retiieaal dense prediction accuracy, and vision- laaiuugl model transfer performance. Notably, all models are trained exclusively on public-acesssible datasets, where we observe consistent performance gains with parameter scaling from 0.3B to 1.4B. We hope that our ViTAR offers the community a versatile framework for advancing multimodal research.
10

---

References
[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words; Transformers for image recognition at scalc. arXiv preprint arXiv:2010.1 19999, 2020.
[2] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14496-14506 2023.
[3] Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n' pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36:2252-2274, 2023.
[4] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
[6] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image tra-iingn In Proceedings of the IEEF/CVF international conference on computer vision, pages 11975-11986, 2023.
[7] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[8] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397/ 2023.
[9] Chameleon Team, Chamelcon: Mixed-modal carly-fusion foundation models. arXiv preprint
arXiv:2405.09888, 2024.
[10] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos. Ibrahim Alabdulmohsin, et al. Scaling vision transformers to billion parameters. In International Conference on Machine Learning, pages 7480- 7512. PMLR, 2023.
[11] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021.
[12] Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24185 24198, 2024.
[13] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint 230.0230... 2023.
[14] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[15] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning, arXiv preprint arXiv:2212.03190 2022.
[16] Yi Wang. Kunchang Li. Xinhao Li, Jiashuo Yu, Yinan He. Guo Chen, Baoqi Pei. Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. European Conference on Computer Vision, pages 396-416 416. Springer, 2024.
[17] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurlPS, 2023.
11

---

[18] Wenzhe Shi, Jose Caballero, Ferene Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub- pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[19] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing ept4v-synthesized data for lite Aision-language models. arXiv preprint 240.::2002.168844 2024.
[20] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. SharegptAv: Improving large multi-modal models with better captions. arXiv preprint A.Ii:23111112991 2023.
[21] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyriis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36:27092-2777 2023.
[22] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Saehoon Kim. Coyo-700m: Image-text pair datasct, 2022.
[23] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NewrIPS, 35:25278 252294, 2022.
[24] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million mext pairs. arXiv preprint arXiv:2111.02111 2021.
[25] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pages 3558-3568, 2021.
[26] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In ACL, 2018
[27] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Fun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming-Hsuan Yang. et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13320-13331, 2024.
[28] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
[29] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang. Ping Luo, Ziwei Liu, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307. 230,06920 2023.
[30] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Decpspeed: System optimizations enable training deep learning models with over 100 billion parameters. In SIGKDD, pages 3505-3506. 2020.
[31] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
[32] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublincar memory cost. arXiv preprint arXiv: 1604::6004066777 2016.
[33] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. NeurlPS, 15:46494-1,355, 2022.
[34] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hicrarchical image database. In CVPR, pages 218-255,220 2009.
Imagenet: large-scale
[35] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2021.
12

---

[36] Dan Hendrycks, Steven Basart, Norman Mu. Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli. Mike Guo, Dawn Song. Jacob Steinhardt, and Justin Gilmer. The many faces robustness: critical analysis of out-of-distribution generalization. JCCV, 2021.
[37] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, pages 5389-5400, 2019.
[38] Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in neural information processing systems, 32, 2019.
[39] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled datasct for pushing the limits of object recognition models. NeurlPS, 32, 2019.
[40] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, pages 18123-18133 2022.
[41] Andrej Karpathy and Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137, 2015.
[42] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67-78, 2014.
[43] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, pages 740-755, 2014.
[44] Quan Sun, Yuxin Fang, Ledell Wu. Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. 15389, 2023.
[45] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar. Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi. and Ludwig Schmidt. Openelip. Zenodo. Version 0.1. httpp:///doi. org/ ort..///oorrr/..000/////ooooooo 5143773 2021. DOL: 10.5 12..28///2oodo554377777
[46] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zeulemoyer, and Christoph Feichtenhofer. Demystifying clip data, arXiv preprint arXiv:2309 16671, 2023.
Shang-Wen
[47] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. 174255 2023.
[48] Xianhang Li, Zeyu Wang, and Cihang Xic. Clipa-v2: Scaling clip training with 81.1% % zero-shot imagenet accuracy within $10000 budget; an extra $4000 unlocks 81.8% accuracy, arXiv preprint arXiv:2306. 15658, 2023.
[49] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. 15389, 2023.
[50] Will Kay, Joao Carreira, Karen Simonyan. Brian Zhang, Chloe Hillier, Sudheendra Vjjaanarasimhan. Fabio Viola, Tim Green. Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv preprint arXiv: 709500, 2017.
[51] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.04000 2012.
[52] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 201 International conference computer vision, pages 2556-2563. IEEE, 2011.
[53] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the iece conference on computer vision and pattern recognition, pages 961-970, 2015.
[54] Jun Xu, Tao Mci, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging vidco and language. In CVPR, pages 5288-5296 2016.
[55] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190 200, 2011.
13

---

[56] Mamshad Nayeem Rizve, Fan Fei, Jayakrishnan Unnikrishnan, Son Tran, Benjamin Z Yao, Belinda Zeng- Mubarak Shah, and Trishul Chilimbi. Vidla: Video- Ballo-language alignment at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14043-14055, 2024.
[57] Shen Yan. Tao Zhu, Zirui Wang. Yuan Cao, Mi Zhang. Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:22/2.04979, 2022.
[58] Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. arXiv preprint arXiv:2402. 13217, 2024.
[59] Ye Liu, Siyuan Li, Yang Wu, Chang-Werr Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9042-3051, 2022.
[60] Ziyun Zeng. Yixiao Ge. Zhan Tong, Xihui Liu, Shu-Tao Xia, and Ying Shan. Tvts22 Learning out-of- the-box spatiotemporal visual representations at scale. arXiv preprint arXiv:2305. 1..3005./77.., /4/73, 2023.
[61] Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal autoregressive pre-training of large vision encoders. arXiv preprint arXiv:241 1. 144402, 2024.
[62] Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07/59. 2020.
[63] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos. Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICML, pages 7480-7512, 2023.
[64] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 633-641, 2017.
[65] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning, arXiv preprint 2rXiv:2310.0374 2023.
[66] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang. Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi- modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11198-11201, 2024.
[67] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-worlh visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[68] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209, 2021.
[69] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, pages 1697-1706, 2022.
[70] Pan Lu. Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
[71] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317-8326, 2019.
[72] Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608-3617, 2018.
[73] Anand Mishra. Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
[74] Yuliang Liu, Zhang Li. Mingxin Huang, Biao Yang. Wenwen Yu, Chunyuan Li, Ku-Cheng Yin, Cheng- Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, (7(1)22200 2024.
14

---

[75] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang. Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[76] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi dustildis multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[77] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Gc, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16122 2023.
[78] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint 2rXiv:23 2023.
[79] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Part IV 14, pages 235-251, 251. Springer, 2016.
[80] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionhench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 14375-14385, 2024.
[81] Kun Zhou Jinpeng Wang Wayne Xin Zhao Yifan Li, Yifan Du and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[82] Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: A training-free free framework for high-resolution image perception in multimodal large language models. arXiv preprint, 2024.
[83] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-22222222 256. JMLR Workshop and Conference Proceedings, 2010.
[84] David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representation learning. arXiv preprint arXiv:2504.0m 200002504.000077 2025.
[85] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu. Ziteng Wang. et al. Cambrian- I: A fully open. vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems, 2024.
[86] Chenyang Si. Weihao Yu. Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. Inception transformer. arXiv preprint arxi::2205. 2022.
[87] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- languaee model with versatile abilities. arXiv preprint arXiv:2308.129%0 12966. 2023.
[88] Qihang Fan, Quanzeng You, Xiaotian Han, Yongfei Liu, Yunzhe Tao, Huaibo Huang, Ran He, and Hongxia Yang. Vitar: Vision transformer with any resolution. arXiv preprint arXiv:2403. /83,, 2024.
[89] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[90] Ting Chen, Simon Korablith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In International conference on machine learning, pages 1597-1607. PmLR, 2020.
[91] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9733 2020.
15

---

[92] Wenhui Wang. Hangbo Bao. Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu. Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal. Subhojit Som, al. Image as foreign language: Beit pretraining for vision and vision-language tasks. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19175 19175-19186. IEEE Computer Society, 2023.
[93] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[94] nannBastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21224, 2020.
[95] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[96] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[97] Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. arXiv preprint 06640000 2022.
[98] Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with cross modal sharing. In Proceedings of the IEEE/CVF winter conference on opplications of computer vision, pages 1236-1248, 2024.
[99] openai. Language models are unsupervised multitask learners. hello-gpt gpt hopt-4o/ 40/
hettps///penaa..commnae
2024.
[100] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang. Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03322
2024.
[101] Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang. Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16888 2023.
[102] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint 40Xiv:240..0376,, 2024.
[103] Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge. Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: Imm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, pages 390-406. Springer. 2024.
[104] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang, al. Deepseek-v12. Mixture-of- experts vision- language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302. 2024.
[105] Yuan Liu, Tian, Xiao Zhou, Xinyu Gao, Kavio Yu. Yang Yu, and Jie Zhou. Points1.5: Building a vision- language model towards real world applications, 2024.
[106] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, Jiasen Lu, Taira Anderson. Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Amavi Chheda, Jenna Sparks, Sam Skjonsherg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng. Jon Borchardi, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Witlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for ofate-of-thh-as vision- lisgua-language models, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. Our main contributions are also detailed in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We analyzed the performance gap with the pure video model the experiment section. In addition, the reduction in inference efficiency caused by native resolution is also mentioned the article.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiattinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Each experiment in the paper provides a detailed description of its setting, referring to Sec. 3 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We publicly- accessalee image-caption dataset. Once the blind review period is finished, we'll open-source code and model checkpoints.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
Moolic//guide//Coeeetuliiis casssiinnooyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppblcccuui.eeeeeeccc Sttelee//ooSSSSeissSioeaa www.ssssiicoooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training hyperparameters are introduced Table IT and the test details are presented along with the experimental results in Sec. 3 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This topic has not been reported with experimental statistical significance in other works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the training resources in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp::/neuuipp:: httpp////pppccccccccciiieeeeeeee
Answer: [Yes]
Justification: We have carefully reviewed the guidelines to ensure that our research strictly adheres to ethical standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This research primarily focuses on fundamental network structures, with no potential social harm.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not foresee any high risk for misuse of work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have carefully reviewed the guidelines and ensured adheres to the standards.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing, human subjects, or other related risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research does not involve crowdsourcing, human subjects, or other related risks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We have carefully reviewed the guidelines and ensured adheres to the standards.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

A The Overview of UniViTAR Pipeline
The proposed VniVI processes visual input its native resolution, and also supports scaling the resolution down or up while maintaining the aspect ratio or resizing to certain square size to accommodate different application scenarios, such as higher computational efficiency or finer-grained visual details. By treating video inputs as temporally extended image sequences, the framework uniformly produces longer variable-length visual token sequences.
z00m out
200m in
Native Resolution
Native Aspect Ratio
Square Resize
Video
Native Resolution
Native Aspect Ratio
Fatches
Fatches
Fatches
Patches
Pacches
UniviTAR Family
Figure 5: The brief illustration of UniViTAR f'amily pipeline.
B More Details of Training Recipe
Data Details. We collect public accessible image-text pairs and build our Merged-1 dataset, which
is composed of DataComp- IB 21]. COYO (2). LAION-2B [23]. LAION-400M [24]. DFN-2B (22]
CC12M (25) and CC3M 26 Moreover, to further enhance the video feature extraction capabilities of UniViTAR we meticulously constructed a dataset Merged-65M of roughly 65 million samples by randomly selecting video clips from three public accessible video datasets, i.e., Panda-70M [27].
Web 10M (28], and Intern Vid-10M 10M-FLT [29]. We refer to the combined image and video data mentioned above Merged- 1.IB. The detailed data composition is summarized in the Table I0
Table 10: Details of the training data for UniVITAR. Note that Merged- and Merged-65M correspond to image and video modality respectively.
Dataset
Source
Language
Samples
Total
Percentage
Used by
DataComp-I
21
En
408M
37.7%
COYO 2
En
248M
22.9%
LAION-2B
En
213M
19.7%
Merged- IB
DFN-2B 47
En
154M
1.08B
14.3%
Stage 1od
LATON-4000 400M
24
52.7M
4.9%
CC12M 26
2.94M
0.3%
CC3M 20
2.32M
0.2%
Panda- 70M
27
52.1M
80.2%
Merged-65M
WebVid-10M
28
65M
10.0%
Stage 4
Vidl Vid--O0M-PPP
6.31M
9.8%
Hyperparameter Details. The detailed hyperparameter configurations for each training stage are presented in the TTl As tabulated, we utilize a progressive reduction the peak learning rate in correlation with increasing visual backbone scale to ensure optimal training stability. Notably, learning rate of text branch in Stage 2 remains consistently one-tenth of the visual component throughout this phase. To enhance training efficiency, we integrated the DeepSpeed library (30 by
employing 7eRO optimizer sharding [1,, gradient checkpointing (32), and flash attention [33 Note all experiments are conducted on H800 GPUs.

---

Table 11: Detailed training hyperparameter of UniViTAR family. Note that the symbol of >> represents the peak learning rate and the minimum learning rate the LR schedule.
Stage
Stage
Stage
Stage 4
Vision Encoder Init.
Xavier init. 83
from Stage-1
from Stage=2
from Stage-3
Text Encoder Init.
LLama
[2
LLama [12]
from Stage=2
from Stuge-3
Input Resolution
224 x 224
224 x 224
Native
Native
Token Range
256
256
64 ~ 16K
64 16K
Global Batch Size
32768
32768
32768
(maage
Patch Dropout
0.5
0.0
0.5
0.5
Warmup Steps
2000
2000
2000
1000
Optimizer LR Schedule
Adam
AdamW
AdamW
AdamW
Cosine Decay
Cosine Decay
Cosine Decay
Cosine Decay 4e 6 0
0.3B
le + le-6
1e-$ + 0
1e-$ + 0
0.68
le le
le-6 +
le +
4e
1B
Se 1e-
6--6 6 +
6--6 +
2e
Train Dataset
Merged IB
Merged-1B
Merged-1B IB
Merged
-8, Mer
Seen Samples
12B
IB
0.6B
C More Experimental Results & Ablation Study
C.1 Verification of the effectiveness of image-video alternative strategy.
To validate the efficacy of the alternating image-video training strategy, conducted initial exper- iments with 100M image-text pairs and 10M video-text pairs. Note that the image-to-vides data ratio approximately 10:1, consistent with the ratio used in stage 4 of the UniViTAR series. We trained a ViiVAAA---.33 model for 3 epochs, comparing mixed training and alternating training strategies. As shown in Table [2] the alternating training strategy outperforms the mixed strategy across key image and video benchmark metrics, demonstrating its effectiveness in enhancing visual representation learning. This performance gain can be attributed to the increased training difficulty arising from the unification of data modalities within each batch.
Table 12: Zero-shot classification performance of image-video training strategy.
Strategy
ImageNet-IK
ImageNet-A
K400
UCF101
Batch-Mixed
70.46
45.89
58.82
75.15
Batch-Alternative
71.25
48.60
61.01
77.66
C.2 Verification of the effectiveness of native resolution for video.
In this section, we conduct an ablation study to explore the role native resolution in video data processing. We dynamically sample maximum of 32 frames (denoted as F) for each video clip. For frames exceeding the sequence length limit, we resize them while preserving their native aspect ratio to smaller resolution. We evaluate 15 maximum video sequence length, ranging from 1024 to 65,536, and test the zero-shot classification performance of ViTAR-O. on the K400 dataset. Note that the minimum video sequence length is fixed to 576. As shown in Figure 6 the performance initially improves and then stabilizes as the sequence length limit increases, reaching a plateau at length 10.240. We attribute this to the fact that, with 32 sampled frames, sequence length of 10,240 corresponds to a resolution of 490 x 256, enabling most frames in K400 to retain their native resolution during data processing. This finding underscores the importance of native resolution in enhancing video understanding capabilities.
C,3 Verification of the effectiveness of data scale.
From an intuitive perspective, data scale has a significant impact on the effectiveness of contrastive learning. In this section, we conduct cold- start experiments on UniViTAR-0.3B to confirm this view. For the experiment setup, seen samples is fixed at IB. We respectively train the UniVITAR-0.3B for epoch using Merged-1B IB and for 10 epochs using Merged- 100M, which contains 100M image-text pairs that randomly sampled from Merged- IB. Result zero-shot classification and retrieval is
25

---

15.06
15/9
9948
VWW
V50
10005
15000
686
67:47
6347
17:44
6044
6747
6024
3044
6744
80
64
1047
62
55.3
85
2.00
010
Manimarn
Bcre Sare-Bhat Chrallalli Aocurary (000000
Nuttre Puerse High Butio TuAal Vidkos Clipo
Figure Performance changes on the K400 dataset across varying sequence length limits.
shown in Table 13 There is observable trend where the performance improves as the dataset scale increases. With larger dataset scale, the model is exposed to a broader range of image-text pairs, facilitating more comprehensive learning and understanding of the visual and linguistic space, thereby enhancing zero-shot performance.
Table 13: Ablation results of UniViTAR-0.33 under varying data scale.
ImageNet Variants
Flickr
COCO
Data
Seen Samples
Overall
Overall
IN-IK
IN-A
IN-R IN-V2 IN-S
O-Net
T-1
I-TT
T-1 I-T
Merged- 100M Merged- IB
IB
60.8
69.7
39.6
79.0
61.6
54.8
60.2
67.0
73.4
88.3
44.2
62.0
IB
64.2
71.7
45.7
82.3
64.3
57.3
63.9
68.9
74.9
90.7
46.0
63.8
C.4 Verification of the effectiveness of LLM scale.
To verify the model's effectiveness across language models varying scales, we integrated UniViTAR-0.6B with progressively larger language backbones, specifically, Qwen2.5-1.5BB, 3B, and 7B, within a multimodal large language model (MLLM) framework. As the model scale in- creases, models achieve average scores of 48.7, 51.9, and 54.6, respectively, across 16 multimodal benchmarks. The experimental results demonstrate consistent scaling behavior, confirming UniVi- TAR's strong compatibility and performance potential when combined with larger language models. These findings support the conclusion that the UniViTAR architecture possesses promising scalability.
C.5 Comparative analyses of additional relevant visual foundation models.
To provide comprehensive comparison with other prominent visual encoders, we conduct system- atic analysis of several relevant visual foundation models. (1) NaViT 3 Quantitative evaluations classification tasks demonstrate that UniViTAR-0.38 exhibits clear advantages over NaVIT. (2) Flexi VIT 22: Introduced in the main text, FiexiViT supports dynamic patch size to handle variable-resolution inputs. We include supplementary linear-probing results on ImageNet variants Table 14 for direct comparison. (3) Web-DINO 84) This self-supervised model shows that scaling data and parameters can approach CLIP-level performance: however, noticeable gap remains relative to CLiP-based paradigms, as indicated by model size (7B vs. 0.3B) and benchmark performance. Preliminary comparisons provided in the accompanying Table [4] (4) Cambrian-1 [85] This vision-centric MLLM family uses a Mixture-of-Featuuee multiple visual encoders reduce information loss. While effective, MoF introduces higher computational costs and integration complexity compared to unified models like ViTAR.
in Table 14 on linear-probie
(MoF) scheme
Table 14: Comparison of UniViTAR and other vision encoder on linear-probing classification.
Model
Pretrain Data
Train Paradigm
IN-IK
IN-A
IN-Real
IN-V2
IN-S
IN-R
NaVIT-L
JFT4B ImageNet-I1 K
Supervised Learning
76.0
65.5
FlexiViT-L.
Supervised Learning
86.
34,1
90.0
76.7
41.2
Web-DINO-7B
MC-2B
Self Sulf-Superiieed Learning
86.4
UniViTAR-O.3B
Merged-J
Contrastive Learning
84.1
90.3
79.5
66.0
26

---

D Related Work
D.1 Flexible Vision Transformers
Vision Transformers have showcased impressive performance in numerous visual tasks, such as image classification M. image language pre-training [TT], etc. Those methods work only a single, fixed resolution. Some works [86] 87) attempt to meet the need for fine-grained visual representation by adapting the model to higher resolution during the fine-tuning stage. However, directly resizing the input to a fixed square resolution still limits their representation capacity in diverse visual scenarios. Recently, there are some works in vision transformers attempting to accommodate images with native resolutions with variable aspect ratios. VITAR [8 proposes an adaptive token merger module to alleviate the constraints of fixed resolution and adapt to multi-resolution inputs. However, still limited by predefied number of tokens that the model ultimately aims to obtain. Va [ introduces sample packing used in language modeling for handling variable sequence length of image patches. Mcanwhile, it introduces factorized positional embedding schema in vanilla ViT to support variable aspect ratios and extrapolate to unseen resolutions. Qwen2.5-VL [5] integrates an Na ViT-like approach to support native input resolutions, and employs multiple training phases for adapting it to multimodal large languages models, including CLIP pre- wee-traiinng, vision-language alignment, etc.
D.2
Vision Foundation Models
The development of vision foundation models has progressed through distinct phases, beginning with supervised learning paradigm dominated by landmark architectures like ResNet [89] and ViT [ which established performance benchmarks through reliance on labeled data. However, the field witnessed a paradigm shift with the rise of self-supervised learning, which circumvented annotation bottlenecks through three principal branches: contrastive learning frameworks like SimCLR [90] and MoCo 1T], masked image modeling methods such as BETT [92 and MAE 933. and ceff-distillation techniques including BYOL [94] and DINO [95]. Recently, language-supervise
contrastive pre- training has emerged as a transformative paradigm, exemplified by CLIP (961, which aligns multi- modal embeddings through noise-robust contrastive objectives, enabling zero-shot task generalization. This approach has been further refined in works like SigL.IP [6, which employs a more efficient igmoiidbase loss function while preserving cross-modal transfer capabilities. Besides images, robust visual foundation model with effective video alignment capabilities serves as another critical building block. The existing strategies for training such models can be classified into three main paradigms: training on video-only data (56 58 13 97], utilizing multimodal data encompassing both and incorporating multimodal data that integrate video, images, audio 98 VideoPrism (58) employs a two-stage video-only pretraining strategy: contrastive learning followed by token distillation, yet lacks image understanding. VidLA (56) adapts CLIP (96) via spatio-temporal attention on video-text data. Intern Video [15] combines masked video modeling with alternating video/image-text pretraining, enhanced by cross-modal attention, while Intern Video2 [16] extends this framework with audio/speech modalities for multimodal alignment.
video and image [13
87
5
and other modalities 16
D.3 Multimodal Large Language Models
Recently, multimodal large language models (MLLMs) have witnessed significant advancements and rapid development [99] 17 87
1o1 [02]. As a critical modality in MLLMs, visual input encounters inherent limitations when relying on conventional ViT with fixed resolutions, which may induce shape distortions, content blurring, and suboptimal handling of images/videon with diverse aspect ratios, high resolutions, or dynamic frame rates. To mitigate these challenges, the field has converged on two principal technical directions: 1) The blinng-based paradigm, as adopted by models like [103] 100 12 104, decomposes hitra--hgh-rrsolutton inputs into a varied number of fix-resolution tiles, and each tile is processed by fixed-resolution vision encoder. As such, it enables MLLMS adaptivity to dynamic-resolution images without padding or shape-distorting resizing. However, the tile limits the model's ability to capture spatial information across different tiles and the primary subjects of the images are often fragmented, leading to the loss of spatial relationships and quantitative information. 2) native-resolution methodology, exemplified by models such as 44, [055 [066, attempts to circumvent the limitations of the tiling-based paradigm by using native resolution input. However, they typically employ a pretrained fixed- reed-resolution vision transformer as vision encoder, which leads to additional costs associated with adapting the VIT's distribution.
5 I2
27


Reviews:
[{"review_id": "nz9ccdY148", "reviewer": "Reviewer_dNd7", "summary": "The paper introduces UniViTAR, a family of homogeneous vision foundation models designed to process native-resolution visual data (images and videos) uniformly, addressing limitations of conventional fixed-resolution Vision Transformers (ViTs). It introduces architectural enhancements (like 2D Rotary Positional Embeddings, SwiGLU, RMSNorm, and QK-Norm) and a progressive training strategy combining curriculum learning, modality alternation, and contrastive learning with feature distillation. Trained on public datasets, UniViTAR scales from 0.3B to 1.4B parameters and achieves strong performance across zero-shot image/video classification, retrieval, and multimodal understanding tasks.", "questions": "How does UniViTAR differentiate itself from prior native-resolution approaches like NaViT and Qwen2.5-VL?", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Open-sourcing benefits the community. Training on publicly accessible datasets and releasing models/code promotes reproducibility and encourages further research.\n\n2. UniViTAR achieves competitive or superior results on a broad range of benchmarks across image, video, and multimodal tasks.\n\n3. The integration of advanced components (e.g., 2D-RoPE, SwiGLU, QK-Norm) and the staged curriculum (e.g., resolution and modality progression) shows a well-engineered and scalable framework.\n\nWeaknesses:\n1. While the paper emphasizes native resolutions, existing models like Qwen2.5-VL already adopt native resolution strategies, reducing the novelty of the contribution.\n2. The multimodal performance (Table 6) is relatively modest, and extensions to large-scale MLLMs remain under-explored.\n3.  The paper would benefit from more granular ablation studies on the architectural choices (e.g., LayerScale, RMSNorm) and their impact across both classification and multimodal tasks.", "was_revised": true, "final_justification": "I'm not totally convinced about the novelty of their native-resolution idea. That said, the paper makes meaningful engineering contributions, and I would recommend it for a borderline accept.", "rebuttal": "> **Weakness 1. While the paper emphasizes native resolutions, existing models like Qwen2.5-VL already adopt native resolution strategies, reducing the novelty of the contribution.**\n\n> **Question 1. How does UniViTAR differentiate itself from prior native-resolution approaches like NaViT and Qwen2.5-VL?**\n\nWe sincerely appreciate the reviewer’s insightful perspective regarding native-resolution approaches and their valid inquiry into UniViTAR’s differentiation. We consider Qwen2.5-VL and NaViT to be significant as previous implementations of native-resolution processing. However, UniViTAR introduces ​several core innovations​​ that fundamentally differentiate it from these prior efforts in terms of both technology and functionality.\n\n**Comparison with NaViT​**​. As an early exploratory work, NaViT's core contribution lies in introducing a native-resolution token packing training strategy to Vision Transformers (ViTs), thereby pioneering efficiency improvements in native-resolution training. While its insights indeed provide foundational inspiration for contemporary native-resolution models, UniViTAR demonstrates distinct advantages:\n\n- ​**​Scalability and Homogeneity​**​. UniViTAR offers larger and more diverse model scales (0.3B/0.6B/1.4B), with all models sharing structural and training homogeneity across sizes and data sources. This facilitates granular community research into scaling both vision foundation model and multimodal large language model.\n- ​**​Flexible Native-Resolution Training Strategy​**​. NaViT primarily employs greedy token packing with sequence padding to fixed lengths during training. In contrast, UniViTAR adopts a random packing strategy combined with batch-wise sequence resizing (#L149~#L153).\n- ​**​More Visual Modality​**​. UniViTAR employs a multi-stage training paradigm (image pretraining → image-video joint training), achieving state-of-the-art performance on both image and video benchmarks. NaViT, limited to image-only pretraining, underperforms many CLIP-based counterparts.\n- ​**​Advanced Architectural Components​**​. For transformer-centric native-resolution modeling, adaptive positional encoding is essential: while NaViT uses factorized positional embeddings, UniViTAR implements 2D-RoPE to encode visual sequences with variable lengths, critically enhancing positional awareness.\n\n**Comparison with Qwen2.5-VL-ViT​**​. As a leading open-source Vision-Language Model (VLM), Qwen2.5-VL similarly adopts a native-resolution multimodal training strategy akin to NaViT. UniViTAR demonstrates the following distinctive advantages:\n\n- **​Purer Native Dynamic-Resolution Representation​**​. Qwen2.5-VL’s vision encoder undergoes extensive multimodal training (primarily on in-house data) across multiple multimodal stages (multimodal pretraining → post-training), resulting in representations highly adapted with Qwen-series LLMs. This alignment substantially constrains ​generic visual transferability​. UniViTAR, conversely, employs contrastive learning only for native-resolution pretraining, delivering superior generic visual representations.\n- **​Granular Training Recipe and Model Scalability​**​. While Qwen2.5-VL releases pretrained VLM weights, it withholds the visual pretraining weights in CLIP phase and critical implementation specifics. UniViTAR systematically documents training methodologies and releases multi-scale models (0.3B/0.6B/1.4B VS. 0.6B), establishing a comprehensive research foundation for the community.\n- ​**​Architectural Distinction in Attention Mechanism​**​. UniViTAR utilizes ​​Full-attention​ exclusively, whereas Qwen2.5-VL employs a hybrid ​​Full-attention and Window-attention​ design. Although window-attention improves computational efficiency, it typically incurs non-trivial performance degradation in visual representation quality. Considering the versatility of downstream tasks and the fact that the scale of language models is usually much larger than that of visual encoders, UniViTAR uses Full-Attention to preserve the visual representation capabilities as much as possible. However, improving the training and inference efficiency of visual encoder at native-resolution will be our next important work.\n\n> **Weakness 2. The multimodal performance (Table 6) is relatively modest, and extensions to large-scale MLLMs remain under-explored.​**\n\nWe sincerely appreciate the reviewer’s insightful observation regarding the multimodal performance in Table 6 and the need for large-scale MLLM exploration. Due to the research-level multimodal traning data, we acknowledge that UniViTAR’s current multimodal results are not state-of-the-art, as our primary focus was rigorously validating the ​**​generic visual representation quality​**​ of dynamically scaled vision encoders and other state-of-the-art vision encoders—rather than optimizing end-to-end MLLM systems.\n\nTo address reviewer's concren, we have conducted new experiments scaling UniViTAR to larger MLLM frameworks, e.g., integrating UniViTAR-0.6B with Qwen2.5-1.5B/3B/7B. Results demonstrate clear scaling laws in the table below, which confirms UniViTAR’s scalability potential when paired with larger language models.\n\n| ​LLM Scale​| ​Avg. on overall 16 benchmarks​ |\n| ------| :-----:|\n|UniViTAR-0.6B + Qwen2.5-1.5B| 48.7|\n|​​UniViTAR-0.6B + Qwen2.5-3B ​  | ​​51.​9​ |\n|​​UniViTAR-0.6B + Qwen2.5-7B ​  | 54.6 |\n\nExploring UniViTAR's generalization capabilities on larger language models (​*e.g.*​, 70B parameters) and expanded data scales constitutes our critical next work to further validate the versatility of the UniViTAR family. We thank the reviewer for highlighting this gap—it directly shapes a more impactful contribution.\n\n\n> **Weakness 3. The paper would benefit from more granular ablation studies on the architectural choices (e.g., LayerScale, RMSNorm) and their impact across both classification and multimodal tasks.**\n\nWe sincerely thank the reviewer for the constructive feedback regarding the need for more granular ablation studies. We acknowledge that deeper analysis of architectural components is critical for understanding UniViTAR’s design efficacy. To address this, we established a validation baseline based on UniViTAR-0.3B, with the overall training size controlled at 0.5B scale (Merge1B for 0.5 epoch). We validated the effectiveness of the three modules, LayerScale, RMSNorm, and SwiGLU. The specific results are as follows:\n\n|  Model (@Zero-Shot Classification)      | Norm Layer | FFN Layer | LayerScale | Avg. | IN-1K | IN-A |IN-R |IN-V2 |IN-S |O-Net |\n|------------|:------------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| UniViTAR-0.3B (baseline) |  LayerNorm | MLP | No |58.25|68.12|33.47|78.09|59.67|53.40|56.76|\n| UniViTAR-0.3B | **RMSNorm**| MLP| No | 59.00|68.26|34.23|78.82|60.92|54.21 |57.57|\n| UniViTAR-0.3B |  LayerNorm | **SwiGLU** | No|60.00|**68.93**|36.63|79.74|60.94|55.03|**58.71**|\n| UniViTAR-0.3B | LayerNorm | MLP |**Yes**|58.58 |68.16|34.39|77.66|60.62|52.85| 57.81|\n| UniViTAR-0.3B | **RMSNorm** |**SwiGLU**|**Yes**|**60.59**|68.89|**37.37**|**81.05**|**61.70**|**56.16**|58.40|\n\nThe ablation studies summarized in Table yield the following conclusions:\n\n- All three investigated modules—SwiGLU, RMSNorm, and LayerScale—enhance visual representation capabilities, albeit to varying degrees. SwiGLU yields the most substantial gains (+1.75%), followed by RMSNorm (+0.75%), and LayerScale (+0.33%). Their synergistic integration delivers cumulative performance improvements (+2.34%), confirming complementary benefits.\n- SwiGLU and RMSNorm, widely adopted in large language models (*e.g.*, LLaMA), demonstrate comparable efficacy in visual representation learning. This validates their generalizability as universal architectural components across modalities.\n\nThese ablation results and associated discussions will be incorporated into the revised manuscript to clarify the impact of individual design choices."}, {"review_id": "HGX5wixipe", "reviewer": "Reviewer_G79o", "summary": "This paper proposes UniViTAR, a family of unified Vision Transformer backbones that support native resolution and both image/video modalities, aiming to serve as a solid visual encoder for multimodal large models. The key contribution is not in architectural novelty, but in the systematic integration of community-accepted best practices—such as 2D Rotary Position Embedding, dynamic patch tokenization, RMSNorm, QK-Norm, and progressive training strategies including staged curriculum from fixed to native resolution, as well as inter-batch image/video alternation. The authors conduct extensive and rigorous experiments on public datasets, covering zero-shot classification, retrieval, linear probing, and dense prediction tasks, consistently demonstrating strong performance scaling across model sizes.", "questions": "1. What concrete differences (in architecture, training, or results) set UniViTAR apart from Qwen2VL's vision encoder? A table would help clarify any unique contributions.\n\n2. How much benefit can be gained by using Dinov2 for distillation in Stage 1?\n\n3. On vision-language benchmarks, the performance gain of scaling the vision encoder from 0.6B to 1B seems to be limited. What is the author's perspective on this phenomenon?", "limitations": "See the weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. The implementation is solid, with carefully staged training, systematic ablation studies, and thorough benchmarking across diverse visual tasks (images, video, multimodal VQA, dense prediction, etc.).\n\n2. The methodology, training details, and experimental setups are clearly described, with extensive tables and analyses covering key baselines, scaling trends, and ablation results.\n\n3. The work serves as a robust “best practices” summary for engineering strong vision backbones for multimodal LLMs, and the systematic experiments further validate the effectiveness of these recipes.\n\n# Weaknesses\n\nThe core architectural design and training strategy closely follow established approaches in the community, especially Qwen2VL and NaViT. The work mainly consolidates community best practices rather than proposing new model components or learning algorithms.", "was_revised": true, "final_justification": "The author’s response addressed my questions well. I think this is a fairly solid piece of work, so I will keep my score of 4.", "rebuttal": "> **Weakness 1. The core architectural design and training strategy closely follow established approaches in the community, especially Qwen2VL and NaViT. The work mainly consolidates community best practices rather than proposing new model components or learning algorithms.**\n\n> **Question 1. What concrete differences (in architecture, training, or results) set UniViTAR apart from Qwen2VL's vision encoder? A table would help clarify any unique contributions.**\n\nWe sincerely appreciate the reviewer’s perspective regarding UniViTAR’s consolidation of community best practices and value their constructive feedback. Pioneering works such as NaViT introduced native-resolution training schemes for Vision Transformers (ViTs), exploring efficiency improvements in native-resolution processing. Similarly, Qwen2VL’s vision encoder adopted NaViT’s principles by leveraging pretrained fixed-resolution visual foundation model (​*e.g*.​, DFN[1]) to advance multimodal representation alignment and pretraining for large multimodal models. In our response to ​**​Reviewer dNd7​**​, we comprehensively contrasted UniViTAR’s distinctions and advantages relative to NaViT and Qwen2.5-VL’s vision encoder. To avoid redundancy, we kindly invite you to refer to the relevant sections of our reply to **Reviewer dNd7** for more detailed comparative analyses. Below, we explicitly delineate key differences between the UniViTAR framework and Qwen2-VL’s vision encoder:\n\n| Model          | Qwen2-VL-ViT               | UniViTAR family                                                         |\n| -------- | -------- | ---- |\n| Architecture        | Inherit from DFN [1]      | Redesign based on the latest advanced modules                   |\n| Model size          | 0.6B                       | **0.3B, 0.6B, 1.4B**                                          |\n| Norm layer          | LayerNorm                  | RMSNorm                                                           |\n| FFN layer           | Vanilla MLP                        | SwiGLU                                                            |\n| QK norm             | No                         | Yes                                                               |\n| **Training**     |  |                       |\n| Pretrain weight     | Load from DFN          | **Scratch training**                                                  |\n| Supervised paradigm | Next text token prediction | Image-Text & Video-Text contrastive learning                                   |\n| Train data          | In-house-Data              | One billion from opensource data                                    |\n| Batch strategy      | Sequence Padding           | Sequence Resizing (See Section 2.2.2)                             |\n| Resolution strategy | Native-resolution training | Pretrained on fixed-resolution and finetuned on native-resolution |\n\n\n> **Question 2. How much benefit can be gained by using Dinov2 for distillation in Stage 1?**\n\nThank you for your perceptive and highly professional inquiry. We did conduct a thorough investigation into the performance gains offered by the distillation branch during the initial model exploration phase. The following summarizes our empirical observations and experimental findings:\n\n-  ​**Acceleration of early-stage convergence**.​  Our experiments on UniViTAR-0.3B involved training on 3B image-text samples from scratch. Evaluating checkpoints at different training intervals (every 0.5B samples) within the same training cycle revealed that utilizing Dinov2 for distillation significantly enhances convergence during the early training phase. For instance, at 0.1B samples, distillation supervision yielded a performance gain of 17.3 points on the ImageNet-1K zero-shot classification. This beneficial effect diminishes progressively as training advances, as detailed in the subsequent comparison.\n\n|        Model       | Dinov2 | 0.1B | 0.5B | 1.0B | 1.5B | 2.0B |2.5B |3.0B |\n|:--------------------:|:---------------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| UniViTAR-0.3B-Stage1 |    No      | 26.88|63.71|67.80|69.75|72.32|74.88|75.72|\n| UniViTAR-0.3B-Stage1 |  Yes  | 44.18|68.15|71.24|73.17|74.71|76.40|77.33|\n| *Performance Gain*        |                    | *17.30*|   *4.44*|  *3.44*|  *3.42*|  *2.39*|  *1.52*|  *1.61*|\n\n- ​**​Improvement in final model performance**.​ Evaluation of the final model trained on 3B image-text pairs demonstrated that employing Dinov2 as the distillation branch consistently delivers an average performance gain of 2.1 points across six zero-shot classification benchmarks. The detailed comparative analysis is presented below.\n\n|        Model       | Dinov2 | Seen | Avg. | IN-1K | IN-A |IN-R |IN-V2 |IN-S |O-Net |\n|:--------------------:|:---------------:|:------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| UniViTAR-0.3B-Stage1 |    No      | 3.0B |70.73|75.72|58.76|87.98|68.22|62.95|70.78|\n| UniViTAR-0.3B-Stage1 |  Yes     | 3.0B |**72.84**|77.33|63.55|89.81|70.40|65.65|70.31|\n\n> **Question 3. On vision-language benchmarks, the performance gain of scaling the vision encoder from 0.6B to 1B seems to be limited. What is the author's perspective on this phenomenon ?**\n\nWe sincerely appreciate this insightful observation regarding the scaling behavior of vision encoders in vision-language models. As the capacity of the visual encoder increases, we observe that the VLM model's performance on established vision-language benchmarks becomes increasingly **constrained by factors other than pure visual representation power**. We would like to humbly submit the following perspectives on this phenomenon:\n\n- ​**​Cross-modal Alignment Limits**.​ Larger vision encoders inherently possess greater potential for highly specialized visual feature extraction. However, the capacity and optimization of the modality fusion mechanism may become a bottleneck, limiting the utility of richer visual features.\n- ​**Late-Fusion[2] Architecture Bound**.​  Current late-fusion methodologies typically entail integrating independently pre-trained modality-specific models, such as connecting vision encoders to LLMs and continuing multimodal training. Crucially, each constituent module (*e.g.*, vision encoder, LLM) exhibits distinct hyperparameters, distinct pre-training data mixtures, and distinct scaling properties with respect to computational resources and training data volume. Exploring the joint scaling strategy of visual encoders and LLMs may be the most necessary point in the future work.\n- ​**​Limited Train Data and Benchmark Saturation**.​ Our models in Table 6 were trained on a comparatively limited multimodal dataset, *i.e.* research-level, consisting of only 2.5M image-caption pairs and 665K instruction tuning samples. Furthermore, the visual encoder parameters remained frozen throughout training for more accurately verifing the ability of pre-trained visual representation. This configuration resulted in diminishing performance gains relative to increasing visual encoder capacity. On the other hand, some benchmarks may lack the granularity or difficulty to fully reflect the nuanced improvements gained from larger vision encoders, leading to apparent saturation.\n\n\n*[1]. Fang, Alex, et al. \"Data filtering networks.\" *arXiv preprint arXiv:2309.17425* (2023).*\n\n*[2]. Shukor, Mustafa, et al. \"Scaling laws for native multimodal models.\" *arXiv preprint arXiv:2504.07951* (2025).*"}, {"review_id": "dv04uUz4AN", "reviewer": "Reviewer_EmmT", "summary": "This paper proposes UniViTAR, a family of Vision Transformer models designed to process images and videos at their native resolution and aspect ratio, addressing limitations of conventional fixed-resolution ViTs. \nThe architecture integrates modern components such as 2D RoPE, SwiGLU, RMSNorm, and QK-Norm, and employs a progressive training strategy that transitions from fixed to native resolution and from image to video modalities. \nA hybrid loss combining sigmoid-based contrastive learning and feature distillation accelerates training. \nTrained solely on publicly available image-caption data, UniViTAR achieves strong performance across a range of visual and multimodal benchmarks.", "questions": "None", "limitations": "No\nThe paper does not discuss its own limitations, although mentioned in check list. For a model targeting native resolution and multimodal understanding, it would be valuable to acknowledge potential weaknesses such as computational cost, scalability, or video performance gaps.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1.This paper proposes a native resolution Vision Transformer that enables effective feature extraction from images of varying resolutions. It adopts a simple yet effective progressive training strategy, and demonstrates strong performance across image classification, video understanding, and multimodal tasks.\n\n2. This paper is well-written and easy-to-follow.\n\n**Weakness**\n\n1.One limitation of the paper is the lack of discussion on related work that tackles similar problems using different architectures. For example, causal convolutions, which are widely used in video generation, have also been applied in encoding tasks for both images and videos, and naturally support variable-length or multi-resolution inputs. Similarly, diffusion transformer (DiT) models for image and video generation share many architectural elements with UniViTAR, such as Transformer backbones with RoPE, QK-Norm, and progressive resolution handling. While these models target generative tasks, they address similar structural and resolution challenges.\n\n2. While positioning UniViTAR as a vision encoder for MLLMs is arguably the most impactful application of this work, the comparison is limited and outdated. The paper mainly contrasts with earlier vision encoders, but omits recent models that specifically support native resolution, such as NaViT and FlexiViT. Furthermore, several recent works focused on multimodal understanding—like Scaling Language-Free Visual Representation Learning and Cambrian-1—propose advanced vision encoders tailored for MLLMs.", "was_revised": true, "final_justification": "my concerns are well addressed.", "rebuttal": "> **Weakness 1. One limitation of the paper is the lack of discussion on related work that tackles similar problems using different architectures. For example, causal convolutions, which are widely used in video generation, have also been applied in encoding tasks for both images and videos, and naturally support variable-length or multi-resolution inputs. Similarly, diffusion transformer (DiT) models for image and video generation share many architectural elements with UniViTAR, such as Transformer backbones with RoPE, QK-Norm, and progressive resolution handling. While these models target generative tasks, they address similar structural and resolution challenges.**\n\nWe sincerely thank the reviewer for this insightful and valuable comment regarding the scope of the related work discussion. Expanding our discussion to encompass these approaches would have significantly strengthened the context of our work.\n\n- ​**​Discussion of Causal Convolutions​**.​ We agree that architectures employing ​causal convolution​ represent an important alternative strategy for handling sequential data, including images and videos, particularly due to their inherent support for ​variable-length inputs​​ and multi-resolution representations. While our work primarily focuses on the quality of visual semantic-level representations and their generalization capabilities in multimodal understanding scenarios, adopting the alternative approach of causal convolutions would introduce multiple substantial limitations:\n  \n  - **Loss of Global Context Modeling**. Bidirectional attention in UniViTAR (*transformer-centric* paradigm) dynamically computes relationships between all  visual tokens, enabling holistic understanding of scene context and object interactions. Causal convolutions' locality fundamentally restricts context aggregation to receptive fields defined by kernel size/depth.\n  - **Suboptimal Static Image Processing**. For non-sequential visual inputs (static images), causal convolutions impose an artificial, task-irrelevant directional bias that has no perceptual justification (​*e.g*.​, prioritizing left patches over right). This conflicts with the isotropy of natural images.\n  - **Representational Capacity Constraints**. While deep stacks of dilated convolutions can increase receptive fields, they remain fundamentally less expressive than attention mechanisms for capturing complex, long-range token dependencies. This may necessitate significantly deeper architectures to compensate, diminishing potential efficiency gains.\n- ​**​Discussion of Diffusion Transformers (DiTs):​** ​While DiTs and UniViTAR share some  architectural elements, we note DiT's primary application domain is ​*generative modeling*​, whereas UniViTAR is primarily designed for ​*discriminative representation learning*​. This key distinction in objective leads to substantial differences in training regimes, loss functions, and final application. Moreover, it is crucial to emphasize that the core contribution of UniViTAR lies in its complete training scheme for native resolution-based Vision Transformers in the open-source community and a family of homogeneous visual foundation models for multimodal undersanding research.\n\n> **Weakness 2. While positioning UniViTAR as a vision encoder for MLLMs is arguably the most impactful application of this work, the comparison is limited and outdated. The paper mainly contrasts with earlier vision encoders, but omits recent models that specifically support native resolution, such as NaViT and FlexiViT. Furthermore, several recent works focused on multimodal understanding—like Scaling Language-Free Visual Representation Learning and Cambrian-1—propose advanced vision encoders tailored for MLLMs.**\n\nWe sincerely appreciate the reviewer’s valuable feedback. Below, we provide comparative analyses of additional relevant visual foundation models referenced in your comments.\n\n- **NaViT​**[1]. By introducing a token-packing strategy, NaViT pioneered the application of native-resolution training schemes in conventional Vision Transformers, concurrently exploring efficiency improvements in native-resolution processing. Detailed distinctions and advantages of UniViTAR relative to NaViT are comprehensively addressed in our response to **Reviewer dNd7**. To minimize redundancy, we refrain from reiterating these points here and kindly refer the reviewer to that discussion. For quantitative validation, we present performance comparisons between UniViTAR and NaViT on linear-probing classification, demonstrating UniViTAR-0.3B’s significant advantages.\n\n|Model (@Linear-Prob)|Pretrain Data|Train Paradigm|IN-1K|IN-A|IN-Real|IN-V2|IN-S|IN-R|\n|:----:|:-----:|:----:|:---:|:---:|:---:|:----:|:----:|:----:|\n|NaViT-L| JFT4B | Supervised Learning|76.0|65.5|-|-|-|-|\n|UniViTAR-0.3B| Merged-1B |Contrastive Learning|**87.6**|**84.1**|90.3|79.5|66.0|90.6|\n\n- **​FlexiViT​**​[2]. This work is referenced in the introduction section of our main text. FlexiViT dynamically adjusts patch sizes to generate variable-length visual token sequences, accommodating both fixed and native-resolution inputs. We acknowledge its exclusion from our primary comparative analysis and clarify two key reasons:\n  \n  - ​**​Model Scale​**​: FlexiViT’s published results predominantly utilize ViT-B architectures, while UniViTAR’s smallest variant (UniViTAR-L) operates at approximately 300 million parameters, rendering direct performance comparisons inequitable.\n  - **​Training Paradigm​**​: FlexiViT-L employs supervised pretraining on ImageNet, whereas UniViTAR adopts large-scale contrastive pretraining without ImageNet data. This fundamental methodological discrepancy precludes rigorous equivalence in comparative evaluation.\n  - Nevertheless, we still provide supplemental comparisons in the following table: one is the comparison results of linear-probing on ImageNet variants (Table 1), and the other is the results of the model on the zero-shot classification and cross-modal retrieval when FlexiCLIP-B at patch-size 8 (Table 2).\n\n|Model (@Linear-Prob)|Pretrain Data|Train Paradigm|IN-1K | IN-A |IN-Real |IN-V2 |IN-S |IN-R|\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| FlexiViT-L| ImageNet-1K| Supervised Learning|86.1|34.1|90.0|76.7|-|41.2|\n| UniViTAR-0.3B| Merged-1B |Contrastive Learning|**87.6**|**84.1**|**90.3**|**79.5**|**66.0**|**90.6**|\n\n|Model (@Zero-Shot)| IN-1K |Flickr30K T2I |Flickr30K I2T |COCO T2I |COCO I2T |\n|:----:|:----:|:----:|:----:|:----:|:----:|\n| FlexiCLIP-B/8|60|57|75|30|49|\n| UniViTAR-0.3B  |**81**|**84**|**95**|**55**|**71**|\n\n* **Web-SSL​**[3]. Web-DINO explores visual self-supervised learning paradigms, demonstrating that scaling both data volume and model size beyond the billion-scale can yield performance approaching that of CLIP-trained vision foundation models. However, a non-trivial performance gap persists between Web-DINO and CLIP-paradigm representations, as evidenced by model parameter disparities (7B vs. 1.4B) and benchmark comparisons. We provide preliminary performance comparisons in the following Table.\n\n|  Model   | Pretrain Data| Train Paradigm|IN-1K (Linear-Prob) | ADE20K (linear) |ADE20K (multi-scale) |\n|:----:|:----:|:----:|:----:|:----:|:----:|\n|Web-DINO ViT-7B | MC-2B |Self-Supervised Learning|86.4|42.6|52.8|\n|UniViTAR-0.3B| Merged-1B |Contrastive Learning|87.6|40.7|54.6|\n|UniViTAR-1.4B| Merged-1B |Contrastive Learning|**88.9**|**45.4**|**56.2**|\n   \n  - Regarding multimodal capabilities, we excerpt key results from *Table 3* of Web-SSL[3] into the following Table for the reviewer’s reference. Notably, Web-DINO at 7B scale underperforms SigLIP-SO400M-378 in multimodal tasks. Similar observations were documented in Cambrian-1[4]. Given the necessity for rigorous multimodal benchmarking, we will incorporate expanded comparative analyses in the revised manuscript.\n\n|Model| Resolution|General|Knowledge | OCR & Chart |Vision-Centric|\n|:----:|:----:|:----:|:----:|:----:|:----:|\n|Web-DINO ViT-7B | 378 |73.9 |47.7 |50.4 |57.7|\n|SigLIP-SO400M| 384 |**75.5** |**48.2**|**55.1** |**60.8**|\n\n- **Cambrian-1**[4].   Cambrian-1 introduces a family of multimodal LLMs designed with a vision-centric approach, which mainly employs the **MoF (Mixture-of-Feature)** paradigm for vision representation with a combination of four exist vision encoders: *OpenAI CLIP ViT-L/14@336, SigLIP ViT-SO400M/14@384, OpenCLIP ConvNeXt-XXL@1024, and DINOv2 ViT-L/14@518*. Considering that UniViTAR primarily focuses on (1) exploring large-scale pre-training strategies for a ​**​single visual encoder​**​ under ​native-resolution setting​ and (2) improving the quality of ​**​general visual representations​**​, we have provided detailed comparisons in the paper between UniViTAR and the CLIP series models employed in Cambrian-1 (see Experiments 3 to 6 in our paper). Generally, the MoF fusion scheme reduces information loss by integrating features from multiple visual representations. However, this approach typically incurs ​​increased computational overhead​​ and introduces complexities in selecting and integrating diverse ​​visual foundation models​​ for practical applications. \n\nTo ensure ​​comprehensive comparison​, we will incorporate detailed discussions of these points in the ​updated version​​ of our manuscript.\n\n*[1] Dehghani, Mostafa, et al. \"Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution.\" *Advances in Neural Information Processing Systems* 36 (2023): 2252-2274.*\n\n*[2] Beyer, Lucas, et al. \"Flexivit: One model for all patch sizes.\" ​*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*​. 2023.*\n\n*[3] Fan, David, et al. \"Scaling language-free visual representation learning.\" *arXiv preprint arXiv:2504.01017* (2025).*\n\n*[4] Tong, Peter, et al. \"Cambrian-1: A fully open, vision-centric exploration of multimodal llms.\" *Advances in Neural Information Processing Systems* 37 (2024): 87310-87356.*"}, {"review_id": "M2pekf18kE", "reviewer": "Reviewer_xyrn", "summary": "The paper proposes UniViTAR (Unified Vision Transformer with Native Resolution), a family of vision foundation models designed to handle visual inputs in their native resolution and aspect ratio across both image and video modalities. Unlike conventional Vision Transformers that operate on fixed-size, square inputs, UniViTAR aims to preserve more spatial and contextual fidelity by avoiding aggressive resizing.\n \nTo achieve this, the authors make several contributions:\n- Versatile Framework: They introduce a family of homogeneous visual foundation models that support native resolution and unified feature extraction across visual modalities. \n- Progressive Training Paradigm: A four-stage training process is introduced, beginning with fixed-resolution pretraining and gradually adapting to native resolution and video inputs. \n- Empirical Performance: Trained only on public datasets, UniViTAR models (ranging from 0.3B to 1.4B parameters) achieve leading performance on a range of image and video classification/retrieval tasks, as well as strong results on multimodal benchmarks.", "questions": "1. Does the training scheme based on native resolution present efficiency issues? In this paper, images are resized during training to ensure that the final sequence length matches the specified maximum length. Compared to directly maintaining the image’s native resolution, this approach significantly increases computational resource requirements during training. Would a method similar to the Qwen series, which directly uses the native resolution of images for training, be a better solution?\n \n2. Does scaling images to the maximum sequence length during training, while keeping their native resolution during inference, impact inference performance? During inference, the model may encounter very small images, whereas most images during training are enlarged to the maximum sequence length. This leads to a distribution mismatch between training and inference data, which could negatively affect the performance on small images. The poor performance of the Native Resolution approach at a sequence length of 256, as shown in the right graph of Figure 3, precisely illustrates this issue. Since many benchmarks in Table 6 include a large number of small images, it would be valuable to investigate whether appropriately increasing the resolution of these small images can further improve performance. Such an ablation study should be included.\n \n3. How efficient is inference with ViT using native resolution? When the image resolution is very high, a large number of tokens will inevitably be generated. Even though this may lead to improved performance compared to traditional methods, will the inference efficiency decrease significantly as a result?\n\n4. While the model is capable of handling higher-resolution inputs, it was not sufficiently trained across the full range of such resolutions. Did the authors analyze how the distribution of image resolutions and aspect ratios in the training data affects the model’s performance on tasks requiring varying input sizes?", "limitations": "The authors did not discuss the limitations of the paper in a dedicated section. In my opinion, the main limitations of this work are as follows:\n \n1. Lack of theoretical analysis or intuitive case. For example, under what specific circumstances does the native resolution ViT outperform traditional ViT models?\n \n2. Lack of fair comparisons. One of the main applications of ViT is as a visual encoder in MLLMs. However, Table 6 does not successfully demonstrate the superiority of native resolution ViT in this regard. This is my primary concern.\n \n3. Computational efficiency issues. For large images, using native resolution produces a significant number of tokens. It is not desirable to sacrifice a great deal of speed for only a marginal improvement in accuracy.\n \nOverall, this is a meaningful work, but I think there are still some pain points that need to be addressed.I will consider raising my score if the authors can adequately address the question I raised.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Originality: Currently, there is no complete training scheme for native resolution-based Vision Transformers (ViTs) available in the open-source community.\n \n2. Significance: Native resolution-based ViTs play a crucial role in visual tasks, particularly in applications that are highly sensitive to resolution requirements, like extreme resolution and ratio.\n \nWeaknesses\n \n1. The comparisons lack fairness. The paper states that models are kept at native resolution during inference; however, in Table 6, benchmarks such as DocVQA, InfoVQA, and HRBench4K are known to contain images with extremely high resolutions. Using native resolution results in a much larger number of tokens (several thousand) compared to algorithms based on fixed resolutions (a few hundred tokens), making the comparisons clearly unfair. It would be more appropriate to follow the approach of LLaVA-NeXT—by using tiled sub-images to increase the token count and enhance dynamic resolution awareness for the baseline methods, or by comparing with models that have more parameters but similar computational resource usage to UniViTAR. Only if the superior performance is maintained under these fairer conditions can the effectiveness of the proposed method be truly demonstrated.\n \n2. The paper lacks intuitive case studies and theoretical analysis. Most of the content is devoted to experimental results showing that the native resolution approach outperforms the fixed resolution approach, but it does not analyze the underlying reasons. Does the native resolution method perform better in all scenarios, or only in cases of extreme aspect ratios or resolutions? It is necessary to further analyze the advantages of native resolution on specific cases or benchmarks.", "was_revised": true, "final_justification": "The author addresses most of my concerns. I would like to keep my rating of borderline accept.", "rebuttal": "> **Weakness 1. The comparisons lack fairness. Only if the superior performance is maintained under these fairer conditions can the effectiveness of the proposed method be truly demonstrated.**\n\nTo address the reviewer’s concerns about comparative fairness, we designed controlled experiments by constraining UniViTAR’s ​**​per-image token range​**​ to align with the computational budgets of baseline encoders (SigLIP-SO400m: 729 tokens; AIMv2-H: 1024 tokens). Specifically, we enforced a token range of ​​288-1170​​ (mean≈729) and ​288-1760​​ (mean≈1,024) during training/inference. Results on three high-resolution benchmarks in below table demonstrate UniViTAR’s consistent superiority under matched computational conditions:\n\n|Model (@Qwen2.5-1.5B)|Resolution| Token Range| InfoVQA|DocVQA|HRBench4K|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|SigLIP-SO400m|378|729|23.2|32.0|45.0|\n|AIMv2-H | 448 |1024|25.8|36.2|44.5|\n|UniViTAR-0.6B |Native| 288\\~1170 (729±441) |  24.6|39.1|46.0|\n|UniViTAR-0.6B |Native| 288\\~1760 (1024±736)  | 27.5|45.2|46.0|\n\n> **Weakness 2. The paper lacks intuitive case studies and theoretical analysis. Does the native resolution method perform better in all scenarios, or only in cases of extreme aspect ratios or resolutions?**\n\nWe sincerely appreciate the reviewer's insightful comment regarding the need for deeper theoretical analysis and intuitive case studies.\n* Theoretical advantages of native resolution processing​​.\n  * ​**​Preservation of Geometric Integrity**.​ Fixed-resolution methods inevitably introduce geometric distortion, either through peripheral information loss (center cropping) or object deformation (aspect ratio warping). In contrast, native resolution processing inherently eliminates such distortions by: (1) maintaining the original aspect ratio without spatial modification, and (2) preserving fine-grained contextual relationships across the entire image canvas.\n  * ​**​Alignment with Natural Image Resolution Distribution​**.​ Native resolution processing better reflects the inherent variability in resolution found in real-world images.\n* The native-resolution approach demonstrates superior performance or greater flexibility in most scenarios. However, it is not universally optimal. Below, we provide some intuitive case studies to delineate specific scenarios:\n  * ​**​Hardware Compatibility Constraints​**​. Modern GPU tensor cores optimize for fixed-dimension computations. Variable-length token sequences cause suboptimal hardware utilization.\n  * ​**​Extreme Aspect Ratio Scenarios (>1:100)​**​. For extreme OOD-resolution inputs, both fixed and native-resolution methods empirically fail to extract good visual features due to inherent limitations in training data coverage.\n\n> **Question 1. Does the training scheme based on native resolution present efficiency issues? Compared to directly maintaining the image’s native resolution, this approach significantly increases computational resource requirements during training.**\n\nWe appreciate the reviewer's insightful questions regarding the computational efficiency of our native-resolution training scheme and the comparison to approaches like Qwen2-VL. We address these points below:\n- We contend that directly training on native resolution (analogous to Qwen2-VL) presents several disadvantages, making it a suboptimal solution:\n  * ​**​Requirement for pre-packing​**.​ To maximize data utilization efficiency and training throughput, a ​**​greedy pre-packing strategy​**​ is typically adopted (as utilized in NaViT). Critically, different packing strategies can ​compromise data randomness​​ during training.\n  * ​**​Constrained resolution distribution​**.​ The ​​resolution distribution​ becomes rigidly determined solely by the training dataset, which leads to a ​​reduction in the diversity of resolutions​  encountered during training. Our proposed strategy dynamically adjusts the token sequence length per image ​within each batch​​, based on ​​both​​ the distribution of image resolutions within the current batch and the predefined ​maximum token limit​. Consequently,  the same image may be resized to different resolutions across distinct training batches, and this inherent variability ​​ensures a richer distribution of resolutions.\n- We argue that our proposed strategy ​**​does not impose a significant increase in computational resource demands​**​. The primary reason is as follows. During training with batch size greater than 1, training frameworks invariably ​​pad variable-length visual sequences within a batch to a maximum sequence length​​ for computational efficiency and minimize ​pipeline bubble. Crucially, regardless of *Batch-Padding-to-Max-Length* strategy in Qwen2-VL or *Batch-Resizing-to-Max-Length* strategy in UniViTAR, the ​upper bound​ on computational resource requirements is ​more dictated by the configured maximum sequence length. In fact, by resizing images within each batch according to our strategy, our method actually **​​reduces the overall padding ratio​** across the training process and ​enhances computational resource utilization efficiency​​.\n\n> **Question 2. Does scaling images to the maximum sequence length during training, while keeping their native resolution during inference, impact inference performance?**\n\n- We first wish that we have correctly interpreted the reviewer’s inquiry. The proposed native-resolution training strategy does not uniformly scale images to a maximum sequence length. Instead, ​**​the resolution of each image is dynamically adjusted based on the resolution distribution of all images within its batch​**​. Under this paradigm, the same image may be resized to different resolution when processed in distinct batches, which may be ​**​upscaled​**​, ​**​downscaled​**​, or ​**​remain unchanged​**​ depending on the resolutions of other images in the same batch.\n- Regarding the visualization results for Figure 3 (right panel), under strict visual sequence length constraints (256 tokens), we think the suboptimal performance of *Native Aspect Ratio* primarily stems from UniViTAR’s early training stages bias (Stage-1/2), where a fixed sequence length of 256 tokens was employed for pretraining. Crucially, as illustrated in the Figure 3, when maximum visual sequence lengths exceed 256 tokens (e.g., ≥576), the following relationship consistently holds: Native Aspect Ratio​​ > Fix-size.\n- Exploring the impact of minimum resolution on benchmark performance is indeed a valuable suggestion. We adjust the min-token number during evaluation, and the results are shown in the table below. Interestingly, performance improvements on certain benchmarks are not linear, which aligns with the findings in Qwen2-VL. Benchmarks like OCRBench contain many very small images, and excessive enlargement of these images leads to distortion of images. Overall, the average performance across the 16 benchmarks shows little difference, indicating robustness of our model. \n| Min Tokens | Avg. Performance|HallusionBench|OCRBench|\n|:---:|:---:|:---:|:---:|\n|72 |49.3|57.4|37.0|\n|144|49.5|57.4|36.8|\n|288|49.5|57.8|36.9|\n|576|49.5|57.6|36.7|\n\n> **Question 3. How efficient is inference with ViT using native resolution? When the image resolution is very high, a large number of tokens will inevitably be generated.**\n\nThe integration of native-resolution processing in ViT inherently establishes a fundamental efficiency-performance trade-off, governed by the quadratic computational complexity of self-attention mechanisms relative to token count. This implies that inference efficiency degrades progressively as token volume increases—a theoretical limitation scaling polynomially with sequence length. To mitigate computational explosion during dynamic-resolution training, we impose constraints on ​**​maximum tokens per image​**​, *i.e*., CLIP pretraining phase​ is 16,384 tokens and MLLM alignment phase​ is 5,832 tokens (after pixel-unshuffle is 2916). This calibrated token ceiling balances inference efficiency against representational fidelity, ensuring tractable resource utilization while preserving performance gains.\n\n> **Question 4. Did the authors analyze how the distribution of image resolutions in the training data affects the model’s performance on tasks requiring varying input sizes?**\n\nOur analysis reveals that native-resolution vision encoder exhibits non-trivial ​**​extrapolation capabilities​**​ to resolutions beyond its training distribution.  To quantitatively assess this, we conducted an experiment using the ​*UniViTAR-0.6B + Qwen2.5VL-1.5B*​ as baseline​. During training, the maximum token count was limited to ​2,916 tokens​​ (corresponding to visual tokens number fed into the LLM). During inference, we progressively increased the evaluation maximum token limit per image to ​5,832 (2×)​​ and ​8,748 (3×)​​ tokens. Performance across our full evaluation suite and specialized high-resolution benchmarks is summarized below:\n\n|Max Tokens (Train)|Max Tokens (Evaluation)| Avg. Performance|HRBench4K|TextVQA_VAL |\n|:---:|:---:|:---:|:---:|:---:|\n|2916|2916|49.5|46.1|50.7|\n|2916|5832 (2x)|49.6|47.3|51.4|\n|2916|8748 (3x)|49.6|48.5|51.4|\n\nIncreasing the maximum token limit to 5,832 (2×) and 8,748 (3×) yields ​​consistent improvements​ on benchmarks containing substantial high-resolution content, namely ​*HRBench4K*​ and ​*TextVQA\\_VAL*. Furthermore, this result complements the **low-resolution extrapolation capability** demonstrated in the Question 2, ​​collectively confirming​ the model's inherent robustness across diverse resolutions beyond its specific training distribution."}]

Author rebuttal:

Meta-review: This paper introduces UniViTAR, a family of vision foundation models designed to process images and videos in their native resolutions. The work consolidates best practices (e.g., 2D RoPE, SwiGLU, RMSNorm, QK-Norm) into a unified training recipe, scaling models from 0.3B to 1.4B parameters and demonstrating strong performance across classification, retrieval, and multimodal benchmarks.

The main critique from reviewers was limited novelty, as the design draws heavily from NaViT, Qwen2VL, and related works. Reviewers also noted the need for fairer comparisons, more granular ablations, and broader discussion of related approaches. The authors addressed these concerns in rebuttal with new controlled experiments, detailed ablations, expanded related work discussion, and additional multimodal scaling results.

Given the rebuttals from authors, all reviewers ultimately recommended borderline accept, recognizing the paper’s solid engineering contributions, clear empirical validation, and community value through open-sourced models and training recipes. The ACs went through the discussions and final recommendations from each of the reviewers, and recommend an acceptance for this work. While the conceptual novelty is incremental, the work is technically sound, and likely to have meaningful impact as a reliable visual backbone for multimodal research. The authors should incorporate the experiments and discussions into the final revision, and release the code and model for reproduction and downstream uses.
Decision: Accept (poster)