UniViT: Unifying Image and Video Understanding in One Vision Encoder
Authors: Feilong Tang
Xiang An
Haolin Yang
Yin Xie
Kaicheng Yang
Ming Hu
Zheng Cheng
Xingyu Zhou
Zimin Ran
Imran Razzak
Ziyong Feng
Behzad Bozorgtabar
Jiankang Deng
Zongyuan Ge
Venue: NeurIPS 2025 poster ()
Keywords: Image and Video Understanding
Primary area: deep_learning

Abstract: Despite the impressive progress of recent pretraining methods on multimodal tasks, existing methods are inherently biased towards either spatial modeling (e.g., CLIP) or temporal modeling (e.g., V-JEPA), limiting their joint capture of spatial details and temporal dynamics. To this end, we propose UniViT, a cluster-driven unified self-supervised learning framework that effectively captures the structured semantics of both image spatial content and video temporal dynamics through event-level and object-level clustering and discrimination. Specifically, we leverage offline clustering to generate semantic clusters across both modalities. For videos, multi-granularity event-level clustering progressively expands from single-event to structured multi-event segments, capturing coarse-to-fine temporal semantics; for images, object-level clustering captures fine-grained spatial semantics. However, while global clustering provides semantically consistent clusters, it lacks modeling of structured semantic relations (e.g., temporal event structures). To address this, we introduce a contrastive objective that leverages these semantic clusters as pseudo-label supervision to explicitly enforce structural constraints, including temporal event relations and spatial object co-occurrences, capturing structured semantics beyond categories. Meanwhile, UniViT jointly embeds structured object-level and event-level semantics into a unified representation space. Furthermore, UniViT introduces two key components: (i) Unified Rotary Position Embedding integrates relative positional embedding with frequency-aware dimension allocation to support position-invariant semantic learning and enhance the stability of structured semantics in the discrimination stage; and (ii) Variable Spatiotemporal Streams adapt to inputs of varying frame lengths, addressing the rigidity of conventional fixed-input approaches. Extensive experiments across varying model scales demonstrate that UniViT achieves state-of-the-art performance on linear probing, attentive probing, question answering, and spatial understanding tasks.

UniViT: Unifying Image and Video Understanding in One Vision Encoder
Haolin Yang", Yin Xie", Kaicheng Yang", Feilong Tang" 33 Xiang Haolin
Ming Hu?.3, Zheng Cheng", Xingyu Zhou", Zimin Ran", Imran Razzak
Ziyong Feng?, Behzad Bozorgtabar', Jiankang Deng", Zongyuan Ge
Monash University, 2 *DeepGlint, MMBZUAI, 'UTS, EPFL, Imperial College London Feilong. Tang@monash edu, xxlangandeeopllnt..com
Abstract
Despite the impressive progress of recent pretraining methods on multimodal tasks, existing methods are inherently biased towards either spatial modeling (e.g., CLIP) or temporal modeling (e.g.. V-JEPA), limiting their joint capture of spatial details and temporal dynamics. To this end. we propose UniViT. a cluster-driven uni- fied self-supervised learning framework that effectively captures the structured semantics of both image spatial content and video temporal dynamics through event-level and object-level clustering and discrimination. Specifically, we leverage offline clustering to generate semantic clusters across both modalities. For videos, multi-granulan gratiirrrtlllrtty event-level clustering progressively expands from single-event to structured multi-event segments, capturing coarse-to-fine temporal semantics; for images, object-level clustering captures Amee-gained spatial semantics. However, while global clustering provides semantically consistent clusters, it lacks model- ing of structured semantic relations (e.g., temporal event structures). To address this, introduce contrastive objective that leverages these semantic clusters as pseudo-label supervision to explicitly enforce structural constraints, including temporal event relations and spatial object co-occurrences, capturing structured semantics beyond categories. Meanwhile, UniViT jointly embeds structured object- level and event-level semantics into unified representation space. Furthermore, ViT introduces two key components: (i) Unified Rotary Position Embedding integrates relative positional embedding with frequency-aware dimension alloca- tion to support position invariint semantic learning and enhance the stability of structured semantics the discrimination stage; and (ii) Variable Spatiotemporal Streams adapt to imputs of varying frame lengths, addressing the rigidity of conven- tional fxee-input approaches. Extensive experiments across varying model scales demonstrate that VIT achieves satte-o-the--ee performance on linear probing, attentive probing, question answering, and spatial understanding tasks.
Introduction
Visual representations are fundamental to the success of various downstream tasks. Contrastive [48] [7I] and self-supervised 66 10 (4) frameworks have strong spatial semantic modeling and cross- modal alignment. However, existing methods exhibit inherent biases toward specific modalities, limiting their joint capture of spatial details and temporal dynamics: Image-centric models (e.g.. CLIP (48) capture static semantics but inadequately model temporal dynamics, while video-centric models (e.g., V-JEPA I0 incorporate temporal cues but exhibit deficiencies fine- mne-grri spatial modeling.
'Equal Contribution. Projecct lead.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Mutti Object Contrastive Loes
Fvent Instance Contr Lock
OkjjectEvent Contrastive Loss
Cosine
Cosine
Cosine
Similarity
Simiarity
Similarity
Temppord Eraedsings
Ereagdings
Sampled Objject Centers
Sampled Event Centers
Eriteddingg
Sampled Object/Event Conters
ViT
VIT
UniVi
Images
Object Centers in Memory Bank
Videos
Event Centers Memory Bank
Images'Videes Object/Event Canters Memory Bank
(a) Image Discrimination
(b) Video Discrimination
(c) UniViT Discrimination
Figure 1: Comparisons of cluster discrimination in image, video, and unified representation. (a) Image multi-label cluster discrimination improves semantic cohesion by assigning multiple samples to cluster centers, capturing various granularities of visual signals at the object level, but is limited to image representations. (b) Video cluster discrimination assigns discrete event-level labels to video segments, modeling dynamic semantics but lacking fine-grained spatial structures. (c) The proposed UniViT adopts unified multi-label discrimination at event and object levels with shared clusters and encoders, bridging spatial semantics and temporal structures within unified representation.
The key to unified visual pretraining lies in jointly modeling static and dynamic semanties through structured, semantically coherent representations. Recent approaches such as UNICOM (2]. MLCD 4 and RICE [64] enhance perception of structured semanties in images by introducing clustering mechanisms. MLCD further advances this approach by employing multi-label clustering to capture multiple semantic components in images, depicted in Fig. (a). Furthermore, Chat- Univi (4] introduces video event clustering for semantic modeling; however, it remains primarily focused on static objects and isolated events, lacking semantic modeling at the structured event level, thus failing to capture structured temporal dynamics. For instance, although actions such as "grabbing a cup" and "grabbing phone" differ in their visual manifestation, model with event-level abstraction can generalize them into a unified "grabbing" event category and capture the semantic relationship between the action and the target object. Moreover, such model can infer the structural role of an action within an event sequence, such as recognizing that "grabbing a bowl" often precedes the event of "serving food." This capability signifies a transition from isolated event recognition to structured event understanding. Therefore, we argue that transitioning from instance semantic recog- nition to cross moos--moaal structured modeling is essential for constructing a unified visual pretraining framework.
In this work, we propose UniViT, a cluster-driven unified self-supervised learning framework that effectively captures the structured semantics of both image spatial content and video temporal dynamics through event-level and object-level clustering and discrimination, as depicted in Fig. [c). Specifically, we design two-stage minster-dicrrmmiatiio training paradigm. In the clustering stage, we employ offline clustering to generate semantic for both modalities. For videos, we perform multi- multi-granuulrity event-level clustering by densely sampling frames at multiple temporal scales, progressively organizing individual events into structured segments of multiple events, thus capturing coarse-to- to-fine temporal semantics. For static images and individual video frames, we employ object- level clustering extracts fine-grained spatial semantics. Subsequently, in the discrimination stage, we introduce contrastive objective that utilizes these semantics as pseudo-label supervision to explicitly enforce structural constraints, including temporal relations among adjacent events and spatial co-occurrences among objects. This approach captures structured semantics beyond isolated categories, aligning dynamic and static semantic content within a unified representation space.
The core of this method is to jointly model structured semantics across image and video modalities within a unified representation space, effectively bridging static spatial details and dynamic temporal relations. Therefore, UniViT introduces two critical components: (i) Unified Rotary Position Embed- ding (U-RoPE), decomposing positional embeddings into distinct spatial and temporal components through frequency- aware allocation, thereby facilitating position-invariant representation learning and enhancing stability of structured semantic representations during the discrimination stage; and (ūi) Variable Spatiotemporal Streams (VS²), adapting to varying frame lengths, enabling the model to flexibly capture fine-graines spatial details and diverse temporal scales simultaneously. Notably, UnT retains the original vision encoder without incurring extra inference costs.
2

---

Class Embedding ng
Full Pustie
Rush
3D-Conv
Nagative
P.Frame Excas Class Center
Event1
Eventz
Event3
Attentive Pooler
push
UniViT
Schrme Dvert Caas Centert
2D-Conv
J-REPE
Image
Image1
Image2
Image3
Imaged
Sinfoliraje Center
05 12 218 AI
push
Figure 2: Overview of the UniViT framework for unified representation learning. Image and video inputs are processed by 2D/3D convolutions, with video features segmented into events. The resulting tokens are fed into shared UniViT backbone with U-RoPE, followed by Attentive Pooling produce a class-level representation for contrastive learning against memory bank class centers.
Extensive experiments across varying model scales demonstrate that UniViT achieves state-of-the-a performance multiple downstream tasks, including linear probing, attentive probing, image and video question answering, and spatial understanding. The contributions are summarized as follows: (i): We propose UniViT, cluster-driven unified self-supervised learning framework that effectively captures the structured semantics of both image spatial content and video temporal dynamics through event-level and object-level clustering and discrimination. (ši): We introduce U-RoPE and V strategies to explicitly disentangle spatial-temporal positional embeddings and adaptively handle varying frame lengths, facilitating position-invarian and structured spatiotemporal representations. (iii): Extensive experiments across diverse model scales and tasks demonstrate that Uni achieves -ecc-of-thhearr performance on multiple downstream tasks.
2
Methodology
Our goal is to achieve unified representation learning across image and video modalities (Fig.
Variable Spatiotemporal Streams (VS*). Given visual samples X { each sample x, is uniformly divided into non-overlapping patches, accommodating both images and videos in unified manner. Specifically, each image sample F; HR x3 is partitioned into spatial patches of size P x P. while each video sample z; E TxxxxXxX is partitioned into variable-length frame sequences [ 1==1 with each sequence Fi,, T.. Hx x3 covering consecutive RGB frames (e.g., 2, 4, 8, or 16 frames) over the same spatial regions, where S; denotes the number of sequences within the i-th video. Subsequently, each sequence Fi., is independently encoded via shared Transformer-basee encoder ob-) into spatiotemporal embedding tokens Z,:
S.
Z, = (o(F.)) ((F.2) o(F.ss)) RRRRMM MxC. with M M,
(1)
where M, and C denote the number of embedding tokens and the embedding dimension for the s-th sequence, respectively. The resulting tokens are projected into D-dimensional vectors, forming token embeddings E, = { 3 E MDD that encode local visual features (21). Subsequently, positional encodings are dynamically assigned using the 2 strategy. Therefore, this design enhances flexibility and spatiotemporal representational capacity.
Event-level and Object-level Clustering. Iterative clustering- discrimination approaches commonly suffer from substantial computational overhead [4.. To address this issue, we adopt single-step offline clustering to efficiently capture both object-level semantics from images and event-level semanties from videos. Specifically, image embeddings are obtained by pooling the features extracted from local object patches, ohj = 6, E TRP, while video embeddings which are derived from fixed-length 16-frame input obtained by concatenating frame-level features within each segment, yielding e [er,1: ..; e,,] E x from variable-length sub-clips of s E {1; 2;...; S.}. We Kim
define set of shared semantic centroids C = (bbb)) ) k=1 represents the total number of clusters across both modalities. The clustering objective is then
Kabi U(eevt K RD R. where K Kobj Kevt.

---

formulated separately for object and event embeddings:
S. min ke[l,Kevd
Cuni arg
min
min Re(IKKobil
lee
obj
2
lle
of 11
(2)
s=1
where N is the number of samples. consistent representation learning.
Cunt integrates object-level and event-level semantics for
Unified Rotary Position Embedding (U-RoPE). Unlike traditional absolute position encoding
defined as p (t, y), U-RoPE adopts relative scheme [53] Ap (41 -1,,31 22: y1 y2) that supports position-invariant semantic learning, enabling better modeling of multi-event structures in videos. Specifically, the rotary position embedding is applied directly the query-key dot-product attention matrix, i.e., A.jj = (q. (k,((, R2) For image inputs, the temporal position & is fixed across the spatial grid (xr v). For video inputs, temporal positions vary across frames while spatial positions are computed the same way as for images. Existing methods, such as M-RoPE [60], typically allocate temporal position encodings with high-frequency components, determined by the rotary frequency On = 3 S. This allocation causes periodic oscillations, leading to unstable frame representations that conflict with dense label discrimination. Therefore, we propose unified frequency allocation strategy that assigns global event-related temporal structures to smoother low-freuuency components, while retaining higu-freyuency components for local spatial details:
h {
+x) C
j [0, 4 1), E 1), Φ (A 1
where & and T respectively denote the rotation frequencies used for spatial and temporal rotary applying to the 21-dimensional embedding space, with L C and / represents the base frequency.
Joint Training Objective. Visual samples commonly exhibit multiple semantic components, includ- ing object-level semantics from images and event-level semantics from videos, rendering single-label assignments inadequate for unified multimodal representation learning. To capture both object-level and event-level semantic structures, we introduce a contrastive objective that leverages these semantic clusters as pseudo-label supervision explicitly enforce structural constraints. Specifically, for each visual embedding e E RD, we identify multiple positive semantic labels from the unified semantic centroid set Cuni E R(lea,)-((covl)d consisting of both object-level Coby and event-level Ceut centroids. The remaining centroids in this unified set treated as negative labels. Subsequently, the joint multi-label semantic discrimination objective is formulated as:
CJoint
Σ me(objeevt)
log(1 +
Σ exp(o"")) log(1 JERR
Σ exp(-o 1m
(4)
where m E {obj. evt} denotes the semantic gramularity level, corresponding respectively to object- level (images or single frames) and event-level (video segments). 227 and S represent the sets of positive and negative semantic labels for granularity level m, while a."" and d² indicate embedding similarity scores positive and negative semantic centroids, respectively. The embedding similarity score o.i is computed as o.A = e,, Cg where and k index visual embeddings and semantic centroids within the corresponding positive or negative sets, respectively. This unified formulation leverages semantic clustering capture spatiotemporal structures for discriminative embeddings.
Unified Image and Video Understanding. As shown Fig 3a the frame-to-frame similarity under the multi-event setting is significantly lower than that of the single-event counterpart, indicating finer- grained temporal discrimination and better action segmentation. The lower training loss in Fig. (3b] indicates that In Fig. 3c] U-RoPE demonstrates faster convergence and improved stability over absolute position encoding and M-RoPE. By decoupling spatial and temporal dimensions and avoiding figh-fireueecy temporal encoding. U-RoPE enables robust position-invariant learning across both images and videos. Fig. illustrates the distribution of cosine similarity between image and video feature embeddings, indicating that video and image representations are related yet significantly different.
temporal modeling in the multi-event setting benefits video understanding.
Experiments
3.1 Implementation Details.
Pretraining Setup. Our models are pre-trained on the LAION400M[50), COY0700M and IntemVid(6]]. We use 80 H800 GPUs for the training process. During training, maintained a 1:1

---

2.00
20-
Arieent hhree
Image Video Smilarry
23
5.03
28
Alleent fummss
alieer Larre
6.75
6:20
24
73
GAP
22
0.31
526
100l
150k
2006
3506
3006
23
50x
1034
150K
2032
7506
30c
0
98%
1036
150M 2004
2506
3034
Rs 63 02
03
04 0.5 44
0.7 0. 05 1.
(a) Mean frame similarity.
(b) Loss cruve.
(c) U-RoPE
(d) Image Video Similarity
Figure 3: Analysis of multi-event approach. (a) Multi-event frames demonstrate significantly lower frame similarity (0.65 vs 0.75, indicating greater inter-frame distinctiveness that improves feature discrimination. (b) Multi-event processing substantially accelerates convergence speed, with higher frame counts (8-16) showing faster loss reduction compared to single or fewer frames. (c) U-RoPE enables better local semantic learning, resulting lower loss compared to other methods. (d)
Distribution of cosine similarity between image and video feature embeddings.
Timls
Vidro Tacks
Valce
Porame
Insggoo
INH
URAM
Avk
Methed
Arch
Mesiaas pretraled
Feuges
256
96M
08.1
90.9
05.4
83.2
61.2
66.3
92.2
31.3
16.3
2iplip22
S
VIT-L/1
256
303M
10B
92.6
96.5
07.2
94.5
94.3
06.0
84.1
65.3
63.9
77.3
17.5
66.5
71.7
05.0
39.8
19.4
VIT-SCNIMM
234
425M
10R
-
13.4
97.1
97.8
95.0
95.8
W6.1
15.5
M..
66.4
76.7
79.1
04.3
76.3
45.0
46.1
20.9
WI-8014
145M
35.6
01.0
83.1
10.3
49.2
83-3
14.3
DINOVD
92.1
90.4
88.1
91.6
234
311M
145M
35.6
14.6
CLIP
L/1
236
224
B6M
304M
400M
40084
91.6
89.0
95.0
95.7
91.4
94.2
94.5
92.6
95.6
93.4
95.7
95.6
81.4
83.2
77.1
54.7
63.7
16.5
67-4
74.8
67.1
74.2
63.2
54.2
66.9
71.3
88.1
92.6
53.1
37.5
14.00
16.3
IJERA
Virggte
234
1011M
90.4
K1.1
3.9
13.8
MI.DD 4
VIT--144
376
304M
1.18
X
92.9
96.8
97.8
05.0
95.2
95.7
84.3
65.4
66.6
75.8
78.9
65.2
$0.4
97.1
46.9
17.4
Merisod pr on
VJJEAA
1V
VIT--166
VIT-1016
234
236
640M
S13M
2M
59.3
$5.0
95.4
94.1
6.3
4.0
44.3
Larguag:Bonl 83
VIT-L/14
224
40734
354
6OLD
72.1
72.3
60.3
71.9
93.3
35.8
1644
VideoMAEV2
V/Tig/4
234
1012M
1.35M
75.0
1.6
27.9
Mettook po on lesage and Waleos
VIT B/16
224
93M
5-B8222M
19:3
94.0
92.7
92.9
91.2
96.4
84.2
50.1
54.D1
66.6
66.2
53.6
65.9
90.3
19.4
15.9
PR-Cure T
VII--/14
316
S1TM
SAB032M
92.1
96.
04.5
06.8
01.5
05.5
85.1
66.6
67.5
79.6
19.8
69-4
17.7
96.8
47.5
23.5
WI-G/14
448
15K2M
S4B/22M
97.3
93.9
86.2
68.4
$0.5
91.3
42.2
23.7
VIT-5/16
224
25M
LIB:500
81.3
71.5
52.0
18.1
15.6
UNIVIT
VIT4-214
VIT-8016
234
234
99M
134M
M1.5
88.0
95.2
09.5
95.1
96
41.1
05.3
95.4
84.3
64.
A%
67.4
74.
85.0
76.8
61.5
71.4
$1.4
70.2
94.
91.1
15.8
12.0
23.1
26.1
VIT-L/14
336
334M
LIB:500
92.8
95.1
90.7
95.1
93.7
95.7
85.4
86.5
73.1
84.1
$43
73.3
53.5
98.2
563
32.1
Table 1: Attentive Probe Evaluation under few-shot settings for label efficiency analysis. Bold
indicates the best performance.
ratio between images and video frames, with an image batch size of 16K and a video batch size of 2K (each video containing 16 frames). In total, our model is exposed to approximately 20B image frames throughout the training. For our standard model, we use 224 resolution images. For the 336 resolution variant, we first train the model 224 resolution, then increase it to 336 and continue training for an additional IB frames. We utilize the Adam optimizer with a learning rate of 0.001 and weight decay of 0.2. The number of classes (k) is million, the ratio of sampled negative class centers (r) is 0.1, and the number of positive labels (I) assigned to each image and video is 8.
Multimodal Setup. For multimodal large language model evaluations, we adopt the LLaVA- NeXT [T framework while maintaining experimental consistency. All training methodologies precisely follow the original LLaVA-NeXT- Video implementation, utilizing identical pretraining datasets and instruction-tumn tuning data. We employ Qwen2.5-7B [68] as our language model back- bone, which effectively mitigates potential hyperparameter biases that might favor OpenAI-CLIP in the original -aaV--eee--Viiaae sonfiguration. This controlled experimental design ensures fair comparison when evaluating the performance of our vision encoders within multimodal systems.
3.2 Comparisons with Existing Vision Encoders
Attentive Probing Results. We evaluate the comprehensive ability of VnT across 14 standard benchmarks, covering a wide range of semantic and vision-centric tasks, using a 50-shot attentive
5

---

Emsrge Tisks
Vilco Tasks
123 EASS
DID
Methed
VIT VTT-8/16
256
8IM
96.3
43.3
99.55
153.5
91.5
63.7
9406
12.5
21.1
Siplip22
VIT-L/16
256
303M
10B
94.3
96.2
97.8
98.2
93.3
98.6
85.2
91.3
59.4
76.5
78.2
66.2
68.5
93.1
12.5
22.7
VIT-504005
224
425M
10B
94.7
99.4
09.2
97.8
94.0
98.6
85.7
92.2
66.5
73.6
79.7
69.3
68.0
94.4
55.1
23.2
929
13.2
VIT42214
224
31IM
145M
93.6
04.5
99.4
08.5
903.
97.5
86.3
$9.0
61.0
73.0
94.1
66.4
91.
40.6
19.6
CLIP
VI7-8416
W14J14
224
86M
304M
40084
300M
19.38
92.2
05.9
97.8
91.3
90.9
96.3
91.4
90.2
81.9
59.2
58.3
70.3
11.3
51.3
66.8
66.4
88.5
933
53.1
37.5
19.0
19.3
IJBRA
V17
224
224
101154
22K
63.9
37.7
98.2
98.7
92.5
95.8
79.3
81.1
50.6
42.0
49.6
50.4
57.3
41.7
74.7
18.8
15.5
15.1
VIT
226
22M
1.18
85.7
98.3
96.5
81.9
924
79.1
54.6
15.9
63.1
49.6
38.3
36.7
61.
25.0
13.0
VIT-811
90.3
46.5
790
28.1
15.6
16.7
376
704M
99.1
94.5
97.9
16.3
93.3
86.9
Minisas prrmaateo on Videor
Largugellidd
LIT
224
40TM
3M0
6556
74.4
75.1
02.5
71.1
92.9
46.9
22.1
VIT-611
21.
91M
5-0022M
191.4
05.2
05.1
06.5
01.9
07.2
81.4
60.55
19.1
69.5
96.8
66.0
88.7
17.5
16.9
PE Cove [T]
L/14
336
31TM
5-B82224
15.0
96.2
99.4
98.5
93.5
56.00
85-7
92.0
63.7
74.5
75.4
02.2
69.1
92.4
55.1
19.0
VI1-G94
448
18524
3-88222
95.2
96.3
05.3
96.1
93.5
97.9
89.5
92.1
68.6
90.8
$1.1
70.4
73.4
94.7
56.3
23.2
UNIVIT
V/T
224
224
334M
LIBe500
LIBIS00
94.3
94.5
98.8
99.0
93.5
98.1
85.5
90.2
67.1
54.3
85.5
74,5
64.1
93.1
92.1
43.8
28.1
25.5
VI--214
376
134M
1IRIMM
94.8
96.6
95.9
99.0
94.9
98
85.5
902
72.0
83.3
88.4
74.9
78.1
96.
56.5
27.4
Table 2: Linear Probe Evaluation. Bold indicates the best performance.
General
MMBerrch
Knowledge
MMF Per 1566.6
MMSStr $4.
MMMU 46.7% 750
POPF
TOIF-QA
Temporal long
40.7%
MME Cog
AED
cMMMU 42.4%
Seed 75.6%
VideoMME 60.19% 19%
Temperal Caption
71 10.0%%
MMBBecc CN 76.4%
VisWiz 61.2%
MIVU 34 9%
Image
CLIP
Siglip22
MLCD
VJEPA (Vileo)
UniVIT(Ous))
Figure 4: Comparisons of Vni and existing vision encoders across diverse multimodal image and video benchmarks, highlighting UniViT's superior unified representation capabilities.
probing setup. which allows us to measure feature quality without extensive task-specific finetun- ing. As shown in Table [ model achieves state-of-the performance multiple standard benchmarks, including object classification, scene recognition, and fine-grained categorization.
Linear Probing Results. As shown in Table 2 report performance the same set of benchmarks as in attentive probe, this time using a standard linear probing setup instead. The trend aligns closely with the attentive probe results; our Vni achieves state-of-the-un performance on both semantic and non-semantic tasks, further validating the quality and generality of the learned representations.
All experimental settings for the compared models strictly follow their original implementations to ensure fair comparison. Under this consistent protocol, despite being pretrained without pixel-level or feature-level supervision (e.g., MAE, JEPA), our Vni still achieves strong visual understanding across a wide range vision-centrie tasks. Notably, previous video encoders cannot handle static images, while image encoders struggle with temporally dependent tasks (e.g., SSV2). In contrast, UniViT leverages unified representation space that captures both spatial and temporal patterns, enabling strong performance across image and video domains. This demonstrates the generalization capability of our architecture, which is designed for unified representation learning across modalities.
UniViT as a Vision Encoder for MILMs. In this section, we evaluate our unified vision encoder, Vni which designed seamlessly handle both image and video modalities. We conduct comprehensive experiments on a diverse set of benchmarks to assess the model's ability to learn shared semantic representations across static and temporal inputs. The evaluation performed within the LLaVA-NeXT--vdes framework under consistent and controlled settings to ensure fair comparison. We benchmark VT on 18 datasets spanning four major domains: General VQA and Knowledge VQA. These datasets cover both image-based and video-based VQA tasks, providing

---

Figure 6: PCA visualization of patch features extracted by our Vni Patches displaying similar colors indicate semantic similarities, reflecting that they embody analogous elements or attributes.
holistic view of the model's multimodal reasoning capability. As summarized in Fig. 4 Vni achieves strong and consistent performance across all categories, notably outperforming conventional mongltymdallty baselines such as CLIP and SigLIP. At resolution of 336px, UniViT surpasses CLIP on 17 out of 18 benchmarks. Importantly, Vni demonstrates robust performance across both Image-VQA and Vidco-VQA highlighting its versatility and effectiveness in learning generalizable multimodal semantics. Notably, these results are achieved without any language supervision, further demonstrating the strength of our unified framework in capturing cross- moos- visual understanding. This positions UniVit as a practical and scalable solution for real-world multimodal applications.
To investigate the scalability of our unified vision encoder, we conduct systematic analysis of its performance across varying configurations using an attentive probe protocol on both image and video tasks. Rather than comparing against other paradigms, scaling behavior of our framework along three progressive dimensions: (1) increasing training data volume, (2) expanding model capacity, and (3) increasing input resolution.
Scaling Behavior:
focus the internal
As illustrated in Fig. 5 behavior of UniViT video tasks by averaging performance scores from seven video datasets. This provides stable estimate of performance trends specific to video understanding. This con- trolled design allows isolate and examine the contribution each scaling factor. We ob- serve consistent and meaningful improvements at all stages. Increasing data volume leads to noticeable gains for both modalities, suggesting that enhanced data diversity improves unified semantic modeling. Expanding model capac- ity. from small to large variants, further boosts representation quality, with more pronounced benefits on fine-grained tasks. Higher input res- olution also contributes positively to overall per- formance, especially in video tasks where cap- turing spatial continuity across frames is crucial for robust representation learning. Compared to existing models, our approach exhibits more efficient scaling behavior, with larger variants delivering consistently stronger improvements across video tasks. These results demonstrate that our model scales effectively with standard training resources while maintaining strong generalization across diverse video benchmarks.
we analyze the scaling
$ B.L.
Input
MLCD
Avg. Video Tasks Accuracy
PE-Care
Laivir
+16.5
-129
+5.2
+10.1
50
45
<18 Data
>1B Data
Figure 5: Scaling Behavior: Performance scala- bility UniViT across model capacities, training data volumes, and input resolutions. Average ac- curacy across multiple video tasks demonstrates consistent improvements with increased scale.
Ablation Study
Pretraining Strategy. As shown in Table 3bb we conduct ablation studies to amalyze the impact of robust pretraining strategies and position embedding designs. Table 3al we progressively apply modifications including attentive pooling, and stronger data augmentation. Each contributes positively to K400 and SSV2, highlighting the importance of stable pretraining for video understanding. In we compare different designs for position embedding. We find that moving from absolute
Table 3b
7

---

Data
K400
SSV2
Method
K400
SSV2
Frames
K400
SSV2
Bascline
64.1
15.0
Abs + ViT
64.2
13.9
Bascline
62.7
13.2
+Atten Pool
65.2
15.5
Abs + 2DRoPE
64.7
14.3
64.4
14.2
+Data Aug
65.6
15.6
M-RoPE 65.1
15.2
+4,8
65.2
15.3
U-RoPE
65.6
15.6
+1,2,4,8
65.6
15.6
(a) Pretraining Strategy. ef- fects attentive pooling, and data augmentation.
(b) Position Embedding. Compar- ison of absolute position, 2D, 3D.
(c) Varing Frames with Multi- Event. Pretraining at varying frames with multi-event.
and unified RoPE strategies.
Method
K400
SSV2
Num Classes
K400
SSV2
only Obj.
62.3
12.5
500k
65.2
15.5
only Evt.
61.9
11.3
1M
65.6
15.6
Obj+Evvt
62.7
13.2
2M
65,8
15.5
Obj.+Muutii-EV
65.6
15.6
5M
65.2
15.4
(d) Clustering Strategy. Perform clustering at the Event-level and Object-level.
(e) Classes Number. The number of classes during event-level and object-level clustering.
Table 3: Ablation experiments K400 and SSV2 under few-shot settings. (a) Pretraining strategies. (b) Position embeddings. (c) Multi-event frame sampling. (d) Semantic clustering strategies. (e) Number of clustering classes. The entries marked in gray are the same, which specify the default settings.
position embedding to 2D-RoPE with Id-absolutioe position and 3D-RoPE leads to substantial performance gains, with 3D-RoPE achieving better results. This suggests that temporal embedding better captures motion dynamics. Our U-RoPE decouples spatial and temporal dimensions during embedding, enabling more flexible handling both image and video modalities, and achieves the best overall performance.
Effect of Variable Spatiotemporal Streams. We begin with baseline where the model is pretrained using only 16-frame video clips. To enhance temporal modeling, we introduce variable-length frame inputs (e.g.. 1, 2, 4, 8, 16 frames) with dense labels, as shown in Table 3c On K400, performance remains stable across frame combinations. In contrast, the more temporally complex SSV2 benefits notably from dense supervision with diverse frame counts, suggesting helps capture short-range temporal dynamics. To qualitatively assess the short-range temporal robustness of our model, we visualize patch-level features using PCA, as shown in Fig. [6 The model produces consistent local semantic representations across variety of image and video inputs, demonstrating strong spatial grounding and frame-invariant semantic encoding. Together, these findings validate the effectiveness of our variable-frame pretraining strategy and highlight the model's generalization capability.
Qualitative Analysis of Frame-Agnostic Semanties. We visualize the object-level feature distri- bution using T-SNE projection on K400, comparing both image and video samples. As shown in Fig. our model produces well-formed and compact clusters, where image and video instances from the same semantic class are consistently grouped together. This indicates a strong alignment of visual representations across modalities. The tight intra-class clusters and clear inter-class boundaries demonstrate the model's ability to abstract high-level semantics that are shared between static and temporal visual data. Compared to the State of the Art model, such as SigL.PP and DINOv2, our method achieves superior intra-class compactness and inter-class separation, highlighting its strength in lcarning modality-invariant and semantically consistent features.
The Effect of Clustering. To investigate the impact of different clustering strategies on video understanding, we perform ablation studies at both the event and object levels. As shown in Table 3d using only object-level clustering achieves 62.3% on K400 and 12.5% on SSV2. Clustering only at the event level yields similar performance. However, combining object-level and event-level clustering leads to consistent improvements on both benchmarks (62.7% and 13.2%). indicating that the two levels provide complementary information. Furthermore, when we extend to tranulrtraulartty event clustering (Obj.+Multi-FFy the performance improves further, achieving the best results on both datasets (65.6% and 15.66 15.6%). These results demonstrate that clustering at varying event granularities, in combination with object-level semantics, significantly enhances video understanding.
8

---

1000
UniViT
MCLD
VJJEPA
IJJEPA
DINOv2
Sigl.ip2
Figure 7: Visualization of object feature distributions using T-SNE projection onto spherical space on K400 and K700 dataset. Class tokens serve as query vectors for QKV computation.
Impact of Number of Class Choices. As shown in Table 3e we jointly vary the number of training classes. We observe that performance improves as both values increase, peaking at 1M classes. Interestingly, despite the significant increase in video data, the optimal number of classes aligns with that used in large-scale image pretraining, suggesting that current vidco datasets exhibit relatively low semantic diversity. At the same time, assigning multiple positives per sample proves beneficial for learning richer and more robust representations, especially in complex temporal tasks such SSV2.
5 Related Work
Advances in Visual Representation Learning. The adoption of Vision Transformers 21 39 has become a prevailing paradigm in the field of visual representation learning. Concurrently, equivariant self-supervised learning approaches (20) 45 26 29 [9] have emerged to predict structured data transformations consistent with group theeeth formulations. Masked image modeling methods B0
22 65|
acquire visual representations by reconstructing masked regions of the input image in the pixel space. Moreover, JEPAS 66 g predict masked regions within a learned latent space rather than in the pixel domain. Contrastive Language- Image Pretraining (CLIP) [1] 55 40 50] aligns images and texts within a shared embedding space through instance-lever contrastive supervision. However, existing approaches predominantly focus on either static image understanding or spatiotemporal modeling in isolation. In this work, UniViT structuring shared semantic space for images and videos by modeling intra- and inter-instance, effective transfer to downstream tasks.
24 67
Cluster Discrimination. Instance discrimination methods [17 31 47), exemplified by CLIP [47
69
leverage instance-level contrastive supervision but neglect semantic similarities actoss 15 assign single pseudo-labels per sample, failing adequately represent images containing multiple visual elements. To better capture semantic structures, cluster discrimination methods [14]
28
instances, whereas cluster-based approaches [14 5
15 typically iteratively assign pseudo-labed labels through clustering and train classifiers based these labels, grouping visually similar instances to encourage semantic coherence. However, conventional approaches assign only a single pseudo-label per image, limiting their capacity represent multiple semantic concepts within issue recently addressed by multi-lahel clustering methods such as Unicom 2 and MLCD [4]. In this work, we adopt multi-label clustering to unify the representation learning of images and videos, effectively enhancing the semantic coherence.
72
instance, an
Efficient Training. Recent literature has explored various strategies for efficient CLIP training, such as large-batch optimization (up to 160K) [46] 50) and specialized optimizers like LAMB (7).. RoPE originally designed for language models [54], has also been adapted vision transformers via two-dimensional extensions B33 J. Additionally, significant efforts have focused on effective
data curation and filtering at scale 25 50 67 as well as image recaptioning using MLLMs (33
38 43
57 3 Motivated by these advances, we extend these methodologies to video data, constructing a unified data engine that facilitates robust representation learning across both images and videos.
66
6 Conclusion & Limitation
In this paper, we introduced UniViT, a cluster-driven unified self-supervised learning framework designed to jointly capture structured semantics across both spatial and temporal modalities through clustering and discrimination. Leveraging multi-grame multi-granullaity event-level clustering for videos and object-level clustering for images, Vni first constructs structured semantic clusters across modali- ties during the clustering stage. Subsequently, in the discrimination stage, Vni explicitly incorpo- rates these clusters as pseudo-labels into a contrastive objective, effectively integrating structured semantic representations into unified embedding space. To address limitations inherent traditional

---

position encoding and Exed-input approaches, we further introduced U-RoPE and VS², enhancing semantic stability and flexibility across modalities. Extensive evaluations across multiple benchmarks demonstrate UniViT's superior scalability and its ability to achieve paae-offtheeane performance on various downstream tasks, highlighting its effectiveness in unified visual representation learning.
Limitation: We clarify the limitations of our proposed UniViT: (i): UniViT relies on offline clustering with pretrained embeddings, potentially introducing biases from the initial feature extraction model and limiting its ability to adaptively update cluster assignments during training. (ä): Although our VS strategy supports flexible temporal lengths and arbitrary input resolutions, long sequences or extremely high nigh-resolution inputs may still incur substantial computational costs, potentially constraining practical scalability
limited scenarios.
7 Acknowledgments
This work was supported by the Center of Excellence for Antimicrobial Therapeutics Discovery and Innovation (CEATDI), Grant No. 8002003.
10

---

References
[1] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélic Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang. Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Suudnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, and Sophia Yang. Pixtral 12b. arXivv2410.07073. 2024.
[2] Xiang An, Jiankang Deng, Kaicheng Yang, Jaiwei Li. Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Unicom: Universal and compact representation learning for image retrieval. In ICLR, 2023.
[3] Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu. Changrui Chen, Chunsheng Wu, et al. Llava-onevision- 1.5: Fully open framework for democratized multimodal training. arXiv preprint ar 2r0: 236661 2025.
[4] Xiang An, Kaicheng Yang, Xiangzi Dai, Ziyong Feng, and Jiankang Deng. Multi-label cluster discrimi- nation for visual representation learning. In European Conference on Computer Vision, pages 428 -44. Springer, 2024.
[5] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. arXiv preprint
2,511,,,,1,1199 2019.
[6] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas, Self-supervised learning from images with joint-emhedding. pages 16619-15629, 2023.
[7] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture, 2023.
[8] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec:
A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
[9] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 106.v:2106.088 082544 2021.
[10] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video, 2024.
[11] Daniel Bolya, Po- Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. 13181, 2025.
[12] Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food- 101 Mining discriminative components with random forests. In ECCV, 2014.
[13] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset. https:///githu:: com httpa/////..aaobbaaaa/oooooooaaa
2022.
[14] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[15] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. arXiv preprint XXi:22006.09888 2020.
[16] Joao Carreira, Eric Noland, Andras Banki- Morki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600, 2018
[17] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
[18] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017.

---

[19] Rumen Dangovski, Li Jing. Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačić. Equivariant contrastive learning. arXiv preprint arXiv:211 2111.
[20] Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module improve self-supervised learning. arXiv preprint 1.00242 2023.
[21] Alexey Dosovitskiy, Lucas Boyer, Alexander Kolesnikov, Dirk Weissenborn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR,
2021.
[22] Alaacldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. arxiv preprint arXiv:2401.0555 2024.
[23] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving CLIP training with language rewrites. In NeurlPS, 2023.
[24] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In ICLR, 2024.
[25] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jicyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: search of the next generation multimodal datasets. In NewrIPS, 2023.
[26] Quentin Garrido, Laurent Najman, and Yamn Lecun. Self-supervised learning of split invariant equivariant. Proceedings of Machine Learning Research, pages 11975-10996, 2023.
[27] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyúska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitagg Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842 5850, 2017.
[28] Tiancheng Gu, Kaicheng Yang. Xiang An. Ziyong Feng. Dongnan Liu, Weidong Cai, and Jiankang Deng. Rwkv-clip: A robust vision-language representation learner. wwXiv:2406.06972 2024.
[29] Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka Structuring representation geometry with rotationally equivariant. 2023.
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
[31] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 9729-9738, 2020.
[32] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth, Furosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFF. Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[33] Byeongho Hco, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In ECCV, 2024.
[34] Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 13700-13710, 2024.
[35] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vjjaannraaimman, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Sulcyman, and Andrew Zisserman. The kinetics human action video dataset. 1oXXx::705.06990 2017.
[36] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
12

---

[37] Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrole, Tomaso Poggio, and Thomas Serre. HMDB: large video database for human motion recognition. In ICCV, 2011.
[38] Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. VeCLIP: Improving CLIP training via visual-enriched captions. In ECCV, 2024.
[39] Kunchang Li, Yali Wang, Peng Gao, Guanglu Song. Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint arXiv:2201.04676 2022.
[40] Xianhang Li, Zeyu Wang, and Cihang Xie. CLIPA-v2: Scaling CLIP training with 81.1% zero-shot imagenet accuracy within a $10,000 budget; an extra $4,000 unlocks 81.8% accuracy. arXiv:2306. 15658, 2023.
[41] Haotian Liu. Chunyuan Li, Yuheng Li, Bo Li. Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
[42] Antoine Miech, Jean Saan-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Rareact: video dataset of unusual interactions, 2020.
[43] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. In NewrIPS, 2023.
[44] Maxime Oquah, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran. Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang. Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve. Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut. Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
[45] Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. arXiv preprint arXiv:2204. 11771, 2022.
[46] Hicu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoe V. Le. Combined scaling for zero-shot transfer learning. Neurocomputing, 2023.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)211-2522, 2015.
[50] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5h An open large-scale dataset for training next generation image-text models. In NeurIPS Datasets and Benchmarks, 2022.
[51] Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. A short note on the kinetics-700-200 human action dataset, 2020.
[52] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arxiv: 2012.
[53] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063. 2024.
[54] Jianlin Su, Yu Lu. Shengfeng Pan, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing 2024.
[55] Quan Sun. Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved training techniques for clip at scale. arXiv:2303. 153889 2023.
13

---

[56] Quan Sun, Jinsheng Wang. Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. EVA-CLIP-I8B, Scaling clip to 18 billion parameters. arXiv:2402.04252. 2024.
[57] Feilong Tang. Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang. Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang. Sijin Zhou, et al. Seeing far and elearly: Mitigating hallucinations in milms with attention causal decoding. In CVPR, 2025.
[58] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Naeem, Ibrahim Alahdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual Languug encoders with improved semantic understanding, localization, and dense features. arXiv:2502. 14786. 2025.
[59] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Vidcomac v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14549-14560, 2023.
[60] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Kegin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- -aiiun- model's perception of the world at any resolution. arXiv preprint
(tt:::2409.12991,, 2024.
[61] Wang, Yinan Hc, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Intervvi: A large-scale video-text dataset for multimodal understanding and generation. arXiv:2337.069442 2023.
[62] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
[63] Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of seene categories. International Journal of Computer Vision, 119:3-23 22, 2014.
[64] Yin Xie, Kaicheng Yang, Xiang An, Kun Wu, Yongle Zhao, Weimo Deng, Zimin Ran, Yumeng Wang, Ziyong Feng, Roy Miles, et al. Region-based cluster discrimination for visual representation learning. In ICCV, 2025.
[65] Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao. Qi Dai. and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[66] Hu Xu, Yao Huang, Xiaoqing Ellen Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen tau Yih, Shang-Wen, Li, Saining Xie, and Christoph Feichtenhofer. Altogether: Image captioning via re- re-aligning tet- EMNLP, 2024.
[67] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang-Woe Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. ICLR, 2024.
[68] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue. Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan. Yuqiong Liu, Zeyu Cui, Zhearu Zhang, and Zihan Qiu. Qwen2.5 2024.
technical report. arXiv:2412.151
[69] Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic caption. In ICCV, 2023.
[70] Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training BERT in 76 minutes. In ICLR, 2020.
[71] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
[72] Xianhang Zhan, Jiahao Xie, Ziwci Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In CVPR, 2020.
[73] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang. Yatian Pang. Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. LanguageBind: Extending video- languae pretraining to n- n-modality by language- based semantic alignment, arXiv preprint 1r.0022310.01852, 2023.
14

---

1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim our contributions and scope in Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of limitations in the last section, Conelu- sion& immitation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
15

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We clearly provide the information of our experiment setting in the section, Implementation Details.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

reproducible, they Answer: [No]
The study is based on a proprietary dataset that is subject to confidentiality constraints. Due these restrictions, are currently unable provide public access the dataset and code. We are exploring possibilities for future release, subject to approval.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.ce cc/
public/guides htt////odaammsssoooooo.o.o
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/publ http:ss.ccppbblc/guuueeee Sotps///ssdooSemesisoiiiiiia
ssioiiolioiyiyiioiocc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We clearly provide our experiment setting in the section, Baseline and Imple- mentation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computation resources in Implemen- tation details
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't It make it into the paper).
paper (e.g., preliminary or failed experiments that http://.w...p.ococcccccccsocoaaa
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper strictly adheres all requirements of the NeurIPS Code of Ethics, including transparency in data usage, fairness in research methods.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly acknowledge the original owners of the assets, including code, data, and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We describe the details of how model is used the vision encoder in the LLM-based MLLM, along with the corresponding experimental settings, in ViT as an Encoder of MILM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy lattps://neurij: http://urrps.cccoooooaarrrr///// for what should or should not be described.
21


Reviews:
[{"review_id": "mJS8K0PP32", "reviewer": "Reviewer_LTUC", "summary": "Conventional pre-training methods tend to specialize in either the spatial (image) domain or the temporal (video) domain. In this study, the authors propose UniViT, a framework that performs self-supervised learning jointly on images and videos. UniViT employs cluster-based contrastive learning, grouping object-level features in images and event-level features in videos. On the video side, the VS² module enables the model to handle clips of varying frame lengths, while a U-RoPE stabilizes spatiotemporal encoding. Experiments show that UniViT outperforms existing methods across a wide range of image and video tasks.", "questions": "- I would like the authors to address the weaknesses I pointed out. \n- The paper states that a 1:1 ratio between images and videos was maintained during training. Does this ratio refer to the number of images versus the number of video frames (e.g., 2 K videos × 16 frames = 32 K frames, which would not match the number of images), or is it defined by the number of tokens after aggregation with VS²? Please clarify this design choice and the reasoning behind it.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The paper clearly and concisely presents the problem background, objectives, and proposed method, making it very easy to follow. Differences from prior work are explicitly discussed, and the novelty of the approach is well-articulated.\n- Training spatial and temporal information with a single encoder is intuitive and convincing. The paper explains the rationale behind each design choice, and the method achieves strong results across diverse image and video benchmarks, demonstrating its effectiveness. In addition, large-scale scaling experiments (data volume, model capacity, input resolution) show a consistent, monotonic performance gain.\n- In the verification, the effectiveness of each feature of the proposed method, such as U-RoPE, was confirmed through various ablations, enhancing the reliability of this study.\n\nWeaknesses:\n- The method is claimed to learn object-level representations for images and event-level representations for videos. Although ablations show that combining both improves performance, further analysis is needed to verify that the learned features (or clusters) truly capture these properties. In particular, it should be analyzed what kind of feature representations can be obtained for long-term and short-term events.\n- As the authors also point out, since clusters are determined offline, biases from initial feature extraction remain, posing challenges related to domain shift.", "was_revised": true, "final_justification": "My remaining concerns have been largely resolved. Since I have already given an Accept rating, I intend to maintain it.", "rebuttal": ">**W1: The method is claimed to learn object-level representations for images and event-level representations for videos. Although ablations show that combining both improves performance, further analysis is needed to verify that the learned features (or clusters) truly capture these properties. In particular, it should be analyzed what kind of feature representations can be obtained for long-term and short-term events.**\n\n**A1:** Thank you for pointing out the need for deeper analysis of the learned object-level and event-level representations. We acknowledge that interpreting feature behavior across different temporal scales is important, and we have added detailed visualization and clustering analyses to better understand how our model captures both short-term and long-term event structures.\n\nOur current evaluation primarily focuses on quantitative results (e.g., performance on K400 and SSV2) and qualitative clustering visualizations (Fig. 6 and Fig. 7), which demonstrate the model’s effectiveness in unifying image and video representations. Our goal is to learn representations at multiple semantic granularities: for example, from single-frame inputs to 16-frame clips, we expect the model to capture both global semantics and fine-grained object-level details.\n\nTo analyze this, we examined cluster visualizations. We observed that object-level clusters often capture single-object semantics (e.g., \"cake\", \"wedding dress\") or simple multi-object scenes (e.g., \"a child with a backpack\"), while event-level clusters correspond to action or event semantics (e.g., \"mixing cement\"). This suggests the model is indeed separating visual concepts across semantic levels. We will include representative cluster visualizations in the final version to further support this analysis.\n\nWhile our current analysis offers initial insights, it does not yet provide fine-grained verification of the temporal specificity of event representations, particularly in distinguishing long-term versus short-term dynamics. Our framework is designed with the intent that long-range segments encode high-level, holistic event semantics, whereas short-term segments capture finer-grained procedural steps or sub-actions. To facilitate this, we adopt an offline clustering strategy with a high-capacity vocabulary (1 million clusters), enabling the generation of detailed pseudo-labels that promote segment-level semantic learning. We consider this an important direction for future research and will provide a more in-depth discussion on temporal decomposition analysis in the final version of the paper.\n\n>**W2: As the authors also point out, since clusters are determined offline, biases from initial feature extraction remain, posing challenges related to domain shift.**\n\n**A2:** Thank you for your constructive comment, this indeed touches on a key limitation of our method. As you pointed out, the clustering process is performed offline based on initial features, meaning the model cannot dynamically update representations during training. This can potentially introduce bias from the initial features and affect generalization, especially under domain shifts.\nHowever, we would like to clarify that generating pseudo-labels via clustering introduces only a weak and indirect dependency on the initial model, in contrast to direct distillation from a teacher model. This mitigates the influence of the initial features to some extent.\nOur choice of an offline clustering strategy was a trade-off between training stability and computational efficiency. Compared to iterative clustering methods like DeepCluster [1], our approach maintains scalability and avoids the instability that may arise from end-to-end joint optimization on large-scale datasets.\nWe recognize that the risk of bias still exists. To address this, we carefully selected the initial model based on findings from the MetaCLIP [2] paper, opting for CLIP, which has been shown to be among the least biased available models.\nIncorporating iterative or dynamically updated clustering could further enhance semantic modeling and cross-domain robustness. This is indeed a key direction for our future work, and we will make this limitation and its potential solutions clearer in the revised paper.\n\n> **Q1: I would like the authors to address the weaknesses I pointed out.**\n\n**A1:** Thank you again for your thoughtful feedback. We have carefully addressed each of the concerns you raised, including verifying the object-level and event-level representations, analyzing short-term and long-term temporal structures, and mitigating biases introduced by offline clustering. We hope our detailed responses and new experimental results help resolve the issues and demonstrate the strengths and robustness of our approach. We would be grateful if you could consider updating your rating and confidence accordingly.\n\n\n> **Q2: The paper states that a 1:1 ratio between images and videos was maintained during training. Does this ratio refer to the number of images versus the number of video frames (e.g., 2 K videos × 16 frames = 32 K frames, which would not match the number of images), or is it defined by the number of tokens after aggregation with VS²? Please clarify this design choice and the reasoning behind it.**\n\n**A2:** Thank you for the detailed question. We follow the first interpretation you mentioned, the 1:1 ratio refers to the number of images versus the number of video frames. For example, 2K videos × 1 clip (e.g., 16 frames) are treated as 2K video samples, matched with 2K image samples.\nWe conducted preliminary ablation studies to compare different image-to-video data ratios. Our design prioritizes improving performance on video tasks while maintaining competitive image performance, aiming to enhance video modeling without sacrificing image understanding, as compared to MLCD. The table below shows the results on UniViT-S with varying data ratios.:\n\n**Table 1:** Ablation results on different image-to-video sample ratios, showing that a balanced 1:1 ratio provides a good trade-off between video and image performance.\n| Data (I:V) | SSV2 | K400 | ImageNet |\n| ----------- | ---- | ---- | -------- |\n| 1:3         | 16.4 | 64.2 | 81.3     |\n| 1:1         | 16.8 | 67.6 | 80.4     |\n| 3:1         | 16.9 | 67.8 | 79.8     |\n\nThese results suggest that using a balanced 1:1 image-to-video sample ratio yields the most favorable trade-off between video and image performance. While increasing the video ratio (1:3) slightly improves SSV2 performance, it leads to a drop in image accuracy. Conversely, increasing the image ratio (3:1) compromises video modeling. This supports our design choice of maintaining a balanced data composition to ensure strong video understanding without significantly degrading image-level representation learning.\n\nReferences:\n\n[1] *Deep Clustering for Unsupervised Learning of Visual Features*\n\n[2] *Demystifying CLIP Data*"}, {"review_id": "bAamFENW08", "reviewer": "Reviewer_FTq5", "summary": "This paper introduces UniViT, a unified vision encoder designed to understand both images and videos within a single framework. The core contribution is a cluster-driven, self-supervised learning paradigm that uses offline clustering to generate pseudo-labels for object-level semantics in images and event-level semantics in videos. The framework is enhanced by two novel components: Unified Rotary Position Embedding (U-ROPE) to better model spatiotemporal relationships and Variable Spatiotemporal Streams (VS2) to flexibly handle inputs of varying video frame lengths.", "questions": "1. The use of offline clustering on pre-computed features is a core design choice. Could you elaborate on the potential impact of the initial feature extractor's quality on the final performance and discuss whether an online or iterative clustering approach was considered to allow the cluster assignments and the encoder to adapt to each other during training?\n\n2. The U-ROPE design allocates low frequencies for temporal encoding and high frequencies for spatial details based on a 75/25 split. What is the intuition behind this specific allocation, and how sensitive is the model's performance to different frequency distribution ratios between the spatial and temporal components?\n\n3. The paper successfully unifies image and video modalities. Could you discuss the primary challenges you foresee in extending the UniViT framework to other data types, such as audio or text, to create an even more broadly unified \"n-modality\" encoder?\n\n4. The combination of object-level and multi-granularity event-level clustering is shown to be highly effective. How were the different temporal scales for multi-event segments (e.g., 1, 2, 4, 8, 16 frames) chosen, and is there an optimal combination or a point of diminishing returns when adding more scales?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper tackles the important and challenging problem of creating a unified representation for both static images and dynamic videos, which is a significant step toward more generalized visual understanding.\n\n2. The proposed method is technically sound, combining established techniques like clustering and contrastive learning in a novel two-stage framework that effectively bridges spatial and temporal semantics.\n\n3. The experimental results are extensive and demonstrate state-of-the-art performance across a wide range of benchmarks, including attentive/linear probing and multimodal VQA tasks, validating the effectiveness of the approach.\n\n## Weaknesses\n\n1. The method's reliance on offline clustering with embeddings from a separate pretrained model is a significant weakness, as it may introduce biases from the initial model and limits the framework's ability to learn end-to-end.\n\n2. The research suffers from a major reproducibility issue, as the authors state that the code and a key dataset are proprietary and cannot be released, making it difficult for the community to verify the results or build upon the work.\n\n3. The ablation study on the number of cluster classes suggests that performance peaks at 1 million classes and then declines, but the reason for this decline is not fully explored", "was_revised": true, "final_justification": "I suggest accept this paper since the authors' rebuttal address most of my concerns.", "rebuttal": ">**W1: The method's reliance on offline clustering with embeddings from a separate pretrained model is a significant weakness, as it may introduce biases from the initial model and limits the framework's ability to learn end-to-end.**'\n\n**A1:** Thank you for your thoughtful examination of this key design aspect. We acknowledge the concern that relying on offline clustering with embeddings from a separately pretrained model could introduce biases from the initial model and may, to some extent, limit the end-to-end trainability of our framework. We would like to clarify the motivation and benefits behind this design choice:\n\n1. Compared to alternatives such as distillation or masked modeling, our clustering-based approach is actually less dependent on the initial model. We do not directly mimic its features; instead, we use the semantic embeddings only to generate compact and robust pseudo-labels through clustering, which then guide downstream training.\n2. Clustering allows us to explicitly model hierarchical semantic structures, e.g., object-level and event-level concepts, which are difficult to capture directly through most self-supervised objectives.\n3. The offline clustering significantly reduces computational cost during training, enabling us to scale effectively to large mixed image-video datasets.\n\nWe also recognize the limitations of this design in certain scenarios, such as domain generalization or task transferability. To address this, future work will explore integrating the clustering objective into an end-to-end trainable pipeline, for example, through online clustering or proto-token matching, to make the pseudo-labeling process learnable, reduce dependency on the initial model, and improve generalization. We will include a discussion of this trade-off and future direction in the final version of the paper.\n\n>**W2: The research suffers from a major reproducibility issue, as the authors state that the code and a key dataset are proprietary and cannot be released, making it difficult for the community to verify the results or build upon the work.**\n\n**A2:** Thank you for your feedback. After careful consideration and discussion, we commit to releasing all pretrained models and code to support the development of vision foundation models and multimodal large language models (MLLMs). We believe this will provide valuable resources to the community for both verification and further research. Additionally, we will open-source the complete evaluation pipeline to ensure full reproducibility of our results.\n\n>**W3:The ablation study on the number of cluster classes suggests that performance peaks at 1 million classes and then declines, but the reason for this decline is not fully explored.**\n\n**A3:** Thank you for raising this important point. You are right that the earlier version of our paper lacked a detailed analysis of clustering sensitivity. To address this, we conducted an ablation study focused on the number of cluster classes. The results are shown below:\n|Num Classes|K400|SSV2|\n|-|-|-|\n|500k|65.2|15.5|\n|1M|65.6|15.6|\n|1.5M|65.6|15.4|\n|2M|65.8|15.5|\n|3M|65.4|15.5|\n|4M|65.4|15.4|\n|5M|65.2|15.4|\n\nWe observed that performance peaks around 1 million clusters, and then gradually declines as the number increases. We believe the degradation is due to several factors:\nAs the number of clusters increases, each cluster contains fewer samples, weakening semantic cohesion and reducing the effectiveness of pseudo-labels.\nExcessively fine-grained clustering can lead the model to overfit local patterns, especially for short or semantically redundant video segments. A larger number of clusters increases the chance of noisy or unstable pseudo-labels, which may hurt contrastive training and generalization. Based on this analysis, we chose 1M clusters as the default configuration, striking a good balance between performance and computational cost. In future work, we plan to explore dynamic cluster allocation strategies to further improve clustering quality and model robustness.\n\n>**Q1: How does the quality of the initial feature extractor impact final performance, and was online or iterative clustering considered to allow joint adaptation of encoder and cluster assignments?**\n\n**A1:** Thank you for the important question. Our use of offline clustering based on pre-computed features is a deliberate design choice aimed at reducing reliance on the initial model while maintaining training efficiency. Unlike distillation or MIM, we don’t directly mimic features but instead use semantic embeddings to generate pseudo-labels via clustering, mitigating initial bias.\nIn addition, this choice brings several practical advantages:\n1. On large-scale image-video mixed datasets, offline clustering significantly reduces computational overhead during training. Unlike online or iterative clustering approaches, we avoid repeatedly alternating between clustering and training, which can be costly and unstable, especially under limited computing resources.\n2. Maintaining global cluster centers allows us to bypass the need for extremely large batch sizes often required by contrastive learning or distillation methods. This leads to a more lightweight and flexible training paradigm.\n\nAs part of future work, we are planning to explore two lightweight alternatives to enable better co-evolution:\n1. Periodically update cluster centers using exponential moving average of features during training, without full re-clustering, to balance adaptability and efficiency.\n2. Design learnable or adaptive strategies to update cluster centers throughout training, enabling the model to better adjust to evolving feature distributions.\n\n>**Q2: What is the intuition behind this specific allocation, and how sensitive is the model's performance to different frequency distribution ratios between the spatial and temporal components?**\n\n**A2:** Thank you for the great question. Our decision to allocate a larger portion of the frequency spectrum to the spatial dimensions (75%) and a smaller portion to the temporal dimension (25%) is based on two core intuitions:\n1. Compared to spatial resolution, videos typically have shorter sequence lengths in the temporal axis. Hence, fewer dimensions are needed to encode temporal positions effectively.\n2. Allocating more dimensions to spatial encoding preserves high-frequency spatial details, which improves spatial semantic representation and enhances local perceptual sensitivity.\nTo assess the sensitivity of the model to this frequency allocation, we conducted an ablation study on the UniViT-S model using different frequency split ratios. The results on SSV2, K400, and ImageNet are as follows:\n|Frequency (Temporal/Spatial) |SSV2| K400| ImageNet |\n|-|-|-|-|\n|5:6|16.6|67.9|80.8|\n|3:4|16.8|67.6|80.4|\n|1:2|16.7|67.6|80.1|\n\nThese results indicate that while the model is relatively robust to small variations in the split ratio, the 75/25 allocation offers a good trade-off between spatial and temporal modeling performance. We will add this discussion to the revised version for clarity.\n\n>**Q3: The paper successfully unifies image and video modalities. Could you discuss the primary challenges you foresee in extending the UniViT framework to other data types, such as audio or text, to create an even more broadly unified \"n-modality\" encoder?**\n\n**A3:** Thank you for the inspiring question. While UniViT is primarily designed to unify visual modalities, we are indeed interested in extending it toward a more general \"n-modality\" unified encoder. However, this direction introduces several fundamental challenges:\n\n1. Visual data are continuous signals in space and time, while text is a discrete sequence of symbols, and audio is a continuous waveform or spectrogram. These modalities differ significantly in representational structure, resolution, and temporal granularity, making unified modeling inherently difficult.\n2. Images and videos can share a common tokenizer, but it is much more challenging to use the same tokenizer across modalities like speech and text.\n\nDespite these challenges, we believe UniViT provides a strong foundation. In future work, we are interested in exploring joint multimodal pretraining, shared cluster anchors, or cross-modal alignment heads as promising directions toward a truly unified multimodal framework.\nWe are very excited by this possibility and would be happy to further discuss any ideas you might have for building such a generalized system.\n\n>**Q4: How were the different temporal scales for multi-event segments (e.g., 1, 2, 4, 8, 16 frames) chosen, and is there an optimal combination or a point of diminishing returns when adding more scales?**\n\n**A4:** Thank you for your question. We will explan for it. In particular, using 2-frame segments contributes little to performance improvement. We believe this is because such short clips are insufficient to capture meaningful event-level semantics, and thus provide limited benefit in modeling video dynamics.\nHowever, we still include 1- and 2-frame segments for an important reason: they help better align video segments with single-frame image representations, especially under the 1-frame setting. This alignment is critical for achieving unified modeling between images and videos. The following table presents our ablation results.\n\n|Frames |K400 |SSV2|Imagenet|\n|--|--|--|--|\n|Baseline |62.7 |13.2|79.7|\n|+8 |64.4 |14.2|79.8|\n|+4,8| 65.2 |15.3|79.7|\n|+1,2,4,8 |65.6| 15.6|80.2|\n\nThese results confirm that while longer event segments (e.g., 4 or 8 frames) contribute most to video understanding, including shorter ones (1–2 frames) still offers gains by improving alignment across modalities. If we further increase the temporal length of input sequences, it remains an open question whether additional temporal scales would continue to yield substantial benefit. We consider this a promising direction for future work."}, {"review_id": "flFdTsS8Ss", "reviewer": "Reviewer_mwRn", "summary": "This paper proposes UniViT, a unified, unsupervisedly trained vision encoder for both images and videos. UniViT uses a cluster-based contrastive learning method. It first groups inputs into sets with varying numbers of frames, then pools the features within each group for contrastive learning. The paper introduces U-RoPE to address the temporal resolution problem in M-RoPE by rearranging temporal position information in higher-dimension channels. Comprehensive experiments are conducted on single-modality understanding, its use as a vision encoder for MLLMs, and its scaling properties.", "questions": "1. Considering in current results table, UniViT excels at only video tasks, a more comprehensive comparison between UniViT with common video encoders is needed to evaluate the effectiveness of UniViT. There should be common video encoders like VideoVAE, V-JEPA on common protocols like full-finetuning, full attentive probing, and linear probing.\n2. The design U-RoPE need more validation. First, periodic oscillation should be confirmed as a real issue, second, why spatial details do not cause this issue?\n3. The writing of the paper need to be polished.", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The experiments are comprehensive. They include attentive probing, linear probing, using the encoder for MLLMs, and evaluating scaling properties. The ablation studies on pretraining, U-RoPE, frame groups, and clustering are also thorough.\n2. The intuition behind U-RoPE is reasonable and novel. The dimensional split in M-RoPE is suboptimal because time (t), height (h), and width (w) are separated into different frequency levels, which can cause resolution issues.\n3. The model's performance is generally superior in attentive probe evaluations and when used in MLLMs.\n\n**Weaknesses**\n1. **Limited novelty.** The paper appears to be a direct extension of MLCD from images to video. An \"event\" is simply a video clip sampled at a different frame rate.\n2. **The experimental section is problematic.**\n\t1. For a unified image and video encoder, most of the methods compared in the experiments are image-based. The model does not demonstrate superior performance on image-based tasks, and few video-only models are included for comparison.\n\t2. In Table 1, the paper uses 50-shot attentive probing, which is an uncommon setting, making it difficult to verify its effectiveness. For example, the paper reports 96.9 accuracy for PE-Core/G in a *50-shot* setting, whereas the original PE-Core paper reports 96.9 accuracy in a *zero-shot* setting.\n\t3. In Table 2, only two of the compared models are trained with video data, and the video-based models from Table 1 are not included. Many video encoders achieve better linear probing performance. For example, on SSV2, VideoMAE achieves 38.9, AdaMAE achieves 40.1, and even MoCo v3 reaches 33.7 (as reported in the VideoMAE paper). In contrast, UniViT only achieves 27.4.\n3. The rationale for U-RoPE is not fully validated. Is the periodic oscillation described in L.115-118 a confirmed issue? It seems that increasing the base wavelength, $C$, could mitigate this issue. For instance, Qwen2.5-VL uses a `rope_theta` of 1,000,000.0, which should be more than sufficient for 16 frames. Furthermore, the performance gap between M-RoPE and U-RoPE shown in Figure 3c is small.\n4. **The writing is often unclear and difficult to understand.** For example:\n\t1. The definitions of \"object\" and \"event\" are not clear. From my understanding, an \"object\" refers to an image or a single frame, while an \"event\" refers to a sampled clip with a varying number of frames. If this is the case, the terminology is convoluted and hinders comprehension.\n\t2. L.94: What is the VS$^2$ strategy? Additionally, the aggregation function is not explained in the text and is only shown in a figure as an \"attentive pooler.\"\n\t3. L.146: What is Figure 3d intended to convey? At a minimum, image-to-image and video-to-video similarities should be presented to demonstrate that the image and video representations are distinct. Even if this were shown, it is unclear how this would strengthen the paper's claims.\n\t4. L.315 mentioned a initial feature extraction model, what is it?", "was_revised": true, "final_justification": "I appreciate the authors' extensive effort in the rebuttal and their engagement during the discussion period. However, the discussion is meant for clarification, not continuous rebuttal, and it is no longer productive. I've carefully read the rebuttal and reviews from other reviewers. Despite the authors' detailed responses, my main concerns about the paper's novelty and clarity remain.\n\n1. Limited Novelty: To me, the method still appears to be an extension of MLCD[1], but with video clips added as input. The core methodology feels very similar.\n\n2. Multi-Scale Approach: Using video clips with a different number of frames is not a new idea. It's a similar technique in self-supervised video learning, used in methods like ASCNet[2]. A more common approach is to use different fps (for fixed-length input) like in TCLR[3]. Framing it as a novel \"cross-granularity semantic transition pathway\" seems like an overstatement.\n\n3. Poor Writing and Clarity: The paper is difficult to read. It initially missed some key details. Even after clarification, some explanations are unconvincing. For example, the justification for Figure 3d is still meaningless; you can't compare similarity scores between different models because they operate in entirely different feature spaces. It is like comparing similarities between different sizes of CLIP models is meaningless, which does not really demonstrate anything.\n\nWhile the paper has some merits as proposed by other reviewers, such as the U-RoPE concept and some strong downstream performance results, the fundamental weaknesses in novelty and clarity are too significant to overlook.\n\nI am raising my rating to \"Borderline reject\" to acknowledge the authors' efforts and the positive aspects of the work. However, I cannot recommend acceptance in its current form.\n\n[1] Multi-label Cluster Discrimination for Visual Representation Learning\n[2] ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency\n[3] TCLR: Temporal contrastive learning for video representation", "rebuttal": "> **W1: Limited novelty.**\n\n**A1:** We understand the concern that the contribution of our work may appear limited. We would like to clarify several key points that demonstrate the conceptual and novelty of our work:\n1. Unlike MLCD, which is primarily designed for image-language tasks, our proposed UniViT introduces a unified Transformer-based architecture capable of jointly handling images and videos event representations. Rather than simply stacking image processors on top of video sequences, UniViT unifies the input encoding scheme, spatiotemporal interaction modules, and training objectives across different modalities.\n2. A direct extension of MLCD to video often leads to degraded image performance. Increasing the proportion of image data to compensate hampers video understanding. To resolve this conflict, we introduce a novel multi-granularity event modeling approach that balances image and video training. This not only preserves image classification performance but also achieves SOTA results on video benchmarks. Furthermore, it enables the model to generalize across varying temporal scales, a capability beyond MLCD.\n3. UniViT consistently achieves superior performance across a wide range of downstream tasks, including image classification, video understanding, and action recognition. These results validate the practical effectiveness of our unified architecture and demonstrate its potential as a general-purpose vision foundation model.\n\nWe will further revise the paper to clearly articulate the motivation and conceptual innovations over MLCD and related work. We sincerely thank the reviewer for raising this point and helping us improve the clarity of our contributions.\n\n> **W2.1: Lack video-only baselines, and image performance is not clearly superior.**\n\n**A2.1:** Thank you for your suggestion.\n1. Our method focuses on extending the video capabilities of the Unified framework, while preserving image performance. As shown in our experiments, our performance on image tasks is competitive with MLCD, while achieving significant improvements in temporal understanding, such as a 10% gain on SSV2. To further address this concern, we have added comparisons with MLCD and UniViT on several image benchmarks, as shown below:\n| Method|Aircraft|DTD |Cars|Birdsnap|Pets|\n|-|-|-|-|-|-|\n| MLCD-L|86.7|80.9|82.8|86.6|89.2|\n| UniViT-L|87.1|81.4|82.3|86.7|88.6|\n\n2. To address the lack of comparisons with video-only models on image tasks, we further evaluate such models under the same attentive probing setting as V-JEPA on image benchmarks. Despite not being tailored for image, these video models provide a reference for image generalization. The results below demonstrate that UniViT achieves clearly superior performance, further supporting its effectiveness in unified representation learning.\n| Method|Places|iNat|Imagenet|\n|-|-|-|-|\n|V-JPEA-L|66.4|60.3|72.6|\n|VideoMAE-H|59.1|65.5|72.3|\n|UniViT-L|70.8|86.4|84.8|\n\n>**W2.2: 50-shot attentive probing is uncommon**\n\n**A2.2:** Thank you for pointing out the ambiguity in our experimental setup.\n1. Our evaluation protocol follows the 50-shot attentive probe setting introduced in V-JEPA, aiming to measure the model's transfer ability and data efficiency.\n2. Regarding the Food dataset, both zero-shot and full-attentive probe results for PE-Core reach the same reported accuracy of 96.9%. We believe this is because PE-Core/G already achieves near-saturation performance on Food, leaving little room for further improvement through probing. This also indicates the strong generalization capability of PE-Core in the zero-shot setting.\n3. Additionally, we re-evaluated PE-Core and reproduced an accuracy of 95.8%, which is close to the originally reported value.\n4. For video datasets, we also adopted a single-view attentive probing setting, training for 20 epochs using the full dataset. The results are summarized below:\n| Method|SSV2|K400|K600|K700|\n|-|-|-|-|-|\n| V-JPEA-L|66.7|75.9|77.7|66.5|\n| UniViT-L|51.3|85.5|86.1|75.9|\n\n>**W2.3: SSV2 result in VideoMAE**\n\n**A2.3:** We sincerely appreciate your detailed review.\n1. Regarding the SSV2 performance, we would like to clarify that the results reported by VideoMAE are based on pretraining only on SSV2, which may lead to overfitting.\n2. In contrast, our UniViT is pretrained on large-scale multimodal data without including SSV2, aiming to evaluate its generalization capability.\n3. To further assess the effectiveness of our method, we conducted an additional experiment by including SSV2 in the training set and reported linear probing results as follows:\n|Dataset|UniViT-L|VideoMAE|AdaMAE|MoCo v3|\n|-|-|-|-|-|\n|SSV2|47.1|38.9|40.1|33.7|\n\n> **W3: Wavelength mitigate periodic oscillation**\n\n**A3:** Thank you for raising this insightful question.\nWe want to explain that our design is driven by two considerations:\n1. We aim to support much longer video inputs in future settings enhancing the model’s ability to capture long-range dependencies without segment-level fragmentation, which often results in information discontinuity. To accommodate this extended length, we cannot rely solely on wavelength-based approaches.\n2. Our dense labeling strategy is designed to reduce redundancy between adjacent frames and enhance local discriminability. While increasing the base wavelength slows the phase progression and might reduce aliasing, it also risks making the model less sensitive to subtle temporal changes, counter to our goal of capturing fine-grained temporal dynamics.\n\nWe will elaborate on these design choices more clearly in the revised paper.\n\n>**W4.1: Definitions of \"object\" and \"event\"**\n\n**A4.1:**  Thank you for pointing this out. We sincerely apologize for the confusion.\nOur choices stem from the nature of the clustering process. In our framework, image-level clustering tends to produce object-level semantics, such as dresses or cakes, which are static and visually coherent. In contrast, clustering of video clips captures temporal dynamics, often corresponding to event-level semantics like a foot-washing ceremony or cement mixing. These inherently involve temporal evolution and are characterized as \"events.\"\nOur goal is to enable the model to learn global representations for longer clips, while shorter sequences help learn fine-grained sub-event representations (such as steps). To support this, we perform offline clustering with up to 1 million clusters, enabling fine-grained labeling of video segments. This provides the model with richer pseudo-supervision and helps capture subtle temporal patterns.\n\n>**W4.2: What is the VS2**\n\n**A4.2:** Apologize for it. The core idea of VS² is to better model temporal structure by dividing a video into multiple event segments of varying lengths before the encoder. To improve temporal discrimination across events, we manipulate the positional encoding such that frames from different events are spaced farther apart in the positional domain, while maintaining the relative structure within each event. For example, consider where a 16-frame video is divided into four 4-frame events. Instead of applying standard positional encoding from 0 to 15, we encode each 4-frame event as [0+i*16,1+i*16,...] for the i-th event. This effectively enlarges the relative distance between different events, encouraging the model to capture fine-grained intra-event details.\nAs for the aggregation function, it is implemented as an attentive pooling module, which maps variable-length feature sequences into a fixed-dimensional representation.\n\n>**W4.3: What is Figure 3d intended to convey?**\n\n**A4.3:** Apologies for the earlier confusion regarding Fig. 3d.  Its purpose is to measure how similar a video representation is to a single frame from the same video, high similarity implies weak temporal modeling. UniViT shows lower similarity than other models, indicating stronger temporal awareness and better distinction between static and dynamic content.\n| | UniViT-L |PE-Core-L| MLCD| \n|-|-|-|-|\n|Similarity|0.65|0.78|0.86|\n\n>**W4.4: What is the initial feature extraction model?**\n\n**A4.4** Thank you for your question. The initial features used during the clustering stage are extracted using the CLIP model, following the same setup as described in the MLCD paper to ensure a fair comparison. We chose CLIP for its strong semantic generalization capabilities, which helps improve the quality of the clustering results.\n\n>**Q1: Common protocol**\n\n**A1:** Thank you for the helpful suggestion. We agree that a more comprehensive comparison between UniViT and representative video encoders is important to assess the effectiveness of UniViT.\nTo address this, we have additional evaluations of UniViT under common protocols. The results are summarized below:\n|Method(AP)|SSV2 (16×1×1)|SSV2 (16×2×3) | K400 (16×1×1)| \n|-|-|-|-|\n| **50-shot** | | | |\n| V-JPEA-L| 44.2| 48.2|74.7|\n| UniViT-L|27.3| 31.6|82.9|\n| **Full**| | | |\n| V-JPEA-L| 66.7(+22.5)| 69.7(+21.5)|75.9|\n| UniViT-L| 51.3(+24.0)| 55.2(+23.6)|85.5|\n| **Finetune**| | | |\n| V-JPEA-L| 73.2| 75.1|-|\n| UniViT-L| 71.6| 73.2|-|\n\n>**Q2: Why spatial do not cause the issue**\n\n**A2:** Thank you for your thoughtful question. We would like to clarify that high-frequency periodic oscillation is not inherently problematic, representing a certain bias.\nUnder a shared RoPE strategy, different dimensions behave differently. Prior works have shown:\n1. Lower dimensions are sensitive to local semantics and relative distances.\n2. Higher dimensions are suited for capturing long-range dependencies.\n\nIn space, high frequencies aid fine detail modeling; in time, they can disrupt temporal coherence. To ensure stable temporal representation, we assign lower frequencies to the temporal axis. We'll clarify this design choice in the revision.\n\n>**Q3: The paper need to be polished.**\n\n**A3:** Thanks for your suggestion. We appreciate your feedback and will carefully revise the writing for clarity and polish in the final version."}, {"review_id": "nTrMNTnzgd", "reviewer": "Reviewer_Sxna", "summary": "This paper presents UniViT, a unified Vision Transformer architecture designed for both image and video understanding. The central innovation lies in its clustering-based self-supervised pretraining strategy. Specifically, this method assigns object-level and event-level classes to image and video data and use such pseudo-supervision for contrastive pretraining. Furthermore, the authors introduce two architectural contributions: U-RoPE, a position embedding scheme that allocates low-frequency components to the temporal dimension, and variable spatiotemporal streams that adapt to input clips of different lengths. UniViT is evaluated across a range of benchmarks, showing competitive results in image, video, and multimodal understanding tasks.", "questions": "Please refer the weaknesses section above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1. Broad Evaluation Scope: The paper evaluates UniViT across diverse tasks including image classification, video action recognition, and multimodal understanding, showcasing its versatility..\n\nS2. Novel Clustering Strategy: The introduction of object- and event-level clustering for generating pseudo-labels is conceptually interesting and empirically beneficial for unified representation learning.\n\n\n\nWeaknesses:\n\nW1. Discrepancy in Baseline Results: There are notable inconsistencies between the reported results and the original papers for several baselines. For example, in Table 1, V-JEPA’s low-shot accuracy on SSv2 is 51.9% in [9], but only 44.8% is reported. Similarly, in Table 2, DINOv2's linear probing result on SSv2 is 35.6% in [43], but only 19.6% is reported here. The paper should clarify the evaluation protocols and account for these discrepancies.\n\nW2. Weak Temporal Modeling: Regarding W1, on motion-centric benchmarks such as Something-Something V2, UniViT underperforms compared to video-specific models like V-JEPA (low-shot results 32.1% vs. 51.9%)—and even to image models like DINOv2 (linear probing results 27.4% vs. 35.6%) in some cases. This raises concerns about the method’s ability to capture temporal dynamics effectively.\n\nW3. Missing Full Finetuning Comparison: The paper lacks a comparison under full fine-tuning settings, which is critical for comprehensive evaluation of the learned representations in downstream tasks..\n\n[a] Wei et al., “VideoRoPE: What Makes for Good Video Rotary Position Embedding?” ICML, 2025.", "was_revised": true, "final_justification": "My main concerns were around performance discrepancies and weak temporal modeling, and I believe both were sufficiently clarified in the provided rebuttal. While the full fine-tuning performance could be stronger, it does not warrant rejection on its own. I thus update my score to borderline accept.", "rebuttal": "> **W1: Discrepancy in Baseline Results: There are notable inconsistencies between the reported results and the original papers for several baselines.** \n\n**A1:** Thank you for your careful review and for pointing out the ambiguities in our experimental setup. We have addressed your concerns from the following three perspectives: (1) evaluation protocol differences across baselines, (2) probing vs. full-data settings, and (3) unified comparison with and without multi-view testing.\n\n1. Unlike works such as V-JEPA, which evaluate models using multiple spatial and temporal views, a strategy known to improve results through test-time ensembling, we intentionally adopted a simpler evaluation protocol that uses only a single center spatial crop and a single temporal clip. This minimal setup is designed to transparently assess the intrinsic quality of the learned visual representations, without relying on auxiliary inference strategies. Our objective is to isolate the encoder’s contribution and enable fair, consistent comparisons across all baselines under identical conditions.\n\n2. We apologize for the confusion caused by the lack of detailed explanation in the main paper. Due to space constraints, we did not explicitly specify that all baselines were evaluated under a unified probing protocol to ensure consistency. To further clarify, we standardized our evaluation setup and also reproduced results under the commonly used multi-view configurations for reference. Specifically, we tested two setups: (1) 16 frames, 1 view, 1 clip, and (2) 16 frames, 2 views, 3 clips. All models were trained for 20 epochs using the same backbone, input resolution, and optimization configuration to ensure a fair and controlled comparison. This design helps disentangle the true representational quality of the model from improvements attributed to test-time ensembling or view aggregation.\n\n3. Thank you also for pointing out that we did not clarify the probing strategy on video. In our original submission, all models were evaluated using a consistent 50-shot attentive probing setup, which emphasizes representation quality under limited supervision. In contrast, results such as the 35.6% reported by DINOv2 on SSV2 were obtained using the full training set, which naturally leads to higher performance. To illustrate this, we reproduced DINOv2’s results under both settings. As shown in Table 2, DINOv2 achieves 19.6% with 50-shot probing and 35.2% under full-data training, confirming that the performance gap is due to differences in evaluation protocol rather than implementation discrepancies. We adopted a uniform probing setup across all baselines to ensure fair comparison, and we hope this clarification helps contextualize the reported results.\n\n**Table 1:** Attentive probing results under 50-shot and full-data settings, showing UniViT's significant improvement and competitive performance across SSV2 and K400.\n\n| Method(AP) | SSV2 (16×1×1)  | SSV2 (16×2×3)  |  K400 (16×1×1) | \n|------------|----------------|----------------|----------------|\n| **50-shot** |               |                |                |\n| V-JPEA-L   | 44.2           | 48.2           |       74.7     |\n| UniViT-L   | 27.3           | 31.6           |       82.9     |\n| **Full**   |                |                |                |\n| V-JPEA-L   | 66.7  （+22.5）       | 69.7   （+21.5）        |       75.9     |\n| UniViT-L   | 51.3    （+24.0）       | 55.2   （+23.6）        |      85.5      |\n\n**Table 2:** Linear probing results on SSV2 under 50-shot and full-data settings, illustrating the performance gap caused by limited data.\n| Method(LP) | SSV2 (50-shot)  | SSV2 (Full)  |\n|------------|----------------|----------------|\n| DINOv2     | 19.6            |     35.2     |\n\nThese results show that the performance gap mainly stems from probing vs. full-data settings. Under full supervision, UniViT achieves comparable or superior results, with discrepancies from original reports within 1%, confirming the reliability of our evaluation. We will release our full evaluation code for reproducibility.\n\n\n> **W2: Weak Temporal Modeling: Regarding W1, on motion-centric benchmarks such as Something-Something V2.**\n\n**A2:**  Thank you for your patient and detailed review. We next provide a more detailed analysis to address your concern regarding temporal modeling performance.\n\n1. V-JEPA was pre-trained directly on the SSV2 dataset, which inevitably introduces a degree of data leakage. In contrast, we intentionally excluded SSV2 from our training data to more accurately assess the generalization ability of our model.\n2. To further avoid overfitting on evaluation benchmarks, we trained our video models exclusively on a further refined subset of the InternVid dataset, without mixing in SSV2. Despite this, our model demonstrates competitive performance. For example, UniViT-L achieves 32.1% on SSV2, outperforming PE-Core-G (23.7%), which similarly does not use SSV2 in training, and even surpassing VideoMAE-G (27.9%), which does include SSV2 during pre-training.\n3. To directly address your concern, we also conducted experiments with SSV2 included in the training set. As shown in the table below, our model achieved a +15% performance gain on SSV2 in the 50-shot attentive probe setting, exceeding the performance of V-JEPA-L. Notably, performance on other benchmarks remains stable or even improved, indicating our model's strong robustness and generalization.\n\n**Table 3:** Results with SSV2 included in training, where UniViT-L outperforms V-JEPA-L across all benchmarks, demonstrating strong generalization.\n\n| Method | SSV2 | K400 |K600|K700| RareAct|\n|--------|--------|--------|--------|--------|--------|\n| V-JPEA-L  |  44.9 | 62.3  | 63.2 | 49.2|6.3|\n| UniViT-L  | 47.1  | 83.9  | 84.3|72.8|33.6|\n\nThe results show that including SSV2 in training leads to improved performance on SSV2 without compromising other benchmarks. UniViT-L not only surpasses V-JEPA-L on all datasets but also maintains strong generalization across diverse video tasks.\n\n4. To further validate our temporal modeling capabilities, we evaluated our models on the Charades dataset under the OpenTAD[2] framework. Importantly, neither model was fine-tuned on the downstream task, yet our model shows significantly stronger performance compared to V-JEPA-L on Charades dataset:\n\n**Table 4:** Temporal localization on Charades, where UniViT-L consistently outperforms V-JEPA-L, indicating stronger temporal modeling.\n\n| Method | mAP | tIoU 0.3 | tIoU 0.4 | tIoU 0.5\t| tIoU 0.6\t| tIoU 0.7 |\n|--------|--------|--------|--------|--------|--------|--------|\n| V-JPEA-L  |  6.2 | 9.05 | 7.91| 6.59 | 5.04 | 3.38 |\n| UniViT-S | 12.5  |17.83 | 15.91 | 13.69 | 10.93 | 7.74 |\n\nThis substantial improvement in mAP and consistent gains across all tIoU thresholds demonstrate that UniViT captures richer temporal structures even without task-specific fine-tuning. These results further validate its effectiveness in zero-shot temporal understanding.\n\n> **W3: Missing Full Finetuning Comparison:he paper lacks a comparison under full fine-tuning settings, which is critical for comprehensive evaluation of the learned representations in downstream tasks.**\n\n**A3:** Thank you very much for raising this important point. Our primary goal is to validate the robustness of UniViT by evaluating it under more challenging and diverse scenarios.\n\nWhile our initial focus was on linear and attentive probing to highlight the transferability of the frozen backbone, we recognize that different methods may exhibit varying behavior under full fine-tuning, especially in terms of task-specific adaptation and upper-bound performance. Therefore, in response to your suggestion, we conducted additional full fine-tuning experiments across a wide range of downstream tasks, covering video datasets (e.g., SSV2). These experiments were designed to assess not only accuracy improvements but also the adaptability of the unified architecture under task-specific supervision.\n\nThe table below is a partial summary of the updated results, demonstrating that UniViT performs competitively or even outperforms strong baselines like V-JEPA under full fine-tuning settings.\n\n**Table 5:** Full fine-tuning results showing UniViT-L is competitive with V-JEPA-L on SSV2 even without pre-training on SSV2.\n| Method | SSV2(16×1×1) | SSV2(16×2×3) |\n|--------|--------|--------|\n| V-JPEA-L  |  73.2 | 75.1  |\n| UniViT-L  | 71.6  | 73.2  |\n\nThese results confirm that UniViT not only offers strong frozen representation quality but also adapts effectively under full supervision. We believe this addresses your concern and demonstrates UniViT’s competitiveness in both transfer and fine-tuned settings. Additional results on more benchmarks will be included in the final version.\n\nWe also acknowledge that VideoRoPE [1] is a very strong and well-designed method. We will include it in the discussion in the revised version of the paper. In the updated manuscript, we will provide the full table with complete results and clearly annotate all experimental configurations (e.g., number of frames, views, training protocols, etc.) to support a comprehensive and transparent comparison, including a detailed discussion of VideoRoPE.\n\nWe sincerely hope that our detailed responses and newly added experimental results effectively address your concerns and help to clarify the strengths and robustness of our approach. We would greatly appreciate it if you would consider updating your rating and confidence accordingly.\n\nReferences:\n\n[1] *VideoRoPE: What Makes for Good Video Rotary Position Embedding?*\n\n[2] *OpenTAD: A Unified Framework and Comprehensive Study of Temporal Action Detection*"}, {"review_id": "33JozkSBZV", "reviewer": "Reviewer_Wzoa", "summary": "This paper proposes UniViT, a cluster-driven, self-supervised vision transformer that unifies image and video representation learning. The model jointly encodes object-level spatial semantics from images and event-level temporal dynamics from videos via shared cluster centroids and a unified transformer backbone. Key contributions include:\n(1) Variable Spatiotemporal Streams (VS²) for flexible modeling of varying-length video inputs,\n(2) Unified Rotary Position Embedding (U-RoPE) that decouples spatial and temporal frequencies for position-invariant semantic encoding, and\n(3) a multi-label contrastive learning objective over object- and event-level cluster assignments.\nUniViT achieves strong results across diverse benchmarks.", "questions": "1.\t1. Could you explain why you chose the 3/4 split for frequency allocation in U-RoPE, and how the use of U-RoPE compares to VideoRoPE in terms of performance or suitability across different tasks\n2.\tHow are image inputs integrated into the VS² module? Are they treated as single-frame segments consistent with the video processing pipeline? This should be made explicit.\n3.\tWhy is InternVideo not included in Tables 1 and 2, despite being a strong video-only model with image transfer performance?\n4.\tWill you consider adding OmniMAE, OmniVL, and Omnivore as baselines or at least explicitly positioning UniViT relative to them?\n5.\tHave you explored how sensitive your clustering strategy is to hyperparameters (e.g., number of clusters or k-means initialization)?\n6.\tCould you provide more detail on the training efficiency and computational overhead introduced by VS² and clustering steps, compared to conventional contrastive or masked modeling frameworks?”", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1.\tThe paper addresses a timely and important problem: unifying image and video representation learning under a single self-supervised encoder.\n2.\tThe design combining cluster-based supervision, multi-granularity temporal segmentation, and shared semantic centroids is well-motivated and technically sound.\n3.\tThe U-RoPE module shares the goal of stabilizing temporal position encoding with VideoRoPE [1], though it adopts a distinct mechanism by re-allocating frequency bands rather than re-designing RoPE structure. These works are complementary and should be discussed together.\n4.\tSlot-VLM [2], which separates object and event slots via slot attention, is another concurrent work that targets structured semantics and should be acknowledged for conceptual overlap.\n5.\tThe experimental setup is solid and covers a wide range of datasets. However, several critical baselines are missing:\n•\tStrong video-only models such as InternVideo [3];\n•\tRecent image-video unified pretraining methods including OmniMAE [4], OmniVL [5], and Omnivore [6].\nTheir absence limits the strength of the empirical comparison and weakens the claim of achieving state-of-the-art in unified vision modeling.\n6.\tWhile the paper is generally clear, some key implementation details are implicit. Notably, the way image inputs are handled in the VS² module (presumably as 1-frame segments) should be explicitly stated.\n\n[1] Wei, Xilin, et al. \"VideoRoPE: What Makes for Good Video Rotary Position Embedding?.\" arXiv preprint arXiv:2502.05173 (2025).\n[2] Xu, Jiaqi, et al. \"Slot-VLM: Object-Event Slots for Video-Language Modeling.\" Advances in Neural Information Processing Systems 37 (2024): 632-659.\n[3] Wang, Yi, et al. \"Internvideo: General video foundation models via generative and discriminative learning.\" arXiv preprint arXiv:2212.03191 (2022).\n[4] Girdhar, Rohit, et al. \"Omnimae: Single model masked pretraining on images and videos.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n[5] Wang, Junke, et al. \"Omnivl: One foundation model for image-language and video-language tasks.\" Advances in neural information processing systems 35 (2022): 5696-5710.\n[6] Girdhar, Rohit, et al. \"Omnivore: A single model for many visual modalities.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "was_revised": true, "final_justification": "Most of my earlier concerns have been addressed in the rebuttal by the authors:\n\nThe explanation of U-RoPE's frequency allocation is understandable, though the choice of the 3/4 split remains somewhat empirical. A short note in the final version would help clarify this.\nThe comparison with InternVideo is better contextualized now. I appreciate the effort to align training protocols, and the clarification that InternVideo’s reported performance includes supervised finetuning while UniViT is evaluated under a frozen setting.\nThe discrepancy in baseline numbers (e.g., V-JEPA) was explained by the use of a uniform 50-shot attentive probing protocol. This should be clearly stated in the main paper to avoid confusion.\n\nThat said, I am keeping my original score. While the frozen probing results are strong, the full finetuning results still fall short of some recent video-specific models. I encourage the authors to include more discussion in the final version on why this might be the case, whether it reflects a limitation of the current design, optimization difficulties, or trade-offs made for generality.", "rebuttal": ">**W1: U-RoPE and VideoRoPE share a common goal but differ in approach, and should be discussed as complementary methods.**\n\n**A1:** Thank you for your suggestion. Both U-RoPE and VideoRoPE target temporal positional encoding stability, though our design is tailored for unified image-video representation learning.\n\nVideoRoPE is a representative and impactful work that redesigns the RoPE mechanism to enhance modeling capacity for very long sequences, especially in large multimodal models. In contrast, our approach focuses on unified modeling for vision foundation models, and introduces a simple yet effective solution by re-allocating frequency bands to decouple spatial and temporal encodings. This design improves both temporal stability and fine-grained semantic sensitivity without altering the core RoPE structure.\n\nWe believe these two approaches are complementary, each suited to different use cases and architectures. We will include a more comprehensive discussion of their similarities, differences, and applicability in the revised version of the paper.\n\n>**W2: Slot-VLM also targets structured semantics via slot separation, and its conceptual overlap should be acknowledged.**\n\n**A2:** Thank you for pointing out the omission of Slot-VLM, a highly insightful and relevant work. Slot-VLM leverages Slot Attention to explicitly separate object-centric (Slow Slots) and event-centric (Fast Slots) representations. It represents a significant advancement in structured semantic modeling, offering explicit semantic disentanglement, dynamic reasoning, and strong interpretability, an increasingly important direction in video understanding.\n\nIn contrast, our method employs offline global multi-granularity clustering to model object-level and event-level semantic structures. Our approach is more focused on learning a shared semantic structure across modalities through dense pseudo-labeling. Nonetheless, the instance-level semantic separation in Slot-VLM offers a fresh and inspiring perspective.\n\nWe are particularly interested in the possibility of integrating dynamic slot assignment mechanisms with our offline clustering framework, which may further improve the model’s generalization and structural awareness. This is a direction we have not explored yet, and we plan to investigate it in future work.\n\nWe will include a discussion of Slot-VLM and cite it appropriately in the revised version to acknowledge its contributions.\n\n>**W3: Key baselines like InternVideo and OmniMAE are missing, limiting the strength of empirical comparisons for unified vision modeling claims.**\n\n**A3:** Thank you for your valuable suggestions. We acknowledge that several important baseline models were missing in the original submission. To address this, we have added additional experiments under the full attentive probing protocol, including comparisons with OmniMAE as a strong unified image-video pretraining baseline. The results are shown below:\n|Method|SSV2|ImageNet|\n|-|-|-|\n|OmniMAE|65.4|76.3|\n|UniViT-L|70.3|84.8|\n\nWe will continue to improve the completeness of our experimental comparisons in future revisions.\n\nRegarding InternVideo, we note that it involves a more sophisticated post-training stage and uses distillation from VideoMAE, making direct comparison with our method, which is trained fully from scratch, somewhat unfair at this point. Nonetheless, we are committed to transparent benchmarking and have trained a new version of UniViT under similar pretraining conditions (including SSV2), but without any post-training. The attentive probing results are as follows:\n|Method|SSV2|K400|K600|K700|\n|-|-|-|-|-|\n|InternVideo-L|41.1|80.2|80.1|70.7|\n|UniViT-L|47.1|83.9|84.3|72.8|\n\nThese results demonstrate UniViT’s strong performance across diverse benchmarks, even without relying on extra post-training stages. We will include these expanded comparisons and clarify training protocols in the revised paper to enhance fairness and reproducibility.\n\n>**W4: While the paper is generally clear, some key implementation details are implicit.**\n\n**A4:** Thank you for pointing out the ambiguity in our explanation. Your understanding is absolutely correct, image inputs are handled in the same way as 1-frame segments within the VS² module. We apologize for the lack of clarity in our original description and will revise the terminology and implementation details in the updated version to eliminate any confusion.\n\n>**Q1: Could you explain why you chose the 3/4 split for frequency allocation in U-RoPE, and how the use of U-RoPE compares to VideoRoPE in terms of performance or suitability across different tasks**\n\n**A1:** Thanks for the interesting question. We chose the 3/4 frequency allocation split in U-RoPE based on the following considerations:\n\n1. In our training and downstream evaluation settings, video sequences are relatively short. Therefore, fewer dimensions are needed for modeling temporal position information.\n2. In contrast, spatial semantics (e.g., textures, object boundaries) tend to be more complex. Allocating more frequency components to the spatial dimension helps preserve high-frequency spatial details, enhancing the model’s spatial resolution and local perceptual sensitivity.\n\nUnlike VideoRoPE, which redesigns the RoPE structure itself, U-RoPE preserves the original RoPE mechanism and achieves spatiotemporal decoupling via frequency reallocation. This makes our approach simpler, more efficient, and easier to generalize across model architectures.\nAs shown in Fig. 3 (c) and Table 3 (b), U-RoPE demonstrates modest yet consistent improvements over M-RoPE. We believe this design provides better task adaptability with minimal implementation overhead.\n\n>**Q2: How are image inputs integrated into the VS² module? Are they treated as single-frame segments consistent with the video processing pipeline? This should be made explicit.**\n\n**A2:** Yes, thank you for raising this point. We apologize for not making this clear in the paper. Your understanding is absolutely correct, image inputs are treated exactly the same as single-frame segments within the VS² module. We will explicitly clarify this implementation detail in the revised version to avoid any confusion.\n\n>**Q3: Why is InternVideo not included in Tables 1 and 2, despite being a strong video-only model with image transfer performance?**\n\n**A3:** Thank you for your thoughtful review. We acknowledge that InternVideo is a strong video-centric baseline with impressive image transferability. In our main tables (Tables 1 and 2), we opted not to include models that benefit from additional supervised post-training (e.g., on K700 or SSV2), in order to maintain a consistent and fair evaluation protocol focused on zero-shot and pretraining-only settings. This choice was intended to better highlight the generalization capabilities of our model under comparable conditions.\n\nWe recognize the value of including strong video baselines for comprehensive evaluation. Accordingly, we provide a variant of UniViT trained under comparable pretraining conditions to InternVideo (with SSV2 included), yet without additional post-training. The results are reported in our response to W3 to facilitate a fairer comparison under aligned training protocols.\n\n>**Q4: Will you consider adding OmniMAE, OmniVL, and Omnivore as baselines or at least explicitly positioning UniViT relative to them?**\n\n**A4:** We have addressed this question in our earlier response above. Thank you again for the suggestion.\n\n>**Q5: Have you explored how sensitive your clustering strategy is to hyperparameters (e.g., number of clusters or k-means initialization)?**\n\n**A5:** We appreciate your insightful comment. You are right that the earlier version of our paper lacked a detailed analysis of clustering sensitivity. To address this, we conducted an ablation study focused on the number of cluster classes. The results are shown below:\n\n|Num. of Clusters|K400|SSV2|\n|-|-|-|\n|500k|65.2|15.5|\n|1M|65.6|15.6|\n|1.5M|65.6|15.4|\n|2M|65.8|15.5|\n|3M|65.4|15.5|\n|4M|65.4|15.4|\n|5M|65.2|15.4|\n\nWe observed that performance peaks around 1 million clusters, and then gradually declines as the number increases. We believe the degradation is due to several factors:\n\n1. As the number of clusters increases, each cluster contains fewer samples, weakening semantic cohesion and reducing the effectiveness of pseudo-labels.\n2. Excessively fine-grained clustering can lead the model to overfit local patterns, especially for short or semantically redundant video segments.\n3. A larger number of clusters increases the chance of noisy or unstable pseudo-labels, which may hurt contrastive training and generalization.\n\nBased on this analysis, we chose 1M clusters as the default configuration, striking a good balance between performance and computational cost. In future work, we plan to explore dynamic cluster allocation strategies to further improve clustering quality and model robustness.\n\n>**Q6: Training efficiency and overhead of VS² and clustering vs. contrastive/masked modeling**\n\n**A6:** Thank you for bringing up this comprehensive question. Compared to contrastive models such as PE-CLIP, our method is more computationally efficient, as we do not require a text encoder, which significantly reduces the training overhead while still achieving better performance.\n\nWhen compared to masked modeling approaches, although our framework introduces slightly higher computational overhead due to the clustering and VS² module, we observe a substantial performance improvement. For example, VideoMAE-g reports an average performance of 35.3%, whereas our method achieves 73.1%, demonstrating a significant gain in representation quality.\n\nWe believe this trade-off is favorable and will provide additional implementation details in the version to clarify the cost-performance balance."}]

Author rebuttal:

Meta-review: ### Summary

This paper introduces UniViT, a unified vision encoder for both images and videos. The authors claim that by performing object-level clustering for images and event-level clustering for videos, their model can effectively learn both image and video representation within a single architecture. The paper also introduces two novel components: Unified Rotary Position Embedding (U-RoPE) to better handle spatiotemporal positional information and Variable Spatiotemporal Streams (VSS) to accommodate video inputs of varying lengths. The authors present extensive experimental results showing that UniViT achieves state-of-the-art performance on a variety of tasks.

### Strengths
- The paper tackles the important and challenging problem of creating a single, unified model for both image and video representations. 
- Novel and solid method. The proposed method, which combines clustering-based self-supervision on both images and videos with contrastive learning, is technically solid. The introduction of new positional embedding U-RoPE and VSS for are necessary additions to address specific challenges in unifying these two modalities. 
- The authors have conducted extensive experiments across a wide range of benchmarks for images, videos, and multimodal tasks. The proposed method demonstrates state-of-the-art or highly competitive performance across most of the evaluated tasks.

### Weaknesses
- Novelty Concerns: mwRn has concerns about the technical novelty, suggesting it is a straightforward extension of a previous method, MLCD. 
[1] Multi-label Cluster Discrimination for Visual Representation Learning.

- Clarity and Presentation: Reviewer mwRn also pointed out some clarity issues, with terms like "object" and "event" not being precisely defined. 
- Reliance on Offline Clustering: The method's dependence on an offline clustering step using a pretrained model was noted as a potential weakness by reviewer FTq5, as it could introduce biases from the initial model.
- Initial Discrepancies in Baseline Results: Reviewer Sxna raised valid concerns about discrepancies in the reported performance of some baseline models.

### Reasons for Recommendation
I recommend this paper for acceptance. The problem of unified image and video understanding is of high importance to the community, and this paper presents a solid and effective solution. The extensive and strong empirical results across a diverse set of tasks are a significant contribution. The authors have also been very responsive and thorough in their rebuttals, addressing most of the reviewers' concerns with additional experiments and clarifications. 
Despite the concerns regarding novelty, the strengths of the paper outweigh the weaknesses. While the core idea builds upon existing work, the specific adaptation for the video domain and the proposed architectural modifications (U-RoPE and VSS) provide sufficient novelty. Majority of the reviewers are also biased towards an Accept. 

### Summary of Rebuttal and Discussion
- Reviewer mwRn's main concerns were about the novelty of the work in relation to MLCD and the clarity of the writing. The authors provided a detailed breakdown of the differences between UniViT and MLCD in terms of input, architecture, and methodology. They also clarified the definitions of "object-level" and "event-level" clustering and provided more details on their experimental setup. Despite the extensive discussion, this reviewer remained concerned about the overall novelty, though they did raise their score , acknowledging the authors' efforts.
- Reviewer Sxna questioned the discrepancies in baseline performance and the temporal modeling capabilities on datasets like Something-Something V2. The authors clarified that the discrepancies were due to different evaluation protocols (e.g., 50-shot probing vs. full fine-tuning) and provided additional experimental results, including full fine-tuning comparisons, to address these concerns. These clarifications were convincing, and the reviewer raised their score to "Borderline accept."
- Reviewer FTq5 pointed out the reliance on offline clustering and initial concerns about reproducibility. The authors addressed the offline clustering by explaining the trade-offs and committed to releasing their code and models, which satisfied the reviewer.
- Reviewers LTUC and Wzoa were generally positive from the start. Reviewer LTUC's concerns about the verification of learned features and the data ratio were addressed in the rebuttal. Reviewer Wzoa had questions about missing baselines and implementation details, which the authors also addressed by adding more experimental comparisons and clarifying their methodology.

This paper received mixed ratings. But AC believes the authors provide a strong rebuttal. The fact that they were able to convincingly address the concerns of most reviewers, leading to score increases, speaks to the solidity of their work. While the novelty debate with reviewer mwRn is a valid point of discussion,  AC and other reviewers think that this work has sufficient novelty compared to MLCD. The overall contribution and the quality of the empirical results are strong enough to warrant acceptance.
Decision: Accept (poster)