Perception Encoder: The best visual embeddings are not at the output of the network
Authors: Daniel Bolya
Po-Yao Huang
Peize Sun
Jang Hyun Cho
Andrea Madotto
Chen Wei
Tengyu Ma
Jiale Zhi
Jathushan Rajasegaran
Hanoona Abdul Rasheed
Junke Wang
Marco Monteiro
Hu Xu
Shiyu Dong
Nikhila Ravi
Shang-Wen Li
Piotr Dollar
Christoph Feichtenhofer
Venue: NeurIPS 2025 oral ()
Keywords: vision encoder
vision-language
CLIP
pretraining
features
foundation model
Primary area: deep_learning
TL;DR: We develop a CLIP model that is SotA on both image and video zero-shot recognition. Using its strong, general features we further create SotA encoders for language and spatial tasks.

Abstract: We introduce Perception Encoder (PE), a family of state-of-the-art vision encoders for image and video understanding. Traditionally, vision encoders have relied on a variety of pretraining objectives, each excelling at different downstream tasks. Surprisingly, after scaling a carefully tuned image pretraining recipe and refining with a robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods: language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together, our PE family of models achieves state-of-the-art results on a wide variety of tasks, including zero-shot image and video classification and retrieval; document, image, and video Q&A; and spatial tasks such as detection, tracking, and depth estimation. We release our models, code, and novel dataset of synthetically and human-annotated videos: https://github.com/facebookresearch/perception_models

Perception Encoder: The best visual embeddings are not at the output of the network
Daniel Bolyal l
Yao Huang Peize Sun Jang Hyun Cho 125t Andrea
Madotto !
Chen Wei Tengyu Ma'
Jiale Zhi'
Jathushan Rajasegaran' Hanoona
R 3f
Junke Wanght Marco Monteiro' Hu Xu' Shiyu Dong' Nikhila Ravi' Daniel Lif Piotr Dollár Christoph Feichtenhofer
Meta UT Austin MBZUAI 4Fudan University
"Joint first author Work done during internships at Meta
Abstract
We introduce Perception Encoder (PE), family of state-of-the of-tth--rtt vision encoders for image and video understanding. Traditionally, vision encoders have relied on a variety of pretraining objectives, each excelling at different downstream tasks. Surprisingly, after scaling a carefully tuned image pretraining recipe and refining with robust video data engine, we find that contrastive vision-language train- ing alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the inter- mediate layers of the network. To draw them out, introduce two alignment methods: language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together, state-of-tth--at results on a wide variety of tasks, including zero-shot image and video classification and retrieval; document, image, and video Q&A; and spatial tasks such as detection, tracking, and depth estimation. We release our models, code, and novel dataset of synthetically and numan-annotater videos: https://githuu: wtt.ub.coo/aacbbookeeeeeaae httproorrreeeerceeeepppppiiooooo wwtppppeicnonpeeeeeeeee models
PE family of models achieves
1 Introduction
For the last decade in computer vision, pretrained vision encoders have been the core building block for most applications requiring perception. From million-scale ImageNet [25] pretrained convolutional networks [41, 60, 79, 121, 128] to billion-scale web-pretrained transformers [19, 24, 28, 53, 127, 156], the dominant strategy vision has been to adapt large-scale pretrained encoders to downstream tasks.
Today, these pretraining objectives come in several flavors: vision-language contrastive losses [103, 158] learn global vision and language embedding well-suited for zero-shot classification ciasfsioot and retrieval as well as provide aision-language alignment for open- open-world [68, 92] and generative tasks [105, 111]; captioning losses [36, 134] learn to predict image descriptions using language decoder, which transfers well to downstream multimodal language model (MLLM) tasks; and spatially self- self-supervised losses [43, 96] learn dense spatial correspondences without language supervision, making them useful for tasks requiring precise localization like object detection. Many works are now attempting to combine two or more of these techniques in different ways [19, 33, 34, 36, 44, 88, 107, 156]. While many have been successful, the complexity of these strategies grows exponentially with number of use cases, which can make scaling difficult. There has not yet been shown a single, simple, and easily scalable pretraining technique that can learn otate-of-the-art features for all downstream tasks.
In this work, we discover that global vision-language contrastive learning alone can be one such approach. We begin by building PE (Fig. 1, left), a large-scale contrastively pretrained model with ofte-oof-hhearr zero-shot performance on both image and video ($2). To accomplish this, we first focus on developing a strong image-only contrastive pretraining recipe to extract general knowledge from billion-scall image-text data ($2.1). We then use the resulting model as a frame-based encoder to develop video data engine ($2.2) for generating well-alin well-aligned video captions. Finetuning on this synthetic video-text data substantially improves performance on both image and video classification
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

State Patte-o--he--ee
Large Scale Contrastivo Phetriinnig
Classty
Rastire
PE Language
$4
Captioning
Video
Grounding
T
T
T
Image
$2
S3
PE Core
State-of- Art
Videos
Vicecs
Alggemant Tonnng
PE Spatial
65
Detect
Segment
Trecc
Citirate
Rix
Musk
Modkke
Dipirs Hap
Figure 1: Perception Encoder (PE) is a family of large-scale vision encoder models with state- of-the-art performance large variety of vision tasks. By using robust contrastive pretraining recipe and finetuning on synthetically aligned vidcos, PE not only outperforms all existing models on classification and retrieval ($2), but it also internally produces strong, general features that scale for downstream tasks ($3). PE unlocks the ability for large-scale contrastive pretraining to transfer to downstream tasks with alignment tuning capitalize those general features ($4, $5).
and retrieval tasks. Finally, we scale our robust image pretraining and well-aligned video finetuning strategy to 2B parameters produce G ($2.3), single unified encoder that outperforms SigLIP2 [135] on zero-shot image tasks and Intern Video2 [143] on most zero-shot video tasks.
After analyzing the performance of G, we found surprising result: inside the model were specific features aligned to OCR, VQA, grounding. detection, depth estimation, and tracking ($3). Compared the -iite--ffthharrr models with captioning [36] and spatially self-supervised [96] pretraining, our contrastive encoder has specific layers that, when used as frozen features, matches or exceeds the performance of the other two pretraining techniques on tasks they should be the best at. The only problem ihet features exist at different layers for each task.
By exploiting this phenomenon with alignment tuning (Fig. I, right), show it is possible to align these features to the end of the network to create state-of-the-art encoders for downstream MLLM ($4) and spatial ($5) tasks-all following the same easily scalable contrastive pretraining. Thus, Perception Encoder unlocks the potential to scale one simple pretraining method to solve many downstream vision tasks. We will release our models, code, and novel PE Video Dataset of IM high-quality stock footage videos and 120K numan-refined captions.
2 Perception Encoder: Core
To build Perception Encoder (PH), we start by training large-scale, robust, and performant vision- language contrastive model for image and video. We have two objectives: to enhance the scalability and data efficiency of contrastive training, and to create unified model for image and video.
We decouple image and video training into two stages. We first develop a strong image pretraining recipe ($2.1) 1) with several regularization techniques to create a robust starting point. Then we use the resulting image model as a frame encoder to develop a video data engine ($2.2) supported by our novel human- refined video-text dataset to generate aligned captions for video clips. Finally, we train the image encoder on the resulting aligned video data ($2.3). Using our data engine design, this short training step substantially improves both image and video performance.
2.1 Robust Image Pretraining
In the first stage of pretraining, we want learn as much visual information possible from large set of image-text data with high regularization, stability, and training efficiency in mind.
Setup. We track our changes with OpenCLIP [50] VIT-L/14 at 224 resolution as a baseline (Fig. 2.1). We fix training budget of around IT GFLOPs (i.e., ZFLOP), and ablate on fixed 2.3B noisy image-text dataset curated using the MetaCLIP [150] text-only curation pipeline, and start by training for 12B samples seen. To assess generality, results as well as an average of 6 common robustness metrics: ImageNet val [25], ImageNet v2 [109], ObjectNet [4], ImageNet Adversarial [46], ImageNet Rendition [45], and ImageNet Sketch [140].
report ImageNet val [25] zero-shot classification
Training. Motivated by [69, 70, 77, 128, 133], we begin by improving training efficiency with progressive resolution (Fig. 2.2). By evenly splitting the baseline 12B sample run into 98, 154, and 224 resolution stages (4B per stage), we half training FLOPs while maintaining performance. We then use the extra budget to double global batch size (Fig. 2.3) from 32K to 64K, increas- ing total samples from 12B to 24B. This makes hard negatives more probable, increasing the

---

"task difficulty" of CLIP. Finally, we switch
Robustness of ImageNet val
Training zhuor
from AdamW W to LAMB [154] (Fig. 2.4),
1. Baseline
75.3
78.9
1.0
which allows us to stably increase learn-
ing rate from x 10-4 to 2 x 10-3 and bet- ter fit the CLIP objective. Overall, these
2. Prog. Res
75.1
78.9
0.5
3. Batch Sz
76.2
79.5
11
changes improve +1.0% on ImageNet val
4. LAMB
76.9
79.9
14
and similar +1.6% on robustness.
5. High Res
78.3
80.4
1.2
6. RoPE
79.2
80.7
Modeling. To assist with scalability [35. 128], we add higher resolution (Fig. 2.5) stage at 336 pixels. To keep FLOPs the same, we adjust the schedule to 10B sam- ples at 98 resolution, 8B at 154, 4B at 224, and 2B at 336. To improve extrapolation, we also add 2D RoPE [124] (Fig. 2.6) to
7. Attn Pool
80.1
81.0
Data Aug
80.8
81.1
1.2
Mask Reg
80.9
81.3
1.2
Figure 2: Robust Image Pretraining. We tune our pretraining recipe ($2.1) to maximize performance on fixed set of data, starting with an OpenCI.IP [50] ViT-I/14 model. We report cumulative zero-shot clas- sification results for each modification. The inner bars show robustness evaluation, calculated as the average of 6 robustness benchmarks [4, 25, 45, 46, 109, 140], and the outer bars show ImageNet val [25] alone. Several changes significantly improve robustness, indicating that ImageNet val scales more with data, while robust- ness can scale with refined training techniques.
each attention layer, keeping the original po- sition embedding. Finally, we follow [158]
in constructing the CLIP embedding us- ing an attention pooling transformer block (Fig. 2.7). Surprisingly, we found keeping the class token as an input to this block is important for small model performance. These changes improve ImageNet val by +1.1% but robustness threefold, by +3.2%.
Regularization. Despite training on billions of samples, we find data augmentation (Fig. 2.8) still important. Adding heavy random cropping, brightness/saturationn jitter, and horizontal flip generally improves robustness without adverse downstream effects (e.g.. for OCR). Finally. we add mask regularization (Fig. 2.9) by duplicating and masking 1/16th of the input batch. At the output, the masked tokens are aligned to their unmasked counterparts by maximizing cosine similarity. Together, these regularization changes improved ImageNet val by +0.3% and robustness by +0.8%.
Overall, our recipe improves ImageNet val by +2.4% and robustness by a significant +5.6% while keeping FLOPs similar and maintaining or improving scaling behavior (see Appendix C.I).
2.2 Bootstrapping a Video Data Engine with Perception Encoder
Our next step is to extend the image-only encoder to video. Unlike web-scale image-text data, which comes in many cases with human- -enarntee descrip- tive alt-text information, videos with aligned language annotation inherently scarce and often low quality. Inspired by the re- cent success of image data en- gines [57, 63, 94, 108, 149], we address the lack of high quality aligned video captions by devel- oping a robust video data engine to generate them. Our approach (Fig. 3) represents the first large-scale exploration of this kind.
Vidco Captioner
Video Caption
Frame Captione
Image Captioner
Forree cxptiin
Frarre 2 caption
Algeed
LLM
T
Prrre a caption
Metadata
ippioniioon
Figure 3: Video Data Engine. We use a PE-based video captioner for video-level captions and an existing image captioner [80] on sampled frames. We use these components along with the video metadata to synthesize short captions with text-only LLM [80].
Video Data Engine. We build our data engine in parts: (1) we construct a video captioning model using an early image-only version of PE as frame-level encoder and Llama [80] as the language decoder. We train with the PLM [21] MLLM training recipe and data mix. In total, the mix consists of 64.7M images and videos covering natural images, charts, documents, exocentric and egocentric videos. (2) to further boost captioning performance, we collect a set of 265K videos (part of which we release as PE Video Dataset, see Appendix A.1), caption them with our base video captioner, and ask human raters to refine the captions. We then finetune our video captioner with this human refined data, significantly improving captioning quality (see Appendix C.2). (3) finally, we synthesize
3

---

the aligned video captions by incorporating captions from our video captioner, Llama 3.2 [80] as per-frame image captioner, and the existing title and description metadata (Fig. 3) summarized with a Llama 3.3 70B text model (see Appendix A.2.4 for prompts).
Video Training. We use the resulting data engine to generate information- captions for a diverse set of 22M videos, with which we finetune the image-only PE model. To do so, we use PE as an frame-level encoder: for each video, we uniformly sample 8 frames, extract the CLIP embedding for each, and average pool to obtain a single video embedding for text embedding alignment. Despite its simplicity, we find this technique produces a strong joint image- video encoder.
Ablations. In Tab. 1, we ablate the im- pact of each component of the video data engine by finetuning an interme- diate image-only Po checkpoint on the recaptioned videos. Compared to the image-only baseline encoder (first row), our video data engine sig- nificantly enhances zero-shot classifi- cation and retrieval performance for both image (72.6-78.2) and video Notably, using video- level and frame-level captions pro- vides significant improvements over relying solely on metadata such as video title and description (second row), highlighting the importance of building robust video data engine to compensate for noise in web videos.
Imsaee Zam-SSou
Video Zaru-Shoot
Video Cyttinn
Frome Caption
Asenge Image
Ascigge Valao
Desciption
ImogeNNe
(125)
(11000
IN Closoot 11)
MS- JS-COCC
seap PH
MS Cls-ccco
Kietliss
Kincles
600154)
MSSR-VTT
VTT
10 and 15511
MSR VT
Thil
77.8
49.
66.8
50.9
69.7
38.0
75.4
83.2
78.2
87.1
47.3
66.0
56.0
74.1
73.5
39.0
57.3
782
83.5
78.4
86.8
56.0
74.3
60.9
73.8
73.4
47.6
48.
78.1
83.7
79.0
87.7
54.1
73.0
60.9
75.4
75.1
46.7
46.5
782
83.7
79.0
87.5
54.6
61.6
75.8
75.5
47.
Table I: Video Data Engine Ablation. We ablate our video data engine in Fig. 3 by finetuning on in-development image-only version of PE by averaging the frame embeddings to create single video CLIP embedding. Video captions are generated by our captioner trained with or without (V") human-refined data. Frame captions are generated by the Llama 3.2 vision model [80]. Taken together, the result is a huge boost to both image and video zero-shot performance. See Appendix C.2 for more ablations and scaling behavior.
2.3 A Unified Encoder for Image and Video
Using a robust, scalable image pretraining recipe and video-pretraining data recaptioned by the proposed video data engine, in this section we present PE a unified image-and-video encoder.
Model Architecture. To capitalize on the promis- ing scaling behavior observed in $2.1, we scale the largest PE model to 2B parameters (G scale). Tab. shows the detailed model configuration of the vision and text transformers and the dimension of the output clip embedding space.
Scale
Tower
Porams
Width
Leepth
MLP
Heads
CLIP Dim
Vision
0.098
768
3072
1024
Text
0.318
1024
24
4096
16
1.
Viston
0,326
1024
24
4095
18
1024
Text
0,318
1034
24
4096
16
Visien
1,888
1536
50
8960
16
1280
Test
0.47B
5120
Table 2: PE Model Configurations.
Model Training. We train PE in three stages:
1. Image pretraining. We scale up image pretraining data to 5.4B publicly available image alt-text pairs curated with MetaCLIP [150] and a total of 86B samples seen to ensure convergence (58B for B and L). We use a global batch size of 131K, with progressive resolution from 98 to up to 448 depending on the model.
2. Image and video finetuming. Following the initial pretraining, we subsequently finetune the model at max resolution with a short schedule for 50M samples on the image pretraining data (as cooldown) followed by 22M samples on the recaptioned videos with a smaller learning rate and batch size. The video captions are produced using the proposed video data engine ($2.2). For each video clip, we uniformly sample 8 frames, encode them, take their average to produce single video embedding, and align them with the corresponding vidco captions using the same contrastive objective in image training.
3. Smaller model distillation. We distill the 2B model (G scale) into smaller contrastive pretrained models at B and I. scales under their final resolutions, using a short, finetuning schedule that covers approximately 4B samples seen (~8% of the pretraining schedule) with a lower learning rate. We still perform stages and 2 for small models (see Appendix C.3).
Detailed training configurations and setups are listed in Appendix B.

---

Enooder Parans
Zero Shor Classificttion
Zero Shot Fine Grained Classifiurtio
Zero Shot Retrieval
Roselntten
Arg Class
ImageNee
(2))))
ImageNet
ObyestNet
ImageNe
ImogeNeet
Rondsiio (43)
ImagaNeet
Sani (1
Arg Fine.
Plowers
Ogfiral
Airerafts
Counttes
(130
Sconns
SUNESI (480)
Saldiie
RESIIC (20)
Are Retrienal
MS. COCO
wwe [Pg]
Say 3od (P4)
Flidke-OKK
Flide-300k
m1o (1:5))
Dots
Model
SigLIP-B/IG
[158]
0.11
224
108
09.9
76.2
09.5
70.7
45.1
90.2
67.9
61.8
85.2
90.8
44.0
15.9
70.0
64.5
09.8
47.2
64.5
77.9
89.6
SigLIP2-B/16t [135]
0.1B
224
100
73.1
78.2
71.4
73.6
55.0
91.7
68.9
66.2
85.7
93.4
54.8
19.2
72.7
71.1
73.7
52.1
65.9
80.7
93.0
PF
224
62.4
68.8
30.5
74,0
73.7
74.3
94.4
L/ [158]
0.38
384
80.7
82.1
75.9
80,9
76.5
95.0
13.6
67.1
94.8
53.2
24.7
72.5
67.9
74.7
52.8
82.6
92.9
SigLIP2-L/161 [135]
0.3B
384
100
83.3
83.1
77.4
$4.4
84.3
95.7
75,5
72.5
90.0
95.8
67.0
31.6
74.8
75.5
76.7
55.3
71.4
85.0
95.2
PFoee!.
0.38
336
5.48
83.9
77.9
84.7
89.0
95.2
73.4
74.6
67.88
45.6
77.4
75.7
7888
57.1
75.9
96.6
DEN-H [39]
0.68
378
81.6
84.3
78.3
79.6
79.6
93.6
73.3
75.2
91.5
96.0
72.5
37.9
77.4
75.9
75.8
55.6
71.8
82.1
93.6
IntemVL-C [19]
5.50
224
5B
82.5
83.2
77.3
80.6
83.8
95.7
74.3
69.9
85.8
94.4
53.3
35.1
76.3
74.4
78.6
58.6
74.9
85,0
95.7
EVA 18R [18]
17.58
224
2B
83.6
83.8
77.9
82.2
87.3
95.7
74.7
73.1
86.0
94.9
59.7
41.1
77.7
76.9
77.5
56.2
73.6
83.1
96.7
SigLiP [135]
1.1B
384
86.2
85.0
79.8
88.0
90.5
96.6
77.4
75.6
91.5
95.9
73.6
40.1
76.3
75.9
78.0
56.1
72.8
86.0
95.4
PR rimage only)
448
86.0
78.2
74.9
G
1.98
448
5.48
86.6
85.4
50.2
88.2
92.6
96.5
76.5
79.4
91.4
94.7
78,2
57.6
78.5
75.8
78.9
58.1
75.4
85.7
96.2
Table 3: Zero-Shot Image Results. Image zero-shot performance of Po compared to the state-of- tre-art. Across all model sizes, PEçore obtains state-of-the-ar results across general classification, retrieval, and finegrained classification. 'Re-evaluated: DFN by [127]; SigLiP and SigLIP2 by us with the same benchmark settings if not reported in [135] (see Appendix B.1.2).
Eocoales Parans
Zaro-Stot Classificttoo
Zero Shot Retrieval
Arg Retrieaa
(151
wile 10 (3)
1o000 (10)
Resolniin
Franes
Video
Kinekss
400 S4
Kineses
Kinetes
700 [54)
MSS. VTT
MSR VTT
VD
MS VD
Activiy
ActiviyyNa
Model
Aug
UCF
CLIP4CLIP [82]
0.18
224
12
ada
32.0
38.5
Sigl 1P2-//66 [135)
0.18
224
58.7
$5.0
48.4
82.0
42.3
79.9
38.5
30.1
49.0
67.2
28.6
25.8
PR
0.18
65.6
65.1
55.8
48.2
76.7
38.4
UMT-L 166]
0.38
224
37.1
74.5
39.4
Sigl IP2- [135]
0.38
384
64.1
65.3
62.5
56.8
86.7
49.3
44.7
41.5
31.4
53.7
74.2
35.9
51.5
PR
0.38
336
22M
71.4
73.4
72.7
65.3
N7.1
58.5
54.8
50.3
50.1
57.2
H2.4
46.4
42.1
L1ne/VL-C(119
5.58
224
69.1
68.9
60.6
44.7
40.2
VitenVidee22 [143]
1.08
224
102M
70.7
73.1
72.8
64.9
88.8
53.9
59.9
51.9
50.9
58.1
83.3
6044
54.8
Sip---p--- [135]
384
61.8
51.8
14.2
74.6
33.4
PR (image only)
1.9
448
70.9
73.1
72.2
61.3
89.5
55.5
47.6
44.3
35.2
54.3
73,9
41.4
36.3
1.90
4:8
8
22M
74,8
76.9
76.1
69.1
90.7
61.1
58.7
51.2
49.9
59.7
85.4
54.7
51.2
Table 4: Zero-Shot Video Results. Video performance of PE compared to recent video and image encoders. PE obtains state-of-the-art in video classification and comparable performance on retrieval benchmarks while using only 22M videos. TiiLLIP2 evaluated by us (see Appendix B.1.2).
Zero-Shot Image Results. In Tab. 3, present PE 's performance zero-shot image benchmarks for classification and retrieval vs. the strongest open models, including SigLIP2 [135]. PE outperforms all other contrastive models across the board on all zero-shot tasks, including the highly competitive average of zero-shot ImageNet robustness metrics [4, 25, 45, 46, 109, 140). This the first to accomplish this in over 3 years without access to Google's 's internal JFT-3B [28] or WebLI [17] datasets. And at the same time, PE also exceeds the existing state-of-the-art
marks significant achievement, as
image-text retrieval and significantly improves on fine-grained classification-the the first to simultaneously hold state-of-the- on all common zero-shot categories.
Notably, this dominant image performance is made possible by video finetuning. Compared to image only, the video finetuned PE G obtains +0.6% general classification, +1.2% fine-grained classification, and significant +4.0% boost on retrieval. Thus, well-aligned video text data does not just improve video performance-it it creates a strictly better model for both videos and images.
Zero-Shot Video Results. We present video results in Tab. 4. Our base image encoder already outperforms all other image-only encoders on both zero-shot classification and retrieval, including SigLIP2-g-opt. With video finetuning, PE G significantly outperforms even native video models that use full temporal attention on video classification, and it nearly matches the state-of-thh ofte--of-the-ar art on video retrieval despite being a simple frame-level encoder. This result underscores the importance of our video data engine, resulting in +3.9% on average zero-shot video classification, and a massive 1.11%% on video retrieval. Moreover, PF does this with fewer videos compared to other video- based approaches like Intern Video2 [143], highlighting the benefits of a joint image-video encoder.
See Appendix C.4 for additional zero-shot and probing results.
5

---

3 General Features in a Contrastive Disguise
Piecre has strong results on zero-shot classification and retrieval, but these are tasks contrastive encoders specialize in. More important is whether or not this strong performance generalizes to downstream tasks. To find out, we compare PE G to state-of-the-an models for other pretraining techniques: captioning (AIMv2-3B [29]) and self-supervised learning (DINOv2-g [96).
Layerwise Feature Analysis. We perform frozen feature analysis of each encoder in Fig. 4 for several downstream benchmarks in 3 cate- gories: classification, language mod- eling, and spatial tasks. For classifica- tion, we probe each model using ran- domly initialized cross attention trans- former block. For language alignment, we learn a projector and finetune a decoder-only LLM (see $4), and for
Classification
Attn Probe ImageNet 1K AAccuuocyy
89.6
86.9
89.8
AIM2
DINOV2 La9e
PE Lope
Language Modeling
Most 621
Q&A
Aug of 4
Layer 2ENG
Layer 2014
Deet:
spatial tasks we train with several dif- ferent decoders (ViTDet [71] Mask- RCNN [42] with Absolute Win [7] for detection, DPT [106] for depth, and zero-shot feature correspondence for tracking [5]]). For each experiment, we sweep over the layers of the model as the optimal features are not neces- sarily the last. In each case, we use an equivalent image size (window size for detection) of 32 32 tokens. In each plot, we normalize performance by the maximum and minimum per- formance across models on that task.
Visual Q&A
Avg of 4 NArturrryy
77.5
837
75.3
70.6
Grounding RefCOCO JAl PgOs
Lave 271rt AIM2 -ayer
Laye 21nnd
Layer S200
DINOV2 Laye
Pore
Spatial Tasks
Detection
COCO Box
42.8
43.2
Layer 1/14
Liger
Layer
0.25
Depth NYU
General Features in Disguise. This analysis reveals several insights. First, as expected, AIMv2 performs well at classification and the best at visual Q&A language tasks. Similarly, DI- NOv2 performs the well on spatial tasks like detection, depth, and even grounding through an LLM. Then as already established by other works: DINOV2 performs poorly on OCR tasks [131]. But interestingly, its per- formance peaks in the middle of the network and then drops by the end. And so do the others on several tasks (AIMv2: tracking, grounding, detection: DINOv2: VQ&A, grounding). PEçore exhibits similar be- havior, but with unexpected results: it can perform well on all tasks, often matching or exceeding the leading models. Remarkably, PE has intermediate layers that perform near to or on par with AlMv2 for language tasks and DINOv2 for spatial tasks, despite being trained with a global contrastive loss. Depth estimation is particularly noteworthy, as contrastive encoders are not typically considered state-of-the-aar in that area. In fact, CLIP models are notorious for poor spatial performance [107].
Layer Mine
Laye 77uny
Layer SANi
Tracking DAVIS Zaro Shot JGF
58.5
AIM2 -xyer
24
DINOVO Leyer
40
PC PE
Figure 4: Frozen Feature Layer Analysis of different pre- training methods: self-supervised (DINOv2-g [96], middle), and our contrastive recipe (PE G. right). Vertical lines denote the best layer and horizontal lines the best performance across models. AIMv2 excels at language but not spatial, and DINOv2 ex- cels at spatial but not language. But, intermediate layers of PE G perform well on both language and spatial tasks.
captioning (AIMv2-3B [36), left), spatially
An Alignment Problem. However, PE 's strong general performance diminishes rapidly towards the end of the network, such as for LLMe- grounding. This behavior is less pronounced the closer the downstream task to the pretraining method, suggesting an alignment problem. Thus, a well-tuned large-scale contrastive model can learn general embeddings in the process of fitting its objective, but it fails to outpul them. We address this issue with alignment tuning in $4 and $5 and analyze why our CLIP model has these general features and its scaling behavior in Appendix C.5.

---

Analysis. The finding that pure CLIP models possess features which match the performance of otate-of-the-art pretraining methods their specialized domains is new. In fact, recent work [30] has shown the opposite- that CLIP models fail to scale on downstream tasks. We next investigate how our approach yields these results.
Last Layer cocO Rox Best Layer CoCo Row
A Argmaax
To start, we perform layerwise frozen fea- ture analysis on COCO detection. PEcore
1. Baseline
28.6
14,01
2. Prog. Res
32.3
34.4
16/24
was particularly "peaky" on this task in Fig. 4, with its best layer on par with DI- NOv2, but last layer significantly worse. We already ablate each change we made from vanilla CLIP in Fig. 2 using a ViT- I/14 model. So to retrace our steps, we run frozen feature analysis on those check- points. For efficiency, we use a lower reso- lution and only sample even layers for this experiment. In Fig. 5, we report COCO box mAP for the last and best layers for each cumulative ablation, along with the index of the best layer. Further, plot the layer- wise mAP for each change in Fig. 6.
Batch Sz
30.7
34.2
16,94
4. LAMB
30.0
33.2
16/24
5. High Res
32.3
35.7
16,24
6. RoPE
33.2
37.0
16:04
7. Attn Pool
31.7
37.1
18:04
Data Aug
31.9
38.0
18:0
Mask Reg
32.7
38.2
18
Figure 5: The Downstream Effects of Robust Pre- training. The ViT-L/14 checkpoints from Fig. 2 eval- uated as frozen features on COCO [74] using Mask R R-CNN [42]. We report the last layer performance, best layer performance, and the best layer's 's index.
Surprisingly, the simple changes we made to CLIP pretraining in $2.1 overall improved the best layer's 's performance by almost 10 mAP! Some improvements are expected like with high resolution (5) and RoPE (6), but unexpectedly data augmentation (8) and especially progressive resolution (2) help considerably. It is possible that contrastive pretraining overfits to a specific resolution through "global tokens" [23], thus changing the resolution during training forces the model to be more robust.
Next, both progressive resolution (2) and attention pooling (7) move the argmax layer deeper into the network (rightmost col- umn of Fig. 5). Attention pooling in particular alters the whole shape of the layerwise performance curve (Fig. 6). Finally, some changes reduced performance: increasing the batch size (3) and using LAMB with a high learning rate (4). Both help fit the CLIP loss better, which after a point may not improve the general features. Moreover, while the best layer improved sig- nificantly, the last layer performance stagnated after (2). This suggests that constructing the CLIP token requires a specialized decoder. Yet, this does not prevent the model from learning general features- just outputting them.
40
COCO BoKmAP
30
20
10
8
16
24
Mookl
Frozen Encoder Layer
Figure 6: Layer Analysis of Fig. 5,
Scaling Behavior. Evidently, our robust recipe can enable contrastive pretraining to produce general features. But, does it scale? In Fig. 7, we answer this by performing frozen feature analysis across S/14, B/14, and L/14 models trained with the same schedule with either the vanilla CLIP recipe or our recipe (see Fig. 14). Immediately, we see a stark contrast between their scaling behaviors: while the vanilla recipe quickly plateaus at L scale (300M), the best layer of our robust pretraining recipe demonstrates scaling to G scale
being trained with a decidedly non-spatially aligned global contrastive loss. Though note this is the best layer. The last layer still stagnates for both. Thus, CLIP loss obfuscates its general features even with our recipe, placing them several layers deep.
Object Detection
Best Layer Scaling
40
40
45
Vanilla
Ours
coco Sosmop
30
30
COCO map
30
20
15
33%
33%
100%
0.18
Frozen Eepth
Presen Encoder Depth
Model Parameter
Figure 7: The Downstream Scalability of Robust Pretraining, Left: frozen feature layer analysis of the S/14, B/14, and L/14 models from Fig. 14 using the same setup as Fig. Right: scaling behavior of the best layer for each model. Note: G has different schedule. See Appendix C.5 for more.
7

---

4 Perception Encoder: Language Alignment
In $3 we have seen that PE already possesses useful features for Multimodal Large Language Models (MLLMs), but those features are not aligned to the end of the network. In this section, we lift these features through alignment tuning to construct a new, ML.LM-speciall
encoder: PEtang
Alignment Method. Aligning vision encoder to an LLM is relatively straightforward. We follow the approaches of [18, 21, 36], where the vision encoder is unfrozen and finetuned as part of an MLLM. In our case, we align PE to a pretrained Llama3.2 3B text-only decoder with both the encoder and decoder unfrozen, connected with a 2-layer MLP. We discard the last layers of P as suggested by [18] and regularize the encoder with LayerScale [132] and DropPath [49]. We train with next token prediction on 70M total samples across OCR Q&A, Captioning. Visual Q&A, and Video Q&A A (following [, and finally extract the vision encoder only as PEiang More training details are available in Appendix B.2 and ablations of this recipe are conducted in Appendix D.
Effects. In Fig. 8, we conduct the same layerwise analysis in $3 on the resulting PElang compared to P. Across all categories, the best layer for the aligned model is the last, no matter the performance of the original checkpoint. Notably, our PEaang train- ing mix did not contain grounding data, which means that this significantly lifted grounding performance is entirely due to the strong intermediate grounding fea- tures in PE now being aligned to the end of the net-
OCR Q&A
Visual Q&A
80
Actunny (kg-s)1))
58
Accuni 2000
67
58
25
40
Pe
17
50
17
33
50
Frozen Encoder
Frozon Encoder Layar
Captioning
Grounding
130
Ro 1g' F.3.5
75
work. Moreover, specific domains such as OCR Q&A
100
57
that were represented in the training mix see signifi- cant boost to performance compared to even the best layer of Pigeee which was already strong. Thus, with an order of magnitude fewer samples compared to pretraining, we were able to language align PE G to create a single, strong encoder for all MLLM tasks.
70
38
PEnw
17
50
17
33
50
Presen Encoder
Frezen Encoder
Figure 8: Language Alignment lifts the strong performance of PE ($3) to the end.
Results. In Tab. 5, compare Pa to existing encoders with good language alignment. To benchmark, we plug each model into a fresh 2-layer MLP and Llama 3.1 8B decoder. The encoder frozen the rest are are finetuned on 2.6M visual Q&A pairs (see Appendix B.2). We evaluate each encoder at native resolution unless otherwise noted. Despite using a different LLM than during alignment, Pa significantly outperforms all other models across all scales, resolutions, and tasks. Results with tiling and different LLM decoders are available in Appendix D.4. In all cases, PEgang exhibits generality. That is, outperforms other models no matter the resolution, decoder, or task.
OCR /hon/ Doc. Q&A
Vieual Q&A
Captioning
Video
Ang: OCR DA
Chanto
Azn. (66))
Doc VQ
Info Q4
A12
Ang. V24
TarVQA
Apc. (122)
VQA
POPE
Aoc (72)
VQAv2
Ang: Cip.
Fieter
COCO
No Cup
cmno [I]
Aug. Gooud
Video
VodcsMME
Acc (P1)
STAR
Auc. (146)
QA
Aos 193
Ego Schema
853
Acc. (83)
Perception Test
dlc. (1G3)
Model
376 Tokess per Image
CLIP--1 (188]
53.5
61.7
49.5
32.8
70.1
72.7
60.7
61.9
83.3
78.9
113.3
92.0
132.9
115.0
65.0
54.2
46.3
52.1
68.6
$7.4
48.5
52,3
AIMv2.L Disill [36]
61.1
31.5
62.8
64.8
89.3
117.8
94.7
121.2
62.6
44.3
65.0
50.0
53.6
SigLIP2-so400M [135]
58.9
69.0
58.3
35.2
73.1
76.8
69.8
67.2
83.7
81.6
116.5
92.1
137.7
119.
67.4
54.5
45.5
53.1
67.2
57.6
49.3
54.5
SigLiP2-g-opt (135)
56.2
63.1
55.3
34.0
72.4
77.0
70.3
66.7
89.6
81.6
117.7
94.9
137.8
120.3
66.5
53.9
46.2
53.9
66.6
53.8
48.5
51.7
PF G1
66.9
76,8
73.6
41.1
76.1
76.2
68.5
66.0
89.1
81.3
119.7
96.1
130.6
133.4
68.9
58.1
48.7
58.9
70.5
61.8
52.7
55.9
1024 Tokens per Image
VnlemViT2.5-L
60.6
74,1
59.2
35.9
73.1
74.2
65.4
61.4
87.6
79.6
112.3
88.4
133,7
114.9
66.9
50.6
45.2
44.8
62.7
54.2
46.0
50.5
SigLIP2-so40M [135]
63.3
72.1
69.3
39.0
72.7
77.9
74.8
66.0
89.0
$1.8
117.4
93.5
138.1
120.2
69.6
55.8
46.2
55.4
67.0
62.0
50.0
54.5
1.
59.4
68.7
62.5
36.6
69.7
74.7
67.7
64.3
88.3
78.7
112.7
89.6
133.
114.9
59.7
50.9
41.7
51.2
61.6
52.6
47.4
50.6
L.
71.1
51.0
81.9
46.4
75.0
77.1
7300
05.5
89.3
117.3
94.3
137.3
130.1
70.5
56.5
47.0
57.2
68.0
59.8
52.3
54.7
DINOW2 [96]
19.6
24.2
19.3
109.4
110.1
39.7
52.1
AIMV2-98 [36]
48.9
40.5
53.9
35.9
61.2
73.0
64.1
64.0
85.2
78.9
115.7
95.8
135.2
118.1
36.1
34.6
45.1
54.5
06.7
55.4
51.7
54.3
IntemViT2.5 [18]
59.9
72.3
59.4
35.2
75.5
63.9
61.9
88.2
115.0
92.2
136.1
116.3
68.0
44.5
47.0
62.6
45.8
18.9
48.5
GF
60.8
69.9
65.4
36.7
71.1
73.3
65.9
60.7
88.4
78.0
112.5
91.6
133.6
112.4
66.6
52.0
42.3
53.1
61.9
51.4
48.8
53.6
G
72.4
80.5
84.4
48.3
76.4
78.1
75.2
65.4
90.1
$1.8
120.1
96.6
140,0
123.6
71.3
58,0
45.0
60.1
69.4
62.0
52.4
56.0
Table 5: MLLM Results. We benchmark PE vs.. other frozen vision encoders with Llama 3.1-
instruct 8B [80] as the L.M. PFang shows strong performance across all benchmarks, outperforming much larger models. Interpolated without extra training. See Appendix D.4 for more results.
8

---

5 Perception Encoder: Spatial Alignment
Unlike for language aligmment with an MLLM, the best way to spatially align a model not obvious. However, the path becomes clear when study an apparent dichotomy in $3 for PE higher level spatial tasks like detection and depth estimation perform optimally around layer 40, while low level tasks like tracking perform the best at around layer 30. Upon analyzing the features directly (see Appendix E. I), we find that locality begins to deteriorate starting at layer 33 due to global tokens [23].
Alignment Method. Following these insights, we design our spatial alignment method with two goals in mind: (1) keep the high level features around layer 40 in tact while (2) improving the locality of the features for lower level tasks. To address (1), we simply finetune Pooo using its own frozen layer 41 features as a teacher with heavy regularization (DropPath [49]. LayerScale [132], 75% masking [144]). Then, we enforce spatial correspondence for (2) using SAM 2.1 [108] mask logits. That is, unlike [44, 107, 116], we do not directly use SAM features but instead sample x22 points in grid and concatenate the SAM 2.1 mask logit for each into single feature map. As shown in Appendix Fig. 19, this provides features with strong locality. See Appendix B.3.1 1 for training details.
Effects. In Fig. 9, we compare layerwise perfor- mance of the original PE checkpoint com- pared to aligning to the teachers described above. We denote aligning to both teachers as PEsparial G. Aligning to G layer 41 alone performs gen- erally well on all tasks, but has lackluster per- formance on tracking, where percise locality is
Detection
Semantic Segm.
50
COCO 1000-awnno nulp
-8 23
ADE20kmOU
17
10
17
33
50
Frozen Encoder Layer
Frozen Encoder Layer
necessary to define boundaries. In contrast, align- ing to SAM 2.1 mask logits lowers last layer performance on every task but tracking. Thus, the optimal approach is to combine both teach- ers. As a result, PEspatal G not only lifts the fea-
Tracking
Depth (L)
DAVIS Zoro Shot
55
0.6
NYU RIAGEL
-35
0.4
tures for all tasks to the end of the network, but
it also improves over self-alignment, especially on tracking and semantic segmentation. Notably, PE G's tracking performance is lower than the SAM-aligned model, but it is still ahead of other methods while being generally good, see results.
17
33
50
17
33
50
Frozen Encodor
Frozen Encoder Layer
PEav6 algnadtz PEow
both
Piooweal
Figure 9: Spatial Alignment of PEçore G.
Last Layer Visualization. In Fig. 10, we vi- sualize last layer features for PE G and the 3 aligned models, with similar colors denoting similar features. In the first column, we see why the last layer performance of PEçore is so poor: while contains information about the salient objects, it seems to have lost spatial coherence. Aligning to the model's 's own layer 41 features fixes this, but its spatial quality is lacking. In contrast, the model aligned to SAM 2.1 mask logits has great locality, but no semantics (e.g.. low similarity between cats in row I and cows in row 2). PEspatial retains the semantics of PE while producing high quality spatial features.
PE
aligned to
wigned to 564 Logit
aligned to both
PE
Figure 10: Last Layer Visualization using PCA (see Appendix B.3.2). More in Appendix E.4.
Tracking
Segmentation
Depth
LNIS [40]
COCO (74)
DAVIS (1) [101]
ADE20K (1)(164)
NYU (1)
Encuder
APrea.
APrusk
Albos
APmak
Encoder
Best
idc
Best
Last
Best
Last
MetaCT.P--II [150]
45.1
41.9
53.2
46.7
SigLIPsss00M [1588
1627
40.1
38.3
.339
369
21/27
SigLLI2 [135]
Sigl 4D0M [135]
51.4
45.3
1527
44.0
42.9
24/27
.306
.329
25/27
Sigf 1P2-g-op [185]
52.9
48.5
57.1
50.2
DINO-2-L [96]
23/24
227
DINOx2-L[[66
55.7
DINO 196)
58.5
58.5
40/00
48.7
48.4
37400
.279
.20
27/40
DIND x [96]
51.5
47.3
57.2
50.0
G
56.8
52/50
41.5
35.6
.249
G
51.9
47.9
57.0
49.8
PF G
61.5
61.5
50,50
49.3
45.9
49/50
.262
.275
46/90
G
54.2
49,3
57.8
50.3
Table 6: Frozen Dense Prediction for the best and
Table 7: End-to-End Detection using Mask
last layers of each model. Details in Appendix B.3.3.
R-CNN [42]. Details in Appendix B.3.4.

---

Encader
Params
Detector
COCO
Results. In Tab. 6, we compare performance on dense tasks with frozen encoder with fixed 448 resolution, reporting both best layer performance and last layer per- formance. Across the board, PEspatiiG outperforms other state-of-the-art models, with its features well aligned to the last layer. In Tab. 7, the same is true when end-to-end finetuning for detection on both LVIS [40] and COCO [74]
SwinV24G [78]
3,08
HICee [14]
62.5
Swind. [77]
DINO [159]
63.2
IntemImage-G [142]
3.06
DINO [159]
65.3
EVA02-L [34]
0.30
CoDETR [165]
65.9
PRopatit
DETA 1971
66.0
Table 8: SOTA Setting Detection on
COCO val. Recipe in Appendix B.3.5.
with a fixed 1024 resolution using Mask-RCNN [42] and ViTDet [71]. Finally, in Tab. 8, we pro- vide system-level comparison Object365 [117] as extra detection data, PEspaia can match the performance of more complex models tuned for detection, while only using simple DETR-style decoder [11, 97]. PEspatiaG marks the first general, contrastively pretrained model to accomplish this.
the absolute sate-offtheearr on COCO val2017. With only
6 Related Work
Vision-language pretrained models have served foundation for zero-shot image classification and image-text retrieval [50, 103, 1 114), open- vven-vocabulary detection [62, 92, 93] and segmentation [22, 27], and multimodal large language models (MLLMs) [3, 5, 76, 91, 98, 131]. PE iterates on this paradigm.
Contrastive Language-Image Pretraining. The early works of Virtex [26], ICMLM [112], and CIRT [161] developed the techniques for learning through contrastive objectives between vision and language modalities. Subsequently, vision encoders such as CLIP [50, 103] and ALIGN [53] scaled these techniques to much larger datasets and model sizes, popularizing vision- vision-language contrastive learning. A series of gpenh contrastive models have been developed to enhance the performance and robustness of CLIP [32, 70, 1 114, 126, 150, 158]. PE is among this effort.
Existing Techniques. Various techniques used in this work have been explored before. BASIC [99] and LAION [1 [1]4] explored scaling the batch size up to 160K, and shows the benefits of large batch sizes during training. EVA-CLIP [127] uses LAMB optimizer [154] for large batch training of clip models. Rotary positional embedding (RoPE) [124] has been successfully adopted in large language models. In vision transformers [2, 47] adopted 2D rotatory positional embeddings. For data engine, series of works focus large-scale sourcing and filtering through efficient data curation [32, 38, 114, 150] and explore recaptioning training images using MLLMs or VLMs [31, 63, 94, 149]. We extend these concepts create a robust training recipe and extend data engines to video.
Intermediate Layers Are Better. Most vision encoders rely on the last layer to extract features. However, when trained on proxy self-supervised tasks, the last layer is often not the ideal candidate for other tasks [8, 15, 16, 29, 83, 104, 118, 125, 139, 157, 163]. This has been shown for image coloration [160, 163], next token prediction [15, 29, 104], image generation [83, 157], and limited extent in CLIP models [125]. In contrast to these works, we first show the same behaviors across multiple classes of models simultaneously. Then we study this behavior for PE specifically in depth, and show it is possible for CLIP training to produce rich spatial and language features in intermediate layers on par with the best existing models for each. Finally, we show how to align these features with short finetuning steps to obtain state-of-ther ohettherrt on a wide variety of tasks. Unlike other alignment [3, 18, 19, 65, 80, 129, 141] and feature combination [44, 107, 116, 157] methods, our main goal is not to instill large amount of new knowledge into the model, but instead to bring out and refine the latent strong general features that already exist in the original PE model.
7 Conclusion
In this work, we have presented Perception Encoders (PE), family of best-in-class foundation models comprising PEgee. PE and PE We have shown that Pf can outperform the leading models in zero-shot image recognition, while also excelling in zero-shot video recognition. We have demonstrated that PE outperforms the best vision encoders for use in multimodal large language models, often by a large margin. We have established that PEspatial outperforms the long- standing state-of-the-art in object detection with a simpler decoder. Throughout all of this, one conclusion is abundantly clear: Perception Encoder unlocks the potential to scale simple contrastive vision- language pretraining to address a wide range of downstream vision tasks.
10

---

Additional Contributors and Acknowledgments. We would like to thank Abhimanyu Dubey, Adel Ahmadyan, Andrew Westbury, Arkabandhu Chowdhury, Azita Shokrpour, Babak Damavandi, Chay Ryali, Ching-Feng Yeh, Cyprien de Lichy, Didac Suris George Orlin, Han Zou, Harry Tran, Jitendra Malik, Joelle Pineau, Joseph Greer, Kavya Srinet, Kirmani Ahmed, Laura Gustafson, Lu Zhang. Muhammad Maaz, Natalia Neverova, Nicolas Carion, Oleksandr Maksymets, Ramya Raghavendra, Romy Luo, Ronghang Hu, Sam Doud, Sasha Mitts, Sean Bell, Shane Moon, Shuming Hu, Soerian Lieve, Stephane Kasriel, Valentin Gabeur, Vanessa Stark, Vignesh Ramanathan, Vivian Lee, Xuan Hu, Yang Li, and Ziyang Wang for their contributions and support for the project. And we thank you, the reader, for reading this far.
Dong Wang, Filip Radenovic,
References
[1] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishahh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In ICCV, 2019.
[2] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophilc Gervet, Soham Ghosh, Amélic Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo. William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang. and Sophia Yang. Pixtral 12b. arXiv:2410 240.00730, 2024.
[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile Eision-language model for understanding, localization, text reading, and beyond. arXiv:2308. 230.:66660.166, 2023.
[4] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurlPS, 2019.
[5] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschanmen, Emanuele Bugliarello, Thomas Un- terthiner, Daniel Keysers. Skanda Koppula, Fangyu Liu, Adam Gryener, Alexey A. Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong. Julian Eisenschlos. Rishabh Kabra, Matthias Bauer. Matko Bosnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevie, Joan Puigcerver, Pinclopi Pa- palampidi, Olivier J. Hénaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3b VLM for transfer. arXiv:2407.00 2407.0726, 2024.
[6] Navanecth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Noft Improving object detection with one line of code. In /CCV, 2017.
[7] Daniel Bolya, Chaitanya Ryali, Judy Hoffmann and Christoph Feichtenhofer. Window attention is bugged: how not interpolate position embeddings. In ICLR, 2023.
[8] Florian Bordes, Randall Balestriero. Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Why removing layers is needed to improve generalization in self-supervised learning. arXiv:2206. 13378, 2022.
[9] Gary Bradski. The OpenCV library. Dr. Dobb's 's Journal: Software Tools for the Professional Programmer, 2000.
[10] Fahian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding, In CVPR, 2015.
[11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[12] Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Saining Xie, and Christopher D. Manning. AuroraCap: Efficient, performant video detailed captioning and a new benchmark. ICLR, 2025.
[13] David I.. Chen and William B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACI., 2011.
[14] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid task cascade for instance segmentation. CVPR, 2019.
[15] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.

---

[16]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[17] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akhari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaha Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly- scalel multilingual language image model. In ICLR, 2023.
[18]
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuchui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang. Tan Jiang, Bo Wang. Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang. Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge. Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-souree source multimodal models with model, data, and test-time scaling.
2024.
[19] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVI: Scaling up vision foundation models and aligning for generic iisual-linguist tasks. In CVPR, 2024.
[20] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017.
[21] Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muham- mad Maaz, Yale Song. Tengyu Ma, Shuming Hu, Hanoona Rasheed, Peize Sun, Po-Yao Huang. Daniel Bolya, Suyog Jain, Miguel Martin, Huiyu Wang, Nikhila Ravi, Shashank Jain, Temmy Stark, Shane Moon, Babak Damavandi, Vivian Lee, Andrew Westbury, Salman Khan, Philipp Krähenbühl, Piotr Dollár, Lorenzo Torresani, Kristen Grauman, and Christoph Feichtenhofer. Perceptionlm: Open-access data and models for detailed visual understanding. arXiv:2504. 18000, 2025.
[22] Seokju Cho, Heeseong Shin. Sunghwan Hong, Anurag Amnb, Paul Hongsuck Seo, and Seungryong Kim. CAT-Seg: Cost aggregation for open-vocabulary semantic segmentation. CVPR, 2024.
[23]
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
[24] Mostafa Dehghani. Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton. Lucas Beyer, Michael Tschamnen, Anurag Amab, Xiao Wang, Carlos Riquelme, Matthis Minderer, Joan Puigcerver, Utku Evei, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos. Yi Tay, Thomas Mensink. Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučié, Xiaohua Zhai Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers 22 billion parameters. In ICML, 2023.
[25] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[26] Karan Desai and Justin Johnson. VirTex: Learning visual representations from textual annotations. In CVPR, 2021.
[27] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR, 2022.
[28] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
[29] Alaacldin EI- No-b,, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre- pre-training of large autoregressive image models. In ICMI., 2024.
[30] David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas. Yann LeCun, Amir Bar, and Saining Xie. Scaling language-free visual representation learning, arXiv:2504.01017 2025.
[31] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving CLIP training with language rewrites. In NewrlPS, 2023.
[32] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In ICLR, 2024.
12

---

[33]
Yuxin Fang. Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
[34] Yuxin Fang, Quan Sun, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. EVA-02: A visual representation for neon genesis. Image and Vision Computing, 2024.
[35]
Christoph Feichtenhofer. X3D: Expanding architectures for efficient video recognition. In CVPR, 2020.
[36]
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, Alexander T. Toshev, Marcin Eichner, Moin Nabi, Yinfei Yang, Joshua Susskind, and Alaaeldin El-Nouby. Multimodal autoregressive ure-training of large vision encoders. In CVPR, 2025.
[37] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Vidco- MME: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv:240..2107. 2024.
[38] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymiss Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang. Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song. Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. DataComp: In search of the next generation of multimodal datasets. In NeurlPS, 2023.
[39] Yash Goyal, Tejas Khot, Douglas Summers-Stur Stay, Dhruv Batra, and Devi Parikh. Making the v in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
[40] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation.
In CVPR, 2019.
[41]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[42] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
[43]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[44] Greg Heinrich, Mike Ranzinger, Hongxu, Yin, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavlo Molchanov. RADIOV2.5: Improved baselines for agglomerative vision foundation models, In CVPR, 2025.
[45] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of -uitof-distribution generalization. ICCV, 2021.
[46] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
[47] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In ECCV, 2024.
[48] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurlPS Deep Learning Workshop. 2015.
[49] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
[50] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namknong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP. 2021.
[51]
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. In NeurIPS, 2020.
[52] Yunseok Jang, Yale Song. Youngjae Yu, Youngjin Kim, and Gunhee Kim. TGIF-QU -GIF-QA: Toward spatio- temporal reasoning in visual question answering. In CVPR, 2017.
[53] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoe Lc, Yun-Hsuan Sung. Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021.
[54] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaanarasimman. Fabio Viola. Tim Green, Trevor Back, Paul Natsev. Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. (tff:1775.00954 2017.
13

---

[55] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects in photographs of natural scenes. In EMNLP, 2014.
[56] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016.
[57] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV. 2023.
[58] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-gr ine-grained categorization. In ICCV Workshop, 2013.
[59] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome:
Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
[60] Alex Krizhevsky, Nya Sutskever, and Geoffrcy E Hinton. Imagenct classification with deep convolutional neural networks. In NewrlPS, 2012.
[61] Hildegard Kuchne, Hucihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: large video database for human motion recognition. ICCV, 2011.
[62] Weicheng Kuo, Yin Cui, Xiuye Gu, A. J. Piergiovanni, and Anelia Angelova. F-VLM: open spcabuocabulary object detection upon frozen vision and language models. In ICLR, 2023.
[63] Zhengfeng Lai, Haotian Zhang, Bowen Zhang. Wentao Wu, Haoping Bai, Aleksei Timofeev. Xianzhi Du, Zhe Gan. Jiulong Shan, Chen-Nee Chuah, Yinfei Yang. and Meng Cao. VeCLIP: Improving CLIP training via visual-enriched captions. In ECCV, 2024.
[64] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? In NeurIPS, 2024.
[65] Bo Li. Yuanhan Zhang, Dong Guo. Renrui Zhang, Feng Li. Hao Zhang. Kaichen Zhang, Yanwei Li. Ziwei Liu, and Chunyuan Li. LLaVA-One Visioon Easy visual task transfer. TMLR, 2025.
[66] Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang. Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training- efficientt video foundation models. ICCV, 2023.
[67] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang. and Yu Qiao. MVBench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024.
[68] Liunian Harold Li, Pengehuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai- Wei Chang, and Jianfeng Gao. Grounded language- image pre-training. In CVPR, 2022.
[69] Xianhang Li, Zeyu Wang, and Cihang Xie, An inverse scaling law for CLIP training. In NeurlPS, 2023.
[70] Xianhang Li. Zeyu Wang, and Cihang Xie. CLIPA-v2: Scaling CLIP training with 81.1% zero-shot imagenet accuracy within a $10,000 budget; an extra $4,000 unlocks 81 1.8% accuracy. arXiv:2306. 15658, 2023.
[71] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer hackbones for object detection. In ECCV, 2022.
[72] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Ri-Rong Wen. Evaluating object hallucination in large rision-languae models. In EMNLP, 2023.
[73] Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. TIP, 2024.
[74] Tsung- Yi Lin. Michael Maire. Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[75] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA- NeXT: Improved reasoning, ocr, and world knowledge, 2024.
[76] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurlPS, 2024.
[77] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen I. Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In ICCV, 2021.
[78] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang. Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022.
[79] Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In CVPR, 2022.
14

---

[80] AI @ Meta Llama Team. The Ilaaa 3 herd of models. arXiv:2407.21788 2024
[81] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
[82] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. CLIP4Clip: An empirical study of clip for end to end video clip retrieval. Neurocomputing, 2021.
[83] Nanye Ma, Mark Goldstein, Michael Alhergo, Nicholas Boffi, Eric Vanden- Bijjden, and Saining Xie. SiT: Exploring flow and inffusion-based generative models with scalable interpolant transformers. In ECCV, 2024.
[84] Muhammad Maaz. Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video- ChatGPT: Towards detailed video understanding via large vision and language models. In ACL, 2024.
[85] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. VideoGPT+: Integrating image and vidco encoders for enhanced vidco understanding. arXiv:2406.09411 2024,
[86] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. 1xxiv:300.51550 2013.
[87] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik, Egoschema: A diagnostic benchmark for very long-form video language understanding. NeurIPS, 2024.
[88] Kevis-Kokitsi Maninis, Kaifeng Chen, Soham Ghosh, Arjun Karpur, Koert Chen, Xia, Bingyi Cao, Daniel Salz, Guangxing Han, Jan Dlahal, et al. Tips: Text-image pretraining with spatial awareness. arXiv:2410. 165/2, 2024.
[89] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A dataset for vqa on document images. In WACV, 2021.
[90] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievqa. In WACV, 2022.
[91] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang. Chong Wang. Nan Du. Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang. Ruoming Pang. Peter Grasch, Alexander Toshev, and Yinfci Yang. MM1: methods, analysis and insights from multimodal LLM pre-training. In ECCV, 2024.
[92] Matthias Minderer, Alexey A. Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenbom, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amnb, Mostafa Dehghani, Zhuoran Shen, Xiao Wang. Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection with vision transformers. In ECCV, 2022.
[93] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. In NeurIPS, 2023.
[94] Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. In NeurlPS, 2023.
[95] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In ICVGIP. 2008.
[96] Maxime Oquab, Timothée Darcet. Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang. Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnacve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
[97] Jeffrey Ouyang-Zhang, Jang Hyun Cho, Xingyi Zhou, and Philipp Krähenbühl. NMSstrikes back. arXiv:2212.06137, 2022.
[98] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furn Wei. Kosmos-
2: Grounding multimodal large language models to the world. arXiv:2306.1488 14824. 2023.
[99] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thane Luong, Yonghui Wu, Mingxing Tan, and Quoe Le. Combined scaling for zero-shot transfer learning. Neurocomputing, 2023.
[100] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- -hrras correspondences for richer image-to to-etennee models. In ICCV, 2015.
[101] Jordi Justt, Federico Perazzi. Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Hornung, and Luc Van Gool. The 2017 DAVIS challenge on video object segmentation. (rXX::1704.00677 2017.
15

---

[102] Viorica Patrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang. Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette. Hamna Klimczak, Raphael Koster, Junlin Zhang. Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In NeurlPS, 2024.
[103] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[104] Jathushan Rajasegaran. Ilija Radosavovic, Rahul Ravishankar, Yossi Gandelsman, Christoph Fe- ichtenhofer, and Jitendra Malik. An empirical study of autoregressive pre-training from videos. NrXiv:2501.05453 2025.
[105] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text-conditional image generation with CLIP latents. arXiv:2204.06125, 2022.
[106] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, 2021.
[107] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. AM-RADIO: Agglomerative vision
foundation medul-reduce all domains into one. In CVPR, 2024.
[108] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM Segment anything in images and videos. In ICLR, 2024.
[109] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? ICML, 2019.
[110] William A. Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. The dollar street datasct: images representing the geographic and sociocconomic diversity of the world. In NeurlPS Datasets and Benchmarks, 2022.
[111] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[112] Mert Bulent Sariyildiz. Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, 2020.
[113] Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis. UNIC:
Universal classification models via multi-teacher distillation. In ECCV, 2024.
[114] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Sri- vatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION- 5h: An open large-scale dataset for training next generation image-text models. In NeurlPS Datasets and Benchmarks, 2022.
[115] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: benchmark for visual question answering using world knowledge. In ECCV, 2022.
[116] Jinghuan Shang, Karl Schmeckpeper. Brandon B May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning. In CoRL, 2024.
[117] Shuai Shao, Zeming Li. Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang. Jing Li, and Jian Sun. Objects365: A large-scale. high-quality dataset for object detection. In ICCV, 2019.
[118] Shashank Shekhar, Florian Bordes, Pascal Vincent, and Ari Morcos. Objectives matter: Undorstanding the impact of self-supervised objectives on vision transformer representations. arXiv:2304. 13089, 2023.
[119] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanprect Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, 2020.
[120] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[121] Karen Simonyan and Audrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
[122] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR, 2019.
[123] Khurram Scomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv:1212.0402. 2012.
16

---

[124] Jianlin Su. Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024.
[125] Lin Sun, Jiale Cao, Jin Xie, Xiaoheng Jiang, and Yanwei Pang. CLiPer: Hierarchically improving spatial representation of CLIP for open-vocabulary semantic segmentation. arXiv:2411. 2024.
[126] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved training
techniques for clip at scale. arXiv:2303. 15389, 2023.
[127] Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. EVA-CLIP-I8B: Scaling clip to 18 billion parameters. rrXiv:2402.02222 2024.
[128] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[129] Gemma Team. Gemma 3 technical report. arXiv:2503 2025.
[130] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Flizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, 2016.
[131] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang. Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-I: A fully open, vision-centrie exploration of multimodal llms. NewrlPS, 2024.
[132] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021.
[133] Hugo Touvron, Matthicu Cord, and Hervé Jégou. DeiT III: Revenge of the VIT. In ECCV, 2022.
[134] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image eaptioners are scalable vision leamers too. In NewrlPS, 2023.
[135] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual vision- langugg encoders with improved semantic understanding, localization, and dense features. WxXix:2502.14788 2025.
[136] Jack Urhanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture is worth more than 77 text tokens: Evaluating CLIP-style models on dense captions. In CVPR, 2024.
[137] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
[138] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
[139] Matthew Walmer, Saksham Suri, Kamal Gupta, and Abhinav Shrivastava. Teaching matters: Investigating the role of supervision in vision transformers. CVPR, 2023.
[140] Haohan Wang. Songwei Ge, Zachary Lipton, and Erie P Xing. Learning robust global representations by penalizing local predictive power. In NeurlPS, 2019.
[141] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men. Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL VL: Enhancing vision- languaee model's perception of the world at any resolution. arXiv:2409.15
2024.
[142] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu. Hongsheng Li, Xiaogang Wang, and Yu Qiao. Internlmage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, 2023.
[143] Yi Wang. Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang. Yu Qiao, Yali Wang, and Limin Wang. InterViide22. Scaling foundation models for multimodal video understanding. In ECCV, 2024.
[144] Chen Wei, Hanqi Fan, Saining Xie, Chao-Yusa Yuan Wu, Alan L.. Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
[145] Bo Wu, Shoubin Yu. Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. In NeurlPS, 2021.
[146] Yuxin Wu, Alexander Kirilloy, Francisco Massa, Wan-Yer Yen Lo, and Ross Girshick. Detectron2, 2019.
[147] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yes Lo, and Ross Girshick. Detectron2, https: //github http:///w///.ooooooor/////////oe 2019.
17

---

[148]
Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN database: Exploring large collection of scene categories. IJCV, 2014.
[149] Hu Xu, Po- Yao Huang, Xiaoqing Ellen Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen tau Yih, Shang- Wen Li, Saining Xie, and Christoph Feichtenhofer. Image captioning via re-aligning alt-text. In EMNLP, 2024.
Altogether:
[150] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang. Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2024.
[151] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT large video description dataset for bridging video and language. In CVPR, 2016.
[152] An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men. Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu, Wenbin Ge. Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zoyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arxiv:2407. 10671, 2024.
[153] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. 2rXiv:2412.15115 2024.
[154] Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In ICLR, 2020.
[155] Peter Young. Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL. 2014.
[156] Jiahui Yu. Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. TMLR, 2022.
[157] Sihyun Yu, Sangkyung Kwak, Huiwon Jang. Jonghcon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In ICLR, 2025.
[158] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
[159] Hao Zhang, Feng Li, Shilong Liu, Lci Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Young Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2023.
[160] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[161] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Contrastive
learning of medical visual representations from paired images and text. In MLHC. 2022.
[162] Hanwen Zheng, Sijia Wang, Chris Thomas, and Lifu Huang. Advancing chart question answering with robust chart component recognition. In WACV, 2025.
[163] Liang Zheng, Yali Zhao, Shengjin Wang. Jingdong Wang, and Qi Tian. Good practice in cnn feature transfer. arXiv:1604.00 30X1::1604.00133,, 2016.
[164] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. CVPR, 2017.
[165] Zhuofan Zong, Guanglu Song, and Yu Liu. DETRs with collaborative hybrid assignments training. In ICCV, 2023.
18

---

Appendix
Table of Contents
A Video Data Engine
20
A.1 PE Video Dataset (PVD)
20
PE Video Dataset Details
21
B Implementation Details
25
B.1 PE: Core
25
B.2 PE: Language Alignment
27
B.3 PE: Spatial Alignment
28
Additional PEçore Results
32
Robust Image Pretraining
32
C.2 Additional Video Ablations
33
C,3 Smaller Models
34
C.4 Additional Results
35
C.5 Additional Layerwise Scaling Analysis
37
D Additional PE Results
38
Alignment Method Derivation
................................
D.2 Layer Analysis Details
................................
38
D.3 Unfrozen Encoder Results
39
D.4
Additional Results
40
E
Additional PEspataa Results
43
E.1 Alignment Method Derivation Derivation......................
43
E.2 Layer Analysis Details..
44
Smaller Models
45
E.4
Additional Qualitative Results
46
F
Broader Impacts
47
19

---

Category: Food Preparation
shows person making cake, person is seen mixing ingredients a bowl, pouring the misture pan, and adding chocolate chips. The video also shows the person using a
caption:
Human rofinement: video shows parson making chocolate cake. The person mixes ingredients in glass bowl and pours the mixture
baking Then, the person adds chocolate chips top. video also shows the person mixing ingredients with an electric whisk and scraping the mixture into the pan with spatula, The parson is wearing blue shirt and is seen hoiding glass
a spatula to mix ingredients and pour the mixture into the pan. The person is wearing blue shirt and is seen holdiing glass bowl and speon. The background of the video white marble countertop, towel on the counter."
is striped
marble countertop, and there is striped towel on the counter."
background
video
Figure 11: PE Video Dataset Example. A sample from PVD, released video-text dataset. Initial captions are generated by our video captioning model and then refined by human annotators. Annotators are instructed to add details and remove model hallucination. In this example, the model hallucination "a spoon" is removed; and more details such as "glass bowl" and the action "scraping" added. See Fig. 12 for more. Data at https: //ai. meta. com /wtasatst/t/s/eeeeee video/
A Video Data Engine
A.1 PE Video Dataset (PVD)
For the benefit of the community, we release new video dataset: PE Video Dataset (PVD). PVD comprises of 1M high-quality and diverse videos with accompanying tags and descriptions. The videos motion-centered, covering both first-person and third-person views with wide coverage of scenes.
We additionally select 120K of these videos with the highest degree of motion to annotate with detailed captions by generating synthetic captions using our video captioner ($2.2) and employing 200 annotators to verify and refine them. We ask the human annotators to improve the synthetic captions by removing any hallucinations, correcting words that describe the video inaccurately, eliminating repetitive or redundant words make the caption more concise, and adding any missing actions being performed in the video.
We release two versions of annotations for the 120K PVD subset: (1) Human verified captions: extended summaries with an average length of 57.1 words that provide a high-level description of each video. These captions are suitable for CLIP-style training. (2) Long automated captions: detailed and fine-grained descriptions with average length of 111.7 words that capture spatial and temporal events. These captions are ideal for fine-grained video understanding.
958,862
Human Captions
118,662
Total Deration
4625 hrs
Duration
16.7:19.
Human Caption Length
57.1:125.4
Model Caption I Length
111.74:43.2
Table 9: PVD Stats.
In Fig. 11, we visualize a video example together with their model and human captions from PE Video Dataset (See Fig. 12 for more). The dataset statisties are summarized in Tab. 9. Finally, We use 105K of these refined samples to improve the data engine ($2.2 phase 2) and 15K as a high-quality video retrieval benchmark.
PVD Benchmark. We use 15K of the human-refined video-caption pairs as a held-out test set, which we introduce as new video retrieval benchmark, PVD Benchmark, to evaluate finegrained video-caption alignment. We follow the format of MSR-VTT [151] to construct the benchmark. We select videos from 10 different categories, including hand actions, object interactions, food preparation, work activities, outdoor scenes, animals, water scenes, object handling, close-up shots, and nature scenes, with an overall average caption length of 51.7 words (see Appendix A.2. for statistics). We use PVD Benchmark to evaluate SigL.P [158], SigL.IP [135], VI. [19], and PE models, and the results can be found in Tab. 25.
20

---

A.2 PE Video Dataset Details
As mentioned above, PVD consists of 1M videos, 120K of which have human- video captions and are selected for high motion content. We also select 15K from the 120K videos benchmark.
A.2.1 PVD Benchmark Distribution
Namber of
Avg. Caption Length
Category
videos
Hand Actions
2145
54.2
Object Inseractions
42,6
Food Preppration
[69]
56.8
Work Activities
47.8
Outdoor Scenes
1558
50.7
Animals
1423
50.9
Water Scenes
Object Handling
1307
51.6
Close up
Nature Scrmer
866
38.4
Table 10: PVD Benchmark Statistics. We created dataset of 15K videos together with human- verified captions. The videos are motion-centered, covering both first-person and third-person views with wide coverage of scenes.
A.2.2 Video Data Filtering Pipeline
The goal of video data filtering is to identify videos that contain motions such as object motion, camera motion, interaction between objects, human actions, sequences of actions, and manipulation of objects, while rejecting videos with static scenes, like landscapes, or those that are artificial or highly edited.
To achieve this, we created a video filtering pipeline consisting of the following steps:
Step 1: Compute motion features. For each video, we compute list of features from video frames, including frames per second (fps), number of frames, number of I-frames, motion vector magnitude, and motion vector variance, using off-the-shelf tools like OpenCV [9].
Step 2:
Extract video frame features. For each video, we uniformly sample three frames and encode them using DINOv2 model [96] and SigLIP model [158].
Step 3: LLM Features. For each video, we also run a multimodal large language model (LLM) like Alava-Onevisioe QwenLM 0.5B [65] to extract MLLM features. We composed a list of 26 questions and performed MILMM inference on the videos. The questions can be found here $..2.3.
Step 4: Video Quality Scoring. We combine all the features collected far and random forest model to predict a score between 0 and 5. To train the model, we manually annotated approximately 1,000 videos with scores between 0 and 5. A low can be nearly summarized by a single frame, while a high score indicates that there are multiple temporal events in the video, requiring several frames to accurately caption it. We use these annotated videos as training data to fit random forest model for video quality score prediction.
indicates that the video is almost static and
Step 5: We apply k-means clustering to the videos and rank them within each cluster. By selecting the top-ranked videos from each cluster, we effectively reduce the number of duplicated videos the final dataset.
A.2.3 LLM Feature Extraction
We use LLaVA-OneViision [76] model to extract LLM features from the videos. For each video, we prompt with 26 different questions to extract features ranging from, "is the video landscape video?" to, "are there any moving objects in the video?" The features are then used by random forest model to determine the video quality score.
21

---

LLM Feature extraction question list
Is the camera capturing the scene static? Reply yes or no. moving? Reply yes the video capturing landscape? Reply yes no. the video capturing a static scene? Reply yes or no. Is the scene captured from a distance? Reply yes or no.
Is the camera capturing the
Is the video captured with drone? Reply
Is the video computer-generated" Reply yes the video content abstract? Reply yes or no. Is there something moving through the scene? Reply yes or no. Is there someone doing something in the video? Reply Are there several things moving in the video? Reply yes there an object that being manipulated? Reply yes or no. Are there animals the video? Reply yes or no.
no.
Is the scene mostly static? Reply yes or no.
Are things occluding each other in this video? Reply yes Is there something obstructing the view apart from the watermark? Reply yes there a large number things in the video? Reply yes or no. Are there more than 5 different objects in the video? Reply yes or no. Is it hard to keep track of some entities because they are moving so much? Reply yes
someone looking a phone, tablet Are they looking at a phone, a tablet or a computer screen during the whole video? Reply yes or no.
computer screen? Reply yes or no.
Are there several moving persons in this video? Reply yes or no. Are there several moving animals in this video? Reply yes there several objects this video? Reply yes or no. Are there several similar-looking objects the video? Reply yes or no.
Do they look similar? Reply yes or no.
A.2.4 Video Caption
LLM Summarization prompt
LLM Summar ization prompt 72 tokens
Create concise caption of video using the provided metadata, video caption, and frame captions.
TASK: Extract key information from the captions and combine into an alt text format using single phrase or set of phrases that includes all relevant details. Steps to Follow:
Review the metadata (title and description) for general context, you rely it entity names do not rely on it as the primary source information for your caption.
1.
2. Blend title / description with video caption and frame captions for the main storyline
Extract the most relevant concise information.
4. Combine extracted information into alt text format using short phrase or set of phrases with approximately 120 tokens, considering special characters like comma as part of the taken count.
5. Prioritize including all key information sentence structure grammar.
Minimize the use of special characters focus key information. What to Avoid:
Avoid adding or inferring information not present in the original metadata and captions.
Avoid using complex sentence structures prioritizing sentence flow.
Create a concise caption video based on the metadata, video caption, and frame captions.
22

---

A.2.5 Human Caption Refinement
We employ human annotators to perform caption refinement for the LLM-generatee captions in PVD, as described Appendix 1. Each annotator is paid a fair wage in compliance with all local laws in the annotators' jurisdictions. The refinement task was developed in accordance with an internal review to ensure ethical consideration for the participants. For instance, the annotators are instructed to reject the job completely if the videos contain any explicit content. Otherwise, we provided the following materials for the annotators as instructions for the task:
Goal. Given a video and caption, directly refine it to make the caption:
No Repeating:
(a) Remove any repeating, redundant information
(b) Note it is ok to have fine-grained or atomic information if the caption already contains it, if the information is still unique
2. Accurate:
(a) Every word the caption is describing fact in the video
(b) If anything doesn't exist in the video at all, remove it
(c) If anything is incorrect comparing to what the video shows, correct it
3. Action Focus:
(a) Add any missing major action information into the caption
i. As mentioned in 1. above, if an atomic action exists already, it is ok to keep it. No need remove it. We only care about adding the missing major actions back.
In summary, the submitted caption should have no repeating information, every single word in the caption accurate reflecting fact in the video, and all major actions shown in the video have been covered.
Refinement Criteria. Use the following guidelines for correcting errors:
1. Error:
If some words describe something doesn't 't show clearly in the video
Remove it from the caption
2. Error: If some words describe something in the video but incorrectly
Correct it from the caption to describe the fact in the video
3. Error: Repeating redundant words
Merge words from the caption to make it concise and accurate or just remove it if no need of merge
4. Error: Action related words
If atomic actions exist, No need remove it. Only remove words for Errors 1 and 3 major actions miss, Add them back into the caption, in a concise and natural way. E.g.
(a) If the missing actions can just be integrated as just part of the original sentence, then just integrate it
(b) a new sentence is more natural to add the missing action back, then just add new sentence.
23

---

MRRA
Category: Hand Actions
Captlon: The casturee dloopp rous typlig laybrard
Captiers andinhii puprerg
giobe and soma surbber can be swem the frams and the deo anck.
altereting black und whete that
Category: Preparation
Category: Wore Activites
India piddl then plece, chopping theae cubea, personia working on visddn cutting frame pérceall atable
Captier The video using scooping eahes remoxingg them the fiuplacs.
Irsplacs
Category: Cutdbor Speees
Category.Antmatt Antmats
Captise: video showey ippentesstrutttr stuctuee modes of siiedd and structures aumoundedd regitation
grwingrea and others wfele maland mnto thstruccure from etande moves aword
Caption The chouts white andigay anditan kittees soul gooining
the framn's uppar wit to par the two Mitaaarara
Category Water Scenes
Category: Object Mandling
Caption:Tre video showe achozl of body right frams. camera pana right.
Captiors The video person boxilot
than
blesss background Marry.
Eategory-Cloeeene Shats
Category: Nature Scenes
Caption: video cocnntee two brown buzileo and yellow backet. them per sightt waaniggrg grean glowe thmm topingtt yallowwww in backggonnd an irabisaoia and mara conle throughour
grear
sallow
Captisa
of fvarntie aeribaaee on fodd Neois numiing
peraon then the fioeeer....
buchels
billwingg spinto the aky.
Figure 12: More PE Video Dataset Examples. For each of the ten categories, we randomly pick one video and show its video caption. The captions were generated by our video data pipeline and then refined by human annotators.
24

---

B Implementation Details
B.1
PE: Core
We provide additional implementation details for building PEçore- Our implementation is based on OpenCLIP.
B.1.1 Architecture and Training Setups
Model Architecture. Following CLIP, Pf comprises Transformer-based [138) vision and a text encoder. We employ customized Transformer configurations as detailed in Tab. 11. For pooling, we
an attention pooling block in the style of SigLIP [158] with 8 heads from the last-layer feature to construct image and video embeddings. Regarding positional embedding, we use 2D RoPE [124] for relative positional embeddings and 2D learnable absolute positional embeddings (abs) the same size as the model's input resolution. We interpolate positional embeddings to enable support for various resolutions beyond the default. The text context length is 72 for G-scale and 32 for B and L-scale models. Originally bug, we find it optimal to not disable the class token when using attention pooling for smaller models. Thus, the B and L models use class token, then the attention pooling layer probes all features at once (class token included). Finally, we use an input mean and standard deviation of (0.5, 0.5, 0.5) for simplicity.
Scale
Parmms
Wilth
Dapiis
Heads
CLIIP
Pooling
Positional
Resalation Comeei Len
Patch
Token Register
Dim
Embedding
8
Vision
0,008
768
3072
1024
RoPEsAhs
Text
0.318
1024
24
4066
16
BOS Token
Abs
32
L
Vision
0.328
1024
24
2016
16
1024
Attn Paci
RoPE4
336
14
Text
0.318
4066
BOS Token
Vision
1,888
1536
50
8950
1280
Aitn Pool
RoPE+Abs
448
14
x
Text
0,47B
BOS Token
Table 11: PE Model Configurations with full details.
PE Core Training. As discussed in $2.3, the training of PE involves three stages: 1) image pretraining; 2) image and video finetuning; and 3) an additional model distillation for smaller models. These three stages work together to develop robust and effective PE model.
We first provide training recipes for 1) image pretraining Tab. and 2) video finetuning in Tab. 13.
config
values
config
values
config
values
optimizer
LAMB (0.9, 0.95)
optimizer
LAMB
optimizer
LAMB
31 R2 weight decay
(1, .22
(0.9. 0.95)
Fi 19
(0.9, 0.95)
0.05
weight decay
0.05
weight decay
0.05
leamnnng rale
2e-3
leaming rate batch
1e-6
learning rate
1c-5
batch
131,072
16384
wom-up stops
2K
wamt-up stops
2X
wamt--p steps
2K
mninnng sieps
L) 656K
training sleps
training steps
269K
data quantit
5.4R
datn guantity
22M
data quamity
5.4R
samples
S8B
86B
samples
samples
max logit scale
100
max logit scale
100
max logit scule
100
mask reg nito musk rog butch
0.4
mamber of frames
teacher logit scale
200 (8C3)
8192
jinter ar (3.3......333
data Table 14: Distillation.
None
112-160-224
00.00..11
prggrssive
98- 164-224.66 (1.)
ang
color jitter j (0..2,0.0..2, 0..322200
98- 154--24--336-455
hilpp p (0.6)
Table 13: Video Finetuning.
aspect jitter (0.75.1.88)
crop 10
aug
color jitter 10 20.03,,00033 030.32.0
adip (Co.5)
Table 12: Image Pretraining.
After training the largest G-scale model, we train the smaller models with image pretraining, then distill with image distillation in Tab. 14, then finally apply video finetuning at the end.
MIT License
25

---

Distillation Method. To maximize the performance of smaller models (B and L scales in Tab. 2), we employ a distillation finetuning approach [48] using P G as the teacher. This process involves short finetuning schedule where both the student and teacher models encode image and text inputs separately to compute image-to-text and text-to-image similarity distributions, similar to CLIP training [103]. The student's 8 distributions are then optimized to match those of the teacher by minimizing KL-divergence, distilling multimodal relational knowledge from the teacher into the student.
Notably, we find that using smaller softmax temperature for the teacher's distributions, specifically 0.5x the temperature used for the student's 's distribution, significantly enhances the effectiveness of knowledge distillation. By leveraging the strong embeddings provided by PE G, our short distillation finetuning schedule significantly boosts the performance of both B and L scale models of Pe (see Appendix C.3).
B.1.2 Zero-Shot Classification and Retrieval
Zero-Shot Evaluation on Images and Videos. We use CLIPBench" for zero-shot classification and retrieval benchmarking. The benchmark datasets and splits are obtained from the original dataset websites or HuggingFace. We extend the CLIPBench zero-shot evaluation to include video datasets such MST and Kinetics, and will release model checkpoints, evaluation code, and scripts for reproducibility.
Prompt Design. For zero-shot image-text and video-text retrieval, we rely solely on the original captions without any additional prompts. In contrast, for zero-shot classification, we utilize task- specific prompts graciously provided by the InternVL [19] authors. All additional prompts will be released.
For example, we employ specific prompts for zero-shot image classification on various ImageNet benchmarks (e.g.. ImageNet val, ImageNet v2) and video classification on Kineties datasets (e.g.. K400, K600, K700).
Zero-Shot Image Classificatin Prompts ImageNet
bad photo of a {c). photo of many (c}. a sculpture of [c). photo of the hard
to see (c]. low resolution photo of the (c). a rendering of a [c). graffiti of a (c).
a bad photo of the (c). a cropped photo of the (c). a tattoo of a (c). the embroidered (c). a photo of a hard to see (c). a bright photo of a (c). a photo of clean (c). a photo of dirty (c). dark photo of the (c). drawing of (c). photo of my (c). the plastic {c}. photo of the cool (c). a close-up photo of {c}. a black and white photo of the (c). a painting
the {c}. painting of (c). a pixelated photo of the (c). a sculpture of the (c). bright photo of the (c}. a cropped photo of a {c}. a plastic (c). a photo of the dirty (c). jpeg corrupted photo of a (c). blurry photo of the (c). photo of the (c). a good photo of the (c). rendering of the (c). a (c) in a video game. a photo of one (c). doodle (c). photo of a (c). the origami (c). the {c) in a video game. a sketch of (c). a doodle of the (c). origami (c). a low resolution photo of (c). the toy {c). a rendition of the (c). a photo of the clean (c). photo of large (c). rendition of [c). photo of nice (c). photo of a weird {c). blurry photo of a (c).
(c). close-up photo of the
cartoon (c}. art of (c}. sketch of the {c). a embroidered (c}. pixelated photo of [c). itap of the (c). a ipeg corrupted photo of the (c). a good photo of (c).
a plushie (c). a photo of the nice (c). a photo of the small (c). photo of the weird (c). the cartoon (c). art of the (c). drawing of the (c). a photo of the large (c). a black and white photo of a (c). the plushie (c). dark photo a (c). itap of a
(c). graffiti the (c). a toy (c). itap of my (c). a photo of cool (c). a photo of
a small (c). a tattoo of the {c).
ftttp:://github.com//AAA/ AI/CLIP benchmark, MIT License
26

---

Zero-Shot Video Classi f cation Prompts Kinetics
photo of {c}. photo of a person {c}. photo of person using {c). photo of a person doing {c}. a photo of person during [c). photo of person performing (c). a photo of a person practicing [c). a video [c). video of a person {c}. a video of a person using {c}. a video of a person doing (c). a video of a person during (c). a video of a person performing (c). video of a person practicing (c). example of {c). example of a person (c). example of person using (c). example of person doing {c). a example of a person during (c). example of person performing (c). a example of a person practicing (c). a demonstration of (c). a demonstration of a person [c}. demonstration of a person using {c). a demonstration of a person doing {c). demonstration of a person during {c}. demonstration of person performing (c). a demonstration of person practicing {c).
Evaluation Method. Several works different input transformations for different datasets when evaluating zero-shot performance (e.g.. [32, 127, 135, 158J). To be as fair as possible, we fol- low [127] in evaluating with two transformations- center crop and non aspect ratio preserving resize ("squash") report the max between the two for all models and all datasets we evaluate. Additionally, ObjectNet has red border around every image to facilitate deduplication, which we remove for evaluation. Finally, we follow [19] in using retrieval reweighting (DSL), applying the softmax score distribution to the similarities used for retrieval:
scores
scores
sof tmax (scores, dim=0)
This slightly improves retrieval for most models, so we do it for all models we evaluate for fairness. Notably, we were able to reproduce the reported numbers for most papers with these techniques, but for cases where we could not, we default to the reported number.
B.2 PE: Language Alignment
We provide details of the MLLM experimental setup in $4. We describe data, model, and training separately.
Data. Our MLLM training contains warmup data and supervised finetuning (SFT) data. Our warmup data is a IM subset image-text pairs of our PE pretraining dataset. For SFT data, we use a diverse data mix consisting of 2.6M unique samples. This dataset is composed of 1.7M visual QAs samples from the Cauldron [64], 0.5M grounded QA pairs from Visual Genome [59], Flickr-Entities [100] and Densely Captioned Images [136), M. image-captioning pairs from COCO [74] and 0.3M text-only samples. This comprehensive data mix allows us to thoroughly assess our model's capabilities in various MLLM tasks.
Model. As described in $D.1, we use a simple vasion-language model architecture where a vision encoder and a pretrained decoder-only L.M are connected by vision projector. For all tables, we use either 1lama3.1-instte as a vision projector. For fair comparison, we use the native resolution for image input. During inference, we evaluate the models on video tasks in zeroshot manner: We concatenate all video frames into a sequence and feed to language model, without seeing video samples during SFT. For all video tasks, we use 8 frames with the same native resolution of height and width. For PE and PE this makes 448 x 448 x input and 32> 32 8 vision tokens.
8B or QwenLM 2.5-instruct 7B as a language model, and 2-layer MLP
Training. MLLM training consists of warmup and supervised finetuning (SFT) stages, In both stages, we freeze vision encoder and train vision projector and LLM. During warmup stage, we use a global batch size of 128 with learning rate of 1 x 10-4 We gradually increase the learning rate from x 1 over 120 steps, and follow a cosine learning rate decay schedule to train total of 8,000 steps. During SFT stage, we use a global batch size 256 with learning rate of 1 x 10-5 Similar to the warmup, we gradually increase the learning rate from steps, and follow a cosine learning rate decay schedule to train a total of 12.5K steps. We truncate text-sequences longer than 2,048 tokens on top the visual tokens. This makes the maximum sequence length to be (num. vision tokens) + 2, 048. With 448 x 448 input resolution and patch size of 14, we set the maximum sequence length to 1, 024 + 2, 048 = 3, 072. To represent bounding boxes on output side for image grounding tasks, we simply use text tokens to represent each bounding box:
1 x 10-6
10-7 to 1 x 10-5 over 300
We excluded multi-ima images samples.
27

---

each coordinate is normalized between 000 and 999, in "[x, y, x, y]" box format for top-left and bottom-right corners (e.g. [012, 122, 633, 782]).
For all baselines, we search for the best intermediate layer features to adapt to LLM. We search over 1-1,-2, -, -, 8, -00, 12, 14, - 16, 18, -20, 40) layers (counting from last) and report the best result in average over OCR/Char/Doccment Q&A, Visual Q&A, Image Captioning and Video Understanding.
B.3
PE: Spatial Alignment
B.3.1
Training Details
Loss Functions. For self-aligning to frozen PE G layer 41 features (Loore), we minimize the negative cosine similarity:
Σ
(S ||Ssoll- ||Tall
L
Leure
Thtok
where S50 denotes the last layer features of the student, Ta1 denotes frozen layer 41 features from PE G, and Too represents the number tokens. Note that we chose 41 fairly arbitrarily (it is layer 40 when written with indexing from 0). Judging by Fig. 4, any layer around 40 should work (and 39 may be slightly better).
For the encouraging locality loss (Troe), we compute the pairwise cosine similarity between a model's own tokens and itself. This forms "spatial correspondence map" for what tokens should be considered similar. We then compute the same for the student, and minimize the difference between the two with MSE loss:
(((Ss))(S5)) ||Ssol|2
(TSAM) (TsAM)(TsAM) ||TsAM||
Lise Σ
Biok
where TsaM denotes the "SAM Mask Logits" constructed in $E.1.2. We also find using
a temperature (t) on the SAM teacher's pairwise cosine similarity term (x) useful: cl(x-1), The full loss is Lispaial Leore + Lioe.
Hyperparameters. Tab. we show the training hyperparameters for spatial alignment, finetuned top of the initial PE G checkpoint. Then in Tab. 16 and Tab. 17, we show the settings for the two teachers and losses. Note that when running the teachers, we run them on the exact same image as the student (same data aug and all). Additionally, because the SAM 2.1 teacher operates resolution of 1024, we upsample the image, generate the mask logits, and then downsample the result. Both teachers are frozen.
config optimizer B2 weight docay
values
config model
values
config
values
LAMB
SAM 2.1.L 1024 (interp- 4445)
model
Phaen
(0.5, 0.95)
41
0.05
resolution
resolation
448
leaming
Se-4
batch size
12,288
loss
Eg
loss
Eq.
wamn-up stops
weight
weight
mmining
24K
tempenttur
20
data quitity
5. (PFaee PT Data)
Table 17: PE G Teacher.
samples
300M
points pred iou threshold
22x33211000
0
resolution
448
stabiliity threshold musk threshold
musk raño mask size
0.75
tokens
Table 16: SAM 2.1 Teacher.
druppath layesscale
0.1
aspect jitter ar (0. color jimter (00.99.0,0.32
ang
hilip (G..5)
Table 15: Spatial Alignment.
28

---

B.3.2 Visualization Method
To visualize the features in Fig. 10 and Fig. 20, our goal is to map 1536-dimensional space down to 3 dimensions to view how the model encodes each token in relation to each other. One naive approach would be apply PCA with 3 dimensions across all token the image. However, we find this alone can be misleading.
Specifically, if the model has rich semanties, it should be the case that most of those 1536 features have some useful information in them. Some of that information could be spatially contiguous, some of it not. We want PCA only select the spatially contiguous information, since we are trying to evaluate the spatial quality of the features. However, naively applying PCA will not necessarily do that, especially for models with information aggregated in "global tokens" ($E. 1.1). Despite these tokens carrying important information, they are not spatially contiguous. Thus, if PCA dedicates large portion of its 3 dimensions to global tokens, the features will look like their spatial quality is bad, despite the features containing good spatial information.
So, how do select for only the spatially contiguous information visualize? The answer is simple: by definition, the spatially contiguous information will be... spatially contiguous. To keep the spatially contiguous information while lowering the impact of the global tokens, can simply apply a low pass filter to the features (specifically, a gaussian blur with kernel size and a of 1). To retain the detail of the original features, we can average the two together. Thus, visualize features, we use the 3D PCA of the of the following. x denotes the model's 's output features, and g(r) denotes gaussian blur.
0.5x + 0.5g k = 3,, 3,
We show the impact of this in Fig. 13. Blurring the features make them appear more detailed! In reality, that information was always there, just PCA did not show it. Thus, great care must be taken when visualizing high dimensional feature spaces. If they were easy to map to dimensions-yoou would not need 1536 of them!
PCA Row
wi Low Poss Filter
Figure 13: Feature Visualization Ablation. With raw features (top row), PCA misses spatially contiguous parts of the feature space and instead focuses on global tokens (which earry information but are not spatially coherent). By applying simple low pass filter (bottom row), we reveal spatial information that PCA originally missed (see column 2: with raw features, the background looks like mess, with the low pass filter the tiles become visible).
Then, to map the PCA dimensions to RBG pixel values, we map each PCA component corre- sponding channel in LCh color space, then convert those LCh colors to RGB to get the final image. Note that we use LCh instead of RGB directly for aesthetic reasons, and also because LCh is cylindrical color space-wheer smooth changes to the values look like smooth changes in colors to humans-and thus easier to discern.
B.3.3 Frozen Feature Dense Prediction
We discuss the detailed settings of the results for dense prediction with frozen features in Tab, 6. Each model is evaluated with its native resolution up to 448 or 448 (whichever is optimal).
Zero-Shot Tracking. We evaluate our pretrained models on label propagation task using the protocols in [51, 104] on DAVIS dataset [101]. This evaluation does not require any finetuning therefore preserves the spatial features the model. Following Toto [104], we use the features from the last = 7 frames to find the nearest neighbor patch in the current frame, and then propagate the masks from the previous frames to the current frame. Note that this evaluation method does not require any training.
probing,
29

---

Semantic Segmentation. For semantic segmentation. we evaluate our pretrained models on ADE20K [164] semantic segmentation task. We use batch norm and a linear layer to map in- termediate spatial features to segmentation masks following [96]. The models are evaluated and then features are resized to 518 x 518. We only use features from single layer. The probing layers are finetuned with Adam W [81] with a learning rate of 0.001.
Depth Estimation. For depth estimation on NYUv2 [120], we follow [73, 96]. We use a DPT- head [106] on top of our frozen pretrained model and use only single layer features. We scale the size of the DPT-head for each models based on the hidden size for each architecture. Because NYU is small dataset and the models we evaluate are large, we observe the results for most models are noisy and prone to overfitting. Thus, for fair comparison we train all models for 20 epochs and for all models take the lowest validation loss over all epochs.
Frozen Detection. For the frozen feature detection results presented in $3, we evaluated using Mask R-CNN [42] as probe. We used resolution of 1024 for Fig. and 768 for the remainining experiments 83. Because the backbones were frozen, we did not add any global attention and instead simply tiled the input image with window size of 32 for the 1024px experiments and 24 for the 768px experiments. All models were interpolated to patch 16. Finally, the backbones were frozen and only the FPN and R-CNN heads trained for 15 epochs COCO with stepwise decay LR without drop path.
B.3.4 End-to-End Finetuning Detection and Segmentation
We provide a detailed discussion of settings of end-tet and finetuning on detection and segmentation presented in Tab. 7 using Detectron2 [147]. The hyperparameters can be found in Tab. 18. We find that the default 100-epoch protocol in ViTDet [71, 146] causes overfitting problems in COCO experiments especially for bellion-level parameter vision encoders, we tune the training epochs, learning rate, drop path and learning rate decay accordingly.
The LVIS experiment setting is the same as COCO except all L-size models use learning rate of 2e-4 and all g-size and G-size models use 75 epochs.
confie
vollaes Adamn
modkll
Ir
opachs
drop path
decay
layers
plohil window index
window size
optimizer
MetaCLIP.G
5e-5
75
01.
0.9
48
(11, 25, 35, 47)
optimizer
IP2-40
1o-4
0.4
weight decay
0.1
Sigl 1P3-
Se-5
75
0.5
0.9
40
(9, 19, 29, 39)
14
leaming
Step wise decay 64
DINOv2
(5. 11,
batch size
DINOV2
5e-5
36
0.5
0.9
40
(9. 19, 39)
32
image
1034x 1024
PRomn
0.5
0.9
(12, 24,
sumentation
LSI jo.1, 2.01 abswin [7]
PR G
56-5
36
0.5
0.9
50
[12, 24,
32
postional extedddnng
patch size
16
Table 18: Settings for End-to-End Finetuning Detection and Segmentation.
B.3.5 System-Level Comparison on Detection
We describe our implementation for system-leved compar- ison to the ofate-of-the-art on COCO object detection in Tab 8. Our implementation is based on the DETA reposi- tory. We replace the vision encoder with our PE and maintain the same hyperparameters as in the end-to-end finetuning settings, while keeping the detector unchanged. The training process consists of three stages:
Tese-Time Time Aug
APhos
No TTA
65.2
SofNNM [6]
63.8
Hlpp Aug.
Multisenie Aug
66.0
Table 19: Test-Time Aug for system- level comparison on COCO in Tab. 8.
Initial Training: Train on Objects365 for 12 epochs with an image resolution of 1024 x 1024, total batch size of 256, and learning rate of 2e-4, which is divided by 10 at the 10th epoch.
http:////htthacc//cccceeetet//ee Apache 2.0
2.0
30

---

2. Increasing Resolution: Continue training on Objects365 for 6 epochs with resolution of 1536 x 1536, total batch size of 128, and learning rate of 5e-5, which is divided by 10 the 5th epoch.
3. Finetuning: Finetune on COCO dataset for 12 epochs with an image resolution of 1728 x 1728, total batch size of 64, and learning rate of 5e-5, which is divided by 10 at the 8th epoch.
4. Further Increasing Resolution: Further finetune on COCO dataset for epochs with resolution of 1824 x 1824, total batch size of 64. To save GPU memory, use SGD optimizer instead of Adam, with a learning rate of 5e-3, which divided by 10 at the 2th epoch.
We apply series of test-time augmentations to further improve the performance, see Tab. 19.
31

---

C Additional PEcore Results
C.1 Robust Image Pretraining
In Tab. 20, we present the raw data for the robustness metrics in Fig. 2. Across the board, each change improved almost all metrics (with the exception of progressive resolution slightly hurting the average and mask regularization slightly hurting ImageNet Adversarial). The fact that there were no tradeoffs to these changes, indicate that their improvements to the features are general. This could be why most of these changes improved performance for downstream tasks as well.
Note that in $2.1, we only discuss changes that we know to work. There are several changes that we have tried that do not work (i.e., do improve performance or lower performance). For instance: average pooling instead of using a class token, increasing the text tower size, using hue or contrast jitter, and maintaining the same resolution throughout training but dropping tokens instead of progressive resolution (FLIP-style).
We also find increasing batch size and increasing training iterations for an L scale model to have equivalent effects. This is in contrast to the batch size scaling observed by [158], but it is possible that this difference is down hyperparameter issue.
Zoro Shot Classifictiin
ImaggoNel pot
ImaggNNt
ImagoNet
ObjestNo
CCCunn (4)
Imagsshat
ImugeNar
Storch (1400)
125
Step
78.9
Progresive Resolation
75.1
78.9
71.8
72.4
69.9
90.5
67.0
High Batch Size
76.2
79.5
72.8
74.1
71.8
91.0
68.1
LAMB 1R
Hight Resolation (336)
78.3
80.4
73.8
75.6
79.2
92.0
68.8
2D RoPE
79.2
80.7
74.1
77.4
80.9
92.7
69.4
Attentoon Puoling
Dats Augmenttaion
80.8
81.1
75.2
80.8
83.1
93.5
71.2
Mask Regulatization
80.9
81.3
75.3
80.9
82.8
93.8
712
Table 20: Robust Image Pretraining Full Results. Raw results for the robustness metrics metrics in
Fig. 2. Almost every change improves every metric, but some metrics are improved more than others (e.g., ObjectNet and ImageNet-A).
Scaling Behavior. In Fig. 14, we show the performance of our recipe (Fig. 2.9) vs. the original CLIP recipe (Fig. 2.1) across S/14, B/14, and I/14 models. For each benchmark, our recipe scales around the same rate or better than the original CLIP recipe. On some difficult datasets like ObjectNet [4] and ImageNet Adversarial [46], our recipe shows distinctly better scaling. This indicates that the improvements performance were not at the cost of scalability, meaning we can further benefit from scaling the model size.
Imegehert.cz
Imageflet
e5
50
Dass
Dan
Oom
Das
65
Figure 14: Scaling Behavior (Model Size). Results before and after recipe changes (Fig. for S/14, B/14, and IJ14 models. Our recipe improves scaling for difficult metrics like ObjectNet [4] and ImageNet Adeversarial [46].
In Fig. 15, we additionally show the performance of our recipe vs. the original CLIP recipe across L/14 models trained with 120K steps (one-third schedule), 240K steps (two-thirds schedule), and 360K steps (full ablation schedule). All models are their own training runs with full learning rate annealing and the progressive resolution schedule adjusted proportionally. We see nearly linear trends for our recipe on most datasets. This suggests we can train longer for more performance, even scale and with 24B samples seen already.
32

---

Imagetiet Va!
Imagefirt op
Objecthlet
Imagellet Aiterrona
Imagelaet Rocihhoos
Image Nee Sketch
100c
300c
200c
b00k
240
1ock
1000
75
29
Vania
Varila
1/2
3
0.02
1.22
Figure 15: Scaling Behavior (Training Steps), Results before and after our recipe changes for an L/14 model trained with 120K, 240K, and 360K steps, adjusting the learning rate and progressive resolution schedules accordingly. Despite our recipe being much stronger than the original, there is still room for further improvement by training longer.
C.2 Additional Video Ablations
Human Refined Data for Captioning. In Tab. 21, we ablate the captioning performance of our video captioner component in our video data engine with or without human-refined video captioning data. For all benchmarks tested, the human-refined data significantly improves captioning performance.
AuroraCaa [12]
VCG Diverse [85]
VCG [84]
Captioner
Ncove
Ace
Noowe
PLM
2.2
51.9
65.1
34,3
71.1
Table 21: Video Captioning, Adding human--effned data greatly boosts captioning performance.
Video Scaling Behavior. In Fig. 16, we investigate the impact of scaling recaptioned video data on later checkpoint of the same image-only model as in Fig. 1. Notably, scaling synthetic video data demonstrates consistent improvement in both image and video benchmarks. Full results of this scaling experiment can be found in the Appendix 13.
In the top row, scaling synthetic video data consistently improves performance on image benchmarks, with monotonic improvements of +1.1% in ObjectNet and +1.6% in ImageNet Adversarial. ImageNet val and ImageNet v2 have smaller gains, with accuracy increases of 0.3% to 0.5%, plateauing at 7MM samples. We also observe significant boost to zero-shot retrieval (here, COCO [74]) of +3.8% to +4.1% % top-1 recall.
The video tasks listed in the bottom row demonstrate a consistent story. We observe a significant jump in performance between none and 3M videos across all video classification tasks, indicating that there is a domain gap for image-only models that hinders their ability to perform well on video out of the box. Further scaling synthetic video data leads to substantial performance gains in both video classification and retrieval. Video classification accuracy improves consistently by +5.6% to +11.7 1.77 without plateauing, while video retrieval shows significant improvements of +7.7 to + +15.3 top-1 recall.
Classification
InageRee
Imagaket
Objectket
Imagatie Adveecatia
0OCO Tor--hraa
CoCO mmage-
65
66
52
75
66
66
17M
17M
17M
Video Classification
Video Retrieval
214
1754
5M
17M
2M
175
170
1754
Figure 16: Video Data Scaling, Finetuning on videos recaptioned by the PE video data engine from OM (baseline image-only model) to 17M samples consistently improves both image and video performance, both classification and retrieval.
33

---

These experiments highlight the quality of our video data engine and its ability to significantly improve encoder performance, even with only a relatively modest 17M videos compared to the billions of images seen during pretraining. Our video data engine is a vital component in build strong, unified image-video encoder.
Video Scaling Behavior Detailed Results. The detailed video data scaling results are presented in Tab. 22. Our experiments demonstrate that increasing the number of synthetic video data generated by the proposed video data engine enhances the performance of classification and retrieval on both image and video benchmarks. On image benchmarks, while improvements on ImageNet val and v2 plateaued earlier compared to ObjectNet and ImageNet Adversarial, MS-COCO retrieval performance continued to show gains. On video benchmarks, scaling synthetic video data consistently yields better performance for both classification and retrieval tasks. We expect that further scaling up the video data with our video data engine will continue to drive performance improvements.
Vi0co Data Size
Anangg Imogee
Image Shro-Shot
Asamge Video
Video ZeniShot
TmaggeNet
(25)
ImageeNet
22 1009
ObjectNex
MSS- MS-CC00
COC-
Kinelis
300 [54]
Kineics
6:0
Kinellss
700
34)
UCF
HMDB 51
2011
MSR VTI
MSSR-V
0M
77.0
83.9
78.6
86.5
90.3
52.1
70.3
57.0
20.3
69.4
61.6
78.5
47.4
40.5
31.4
3M
71.7
78.8
53.3
61.6
72,4
12.2
53.8
37.6
6M
78.0
84.2
79.0
86.7
91.1
$4.0
72.7
63.6
73.5
73.4
66.0
88.9
51.6
44.9
43.6
78.4
45.5
11M
78.6
84.2
79.2
87.2
91.8
55.4
73.8
65.2
75.1
75.0
67.6
89.7
55.6
47.7
45.8
14M
78.8
67.9
55.8
46.3
17M
78.9
84.2
79.2
87.7
92.0
55.8
74.3
65.8
75.7
15.5
68.2
90.2
56.0
48.3
46.7
Table 22: Scaling Video Data. Increasing the number of synthetic video data generated by our
proposed video data engine consistently enhances the performance of image and video classification and retrieval tasks.
C.3 Smaller Models
Ablation: Distillation Temperature. To optimize the performance of smaller models and L-scales in Tab. 2), we utilize a distillation finetuning approach with PE G as the teacher model. During this process, both student and teacher models encode image and text inputs to compute image-to-text and eext-to-image similarity distributions, similar to CLIP training [103]. The student's is distributions are then optimized to match those of the teacher by minimizing KL-divergence loss on both image-to-text and text-to- extttt--image similarity distributions.
We find that using fixed and smaller temperature (i.e., higher logit scale), which controls the range of logits in the softmax, significantly enhances the effectiveness of distillation. This results a sharper distribution for the teacher's distributions. In contrast, the student's temperature remains learnable, consistent with our pretraining procedure and CLIP training.
In Tab. 23, we present ablation study examining the impact of temperature the teacher's distribution. For this analysis, we utilize a pretrained vanilla CLIP model (ViT-B/14, resolution 224), which serves baseline for comparison (see $2.1 for details). The models are finetuned using
Teacher's Temp
Shot Cluriffiation
ImageNet
ImageNet
OlgestNer
Chasser 40
ImngeNee
ImageNet
Bosiition 45
ImsgeNet
(M))
Model
3gg
vanilla pretrained model
66.2
74,2
67.4
62.5
50.2
83.0
59.8
x2
65.2
71.8
65.5
61.4
50.
83.6
58.6
distillation
x1
68.0
76.9
68.1
61.7
54.1
85.3
61.1
65.
85.1
x0.5
68.3
75.2
68.2
65.3
542
85.2
61.4
Table 23: Ablation Study on Teacher's Distribution Temperature. We evaluate the effect of varying temperatures on the teacher's distribution, using pretrained vanilla CLIP model (VIT-B/14, resolution 224) as a baseline (details $2.1). The models are finetuned via distillation with short schedule of 50K steps.
34

---

distillation with a concise schedule of 50K steps. Notably, our results show that employing smaller temperature for the teacher's distributions yields improved performance on zero-shot ImageNet benchmarks.
Building strong smaller models. In Tab. 24, we demonstrate our step-by-step training strategy for building strong smaller models at the L scale, as discussed in $2.3. Specifically, we outline our approach to image pretraining, image distillation, and video finetuning, and distillation. Leveraging the robust foundation established by our pretraining techniques ($2.1), we show that distilling from Go our strongest unified perception encoder, yields improvements on both image and video benchmarks. Furthermore, a short-scheduled video finetuning provides an additional boost in performance on both benchmarks.
image Sert-Shitt
Video Zon--Shot
Vidoo
Imagehhet
Imagge
1089
ObjectNet
Chaner H4]
MSS-CDCC
-o0 [P]]
MSSO
[74)
Kinetiss
(4)
Kinetcs
60015-
Kineles
20 24
UCF 101
HMDD St
MARC VTT
(011
Model
Stage
1P2-112211116
76.0
84.4
84.3
553
65.3
56.8
41.5
31.4
P.
image pretraining
75.1
82.9
76.8
81.8
85.6
53.0
70.4
59.0
68.0
67.7
58.5
85.5
57.7
42.0
33.4
P.ccoo.
eimage distill from P
77.6
83.6
78.1
84.4
88.9
560
74.7
64.5
73.0
72.6
64.8
86.5
58.0
47.9
48.4
P..vel.
exdonn finetunin
83.5
73.9
73.4
65.3
Table 24: Building Strong Smaller Models. This table illustrates the step-by- process of
developing the LL 336px model, as outlined in $2.3. Starting with the pretrained L both image distillation, along with video finetuning, enhance performance across image and video benchmarks, resulting in a unified L-scale model.
C.4 Additional Results
Additional Zero-Shot Benchmarks. We further evaluate PE on additional set of zero-shot classification and retrieval benchmarks we construct in Tab. 25 to address key gaps in common bench- marks. For comparison, we also evaluate SigLIP2 [135] and Intern [19] on these benchmarks.
First, we note that the version of ObjectNet [4] that is standard to benchmark robustness (e.g., in Tab. 3) is not the full set. ObjectNet consists of 313 classes of objects in challenging and uncommon orientations, locations, and viewpoints. However, the standard version used for benchmarking is a 113 class subset of classes that overlap with ImageNet-1K 1k [25]. Naturally, benchmarking in this way rewards performing well on ImageNet classes over generality. To remove this bias, we construct the full ObjectNet set with all classes and compare the reduced ObjectNet set in Tab. 25. Surprisingly, we find that while P performs +7.6% over Intern VL-C and only +0.2% over SigL.I22---opt on the reduced ObjectNet set. performs +1 1.8% over Intern VL-C and +0.9% over SigLIP2-g-op on the full set of classes, highlighting PE's generality.
Next, we include iNaturalist [137] as zero-shot benchmark because of its level of specificity with 2, 101 fine-grained long-tail classes. PE G outperforms the next best SigLIP2-g-on model by +9.6%, emphasizing PE's long tail knowledge. We then evaluate PE's cultural diversity Dollar
Model
Enooder Pumans
Resalution
ObjectNeii 14
Zero Shot Classifcction
Objecthär
H
Anl Giaor (522
Nararaaa
2017 (377)
Doller St
S4 109]
CepCCps
Xb0 (1:9)
Zero Shot Retrieval
Flip
VSS 7M 1199
PVD Bonch
Dan
SigLIP2 [135]
0.18
224
108
73.6
59.1
16.9
$5.9
72.0
69.8
53,9
60.1
PE
0.18
5.48
71.9
72.3
0.38
384
108
$4.4
73.2
26.7
57.6
78.0
76.2
61.9
PFum
0.38
InteVV-C1191
5,38
67.2
72.3
67.8
63.4
1.18
384
108
$5.D
78.1
31.5
59.3
78.8
76.9
62.5
67.1
PFomeG
1.98
448
5.48
88.2
79.0
41.1
62.3
78.8
78.7
77.0
76.6
Table 25: Additional Zero-Shot Results. We present several additional zero-shot benchmarks from
existing datasets and our own PVD ($A.1) to address evaluation gaps left by standard benchmarks.
35

---

Street [1]0], which consists of images of under- repressnepee populations. We use the version provided by [38] and re-evaluate models ensure a fair comparison. Here too we find PE G outperform existing methods, with +3.0% over SigI.IP2-g-opt Further, we test OCR performance by setting up TextCaps [119] as a retrieval dataset. Notably, Pe performs on par or better than SigL.P, which known for good OCR performance. This is potentially surprising, as the horizontal flip augmentation we used during robust pretraining ($2.1) is typically thought to hurt OCR performance. However, instead it seems to have given PEgore the ability to read backwards: we test the same TextCaps retrieval but with images horizontally flipped. Other models suffer from this, but PE performance only drops by 0.1%. Finally, we evaluate PE G on the PVD benchmark ($A.1), challenging video retrieval task on 15K diverse and human-refined videos. Here, PE G significantly outperforms Intern VL [19] by +13.6% on text and +9.5% SigLIP2 [135] tideo--text.
Frozen Encoder Probing Results. To compare against models that are not capable of zero-shot classification, additionally evaluate PEçore using k nearest neighbors (following [96]). linear probing (following [19]), and attention probing (following [36]) on top of the ImageNet- 1k [25] train set. We present these results in Tab. 26 and compare other encoders using their reported numbers. In every case, P G outperforms all existing open encoders, including those with significantly more parameters.
Encnder Params
Eactder Prohing
ImugeNel (25)
Resolrtin
ImageNel
IrogeNor
Amnmtto
Dat
R0N
DINDW2-2 [96]
I.IR
224
145M
83.5
86.5
87.21
R-DIOv2.5gg44)
L.1
518
85.3
AIMv2 [3]]
2.78
448
7.20
89.5
EVA 188 [127]]
PEg
1.90
448
5.40
96.8
89.5
89.8
Table 26: Encoder Probing Results. We evaluate PE G's frozen features using the typical probing methods to compare to models without zero-shot support. "from [36].
36

---

C.5 Additional Layerwise Scaling Analysis
In the main paper, we explored the scalability of our pretrianing recipe v.s. the original CLIP recipe. However, we only analyzed it there for single spatial task. To see whether the trend is consistent, we repeat this scaling analysis on wide variety of downstream language modeling tasks using the same frozen evaluation setup as Fig. 4 and report the results in Fig. 17. Surprisingly, the simple change pretraining recipe improves scaling for most language tasks as well-including output-side grounding (RefCOCO). Note that in this benchmarking setup, the LLM never sees videos during training so the Video Q&A per-layer results are noisy. Yet, the best layer trend is still the same.
OCR Q&A
Best Layer Scaling
64
Vanilla
Accurecy legefa
Aocuracy 4f
50
50
52
90
99
Curs Varilla
0%
33%
67%
100%
0%
33%
67%
100%
0.18
Frozen Enooder Depth
Prezen Enooder Depth
Model Par srrmmteers
Visual Q&A
Best Layer Scaling
Vanilla
Accuraty Augefa
Accunay of
65
65
63
53
Ours
42
Verlla
0%
33%
67%
100%
0%
33%
67%
100%
Encoder Depti
Encoder Depth
Parsmeteree
Captioning
Layer Scaling
120
Vanilla
Ours
0DE: Hugef3
CID Angors
100
62
62
Ours
35
35
Varilla
0%
33%
67%
100%
0%
33%
67%
100%
Encoder
Encoder
Parametera
Grounding
Best Layer Scaling
70
70
ho/cocolgi- Fgo
Vanilla
Ours
Retcocolgg+ PgG5
37
37
Our
20
20
Varilla
0%
33%
67%
100%
0%
33%
67%
100%
Encoder
Paremetera
Video Q&A
Layer Scaling
Accurecy Argof12
Vanilla
Ours
Accuracy of
49
41
41
Ours Varlla
34
34
0%
33%
67%
100%
0%
33%
67%
100%
0.18 anneters
1.09
Figure 17: Further Scalability Analysis. We repeat the analysis from Fig. 7 on wide range of downstream tasks by adapting to language model. Each category an average of several downstream tasks (see $4).
37

---

D Additional PEtang Results
D.1 Alignment Method Derivation
Here we discuss in detail the derivation of the aligmment approach discussed in $4. We design the alignment approach not just keeping in midn the best performance, but also ensuring that the resulting aligned model is general. That is, we want the aligned model to be transferrable to as many settings as possible regardless of resolution, decoder, or training setup.
MLLM Evaluation Tasks. In this section, our main testbed is to adapt vision encoders to MLLMs and test various MLLM tasks. We evaluate the downstream performance of each MLLM across five task categories: (1) OCR, Chart, Document Q&A on ChartQA [162], DOe [89], In- foQQA [90] and A12D [56]; (2) Visual Q&A on TextVQA [122], OK-VQA [115], POPE [72], and VQAv2 [39]; (3) Captioning on Flicker [155], COCO [74], and No Cap [T]: (4) Video Under-
standing on VideOMME [37], STAR [145), TGIF-QA [52], EgoSchema [87], MVBench [67], and
Perception [102]; and finally (5) Grounding on RefCOCO [55].
To arrive the optimal alignment recipe, first con- duct ablation studies using a 20M subset of the data. In Tab. 27, we ablate the LLM sizes, training param- eters, vision projector types, output layers to project, and encoder regularization. We evaluate across OCR Q&A, Captioning, Visual Q&A, and Video Q&A and find the best configuration.
Repuheization?
LLM unfrrcom
LLM scale
Projecoo
Arrage ofe
Captioning
Aurgge of
Visual QR&
dumog ord
Aarrg oro
Layer
Arg.
I.M Seuu
18
MLP
76.5
60.7
115.1
76.0
54.0
3B
MLP
47
78.1
65.9
115.7
76.6
54.1
3B
MLP
78.4
65.8
117.6
76.3
53.7
LLM Setup. We explore different scales (IB or 3B parameters) and freezing weights of the LLM. We observe that going from IB to 3B parameters increases average score by 1.6 points (76.5-78.1).. Unfreezing the LLM boosts this number to 78.4.
Viston Projector
Linear
77.2
114.1
76.5
3B
MLP
47
78.1
65.9
115.7
76.6
54.1
PE Outpar Lover Loyer
3B
MLP
50
75,9
56.6
116.7
76.5
53.7
MI.
78.1
65.9
115.7
76.6
3D
MLP
41
76.9
65.5
112.8
75.4
53.9
PF Regulorisation
Vision Projector. Using 2-layer MLP vision pro- jector instead of a linear layer improves the average score from 77.2 to 78.1, while only adding few pa- rameters (13.5M + 27M).
MLP
79.9
117.5
73.4
3B
MLP
47
80.1
68.7
118.3
71.0
56.3
Table 27: Language Alignment. We find the best configuration to language align PF G using autoregressive language training.
PE Output Layer. As shown in $3, G has intermediate layers that perform significantly better than the last layer when used as features for certain tasks. However, it is not clear if that same behavior applies when finetuning. We test applying the projector to layers 41, 47, and 50 (the last layer), and find that layer 47 works best. Incidentally, this is also the optimal layer for frozen VQ&A in Fig. 4.
PE Regularization. We apply LayerScale [132] and DropPath [49] the vision encoder during the alignment, for stabilizing training. This improves the 78.1 average score to 79.9 (+1.8 points). Unfreezing the L.MM boosts this number further 80. We choose this configuration (last row) as our final alignment setup.
To construct PEtang scale this recipe up the 70M samples covering natural images, docu- ments/Chartssdiagraams. and videos, perform alignment as described, and extract the resulting vision encoder. Compared to the 20M sample ablation setting in Tab. 27, the final PEgang trained on 70M total samples gives another +2.1 points to 82.2 on the average across OCR Q&A, Captioning, Visual Q&A, and Video Q&A.
D.2 Layer Analysis Details
In Tab. 28, we present the raw numbers for the layer analysis plots in Fig. 8. Note that layer analysis was not performed exhaustively on every layer. Additionally, PFang removes the last four layers of the model.
38

---

OCR V04
Captooinn Natural VQA Grownding
P5au
PEaan
PRooor
Layer
50
48.6
114.0
74.1
39.1
49
48
56.3
114.8
74.8
55.6
47
57.3
72.4
113.9
120.1
75.3
78.1
$7.8
71.2
46
72.2
120.0
75.1
78.2
70.8
45
72.0
114.6
119.7
75.1
77.7
70.7
43
59.9
71.5
113.8
117.7
74.8
77.4
63.1
70.5
42
71.3
116.6
73.8
77.2
64.1
70.8
115.4
76.7
66.6
71.22
40
61.4
69.8
112.0
115.0
74.0
76.3
66.5
71.0
39
70.0
113.5
74.0
75.6
67.1
71.1
38
69.3
75.5
70.3
36
61.3
67.9
108.5
109.4
73.0
74.0
67.8
70.4
34
104.7
70.0
72.2
69.6
33
61.1
100.1
102.7
71.3
65.9
68.9
32
54.2
61.
96.9
100.1
67.8
70.6
65.5
68.7
31
60.7
96.7
68.7
67.3
30.3
71.3
52.0
51.6
11
28.8
9
47.4
99.0
47.0
49.2
30.9
43.
28.7
42.8
43.
272
Table 28: Raw Language Layer Analysis Results. The raw values for the plots in Fig. 8.
D.3 Unfrozen Encoder Results
In our standard MLLM evaluation, we always freeze the vision encoder when tuning the LLM for downstream MLLM tasks. This is to ensure that we test the quality of each vision encoder without any bias from our finetuning setup. However, this introduces a lingering question of whether unlocking the encoder during LLM finetuning would eliminate any lead PE has over the other models.
Thus, in this section, we repeat the same MLLM evaluations as the main paper but with the encoder unfrozen. Each experiment uses 1024 tokens per image. In Tab. 29, we show the unfrozen encoder results compared to AIMv2 3B [36] and SiglIP2 g-opt [135]. It seems all models, including Pa G, benefit from unlocking the encoder. However, Pfauang G still outperforms the other models overall, often by a significant margin.
We perform similar evaluation in Tab. 30, this time comparing across PElang model scales and to the original PE And here see that both PEaang models significantly outperform the PEecoe ones in this unfrozen setup, especially for the larger G size. Thus, it seems that a language alignment step is still necessary even when the encoder is unfrozen during MLLM construction.
OCR/ Cearthhtt//e / Dac. Q&A
Wanaf Q44
Captioning
Neto (83)
Vidro
Arg. OCR @a
CharQA
(00)
Info QA
AZD
199)
Arg. V24
OK- VOA
Aec [15]
POPE
VQAx3
Arc (9))
Ang. Caps
Flicker
COCO
C1DE [4]]
Oip
CE05
Arg. Ground
Ang. Video
VileaMME
(37)
STAR
Arc (145)
TOIF QA
EgoSehemm
MVBoaeh
(07)
Pooeqtion
Model
ATMv2 [16]
65.0
77.4
68.8
74.4
76.9
73.1
64.4
85.4
81.7
119.2 97.1
139.7
69.7
54.3
46.8
51.2
64.5
48.9
SigLIP2-g-opt (135]
65.0
77.5
68.5
39.6
74.6
78.8
74.9
67.7
89.5
N3.2
120.0
97.9
140.7
121.5
73.4
54.4
43.8
52.7
66.8
59.0
51.2
52.9
82.9
141.7
70.4
50.9
Table 29: MILM Results with Encoder Unfrozen (PEtang vs. Others). Same setting as Tab. 5
using 1024 tokens per image, but with the vision encoder unfrozen during LLM finetuning.
OCR/Chart / Dan. Q&A
Vinaf Q4A
Captioning
DC00 (83)
Vidro
Ang. OCR Qa
CharQA
DonVQA
Arc. (80)
QA
AED
190)
Ang. YBA
RodVQA
VQA
Arc. (155)
POPE
VQAVO
Aoc (39)
Ang: Caps
Flicker
(15g
COCO
CI0G [d]
No Cap
Groand,
Ang Vades
VilesMME
Arc. 187)
TOIF-QA
Schema
Arc (97)
MVBearch
Noc (0)
13 PonegtionTes
Model
Arc
P.
64.4
70.6
7088
75.8
64.5
80.3
115.9
136.8
70.9
53.3
65.4
53.4
58
P
71.2
81.2
82.5
45.6
75.3
76.6
71.7
64.4
89.2
80.9
114.5
85.1
138.7
119.7
73,0
$5.5
47.1
55.1
68.4
58.4
49.9
$4.3
G
62.8
73.6
67.2
38.5
71.7
76.0
68.9
65.6
88.4
80.9
117,2
94.0
137.9
119.7
69.2
55.0
44.2
56.0
65.3
60.6
505
52.5
G
72,8
81.6
84.7
48.1
768
784
74.1
66.9
89.6
82.9
120.8
97.4
141.7
123.4
73.4
57.9
48.0
58.9
70.4
63.0
50.9
56.2
Table 30:
MILMM Results with Encoder Unfrozen (Core vs. Lang). Same setting as Tab. using
1024 tokens per image, but with the vision encoder unfrozen during LLM finetuning.
39

---

D.4 Additional Results
Here we provide additional comparisons of PEcore and PEiin with other vision encoders that are popular choices in MLLM literature: MetaCLIP [150]. SigLIP2 [135], CLIP [103], AIMv2 [36], DINOv2 [96], and Intern ViT2.5 [18]. Overall. these encoders span several different pretraining losses (e.g., contrastive, captioning, self-superv,s,d, and mixed supervision), encoder sizes (from 300M to 6B parameters), and resolutions (from 224 to 512). For all vision encoders, we find the best intermediate layers to train MLLM for fair comparison (more details in Appendix B.2).
Main Results. In Tab. 5, we showed benchmarks results for native resolution input across existing encoders, Peecre and Pflang- Here we provide additional comments about those results. Notably, AIMv2 [36], Intern ViT2.5 [18]. SigLIP2 [135] and Phnngg are trained jointly with a language decoder using next token prediction objective, and thus they perform better overall compared to the base contrastive and self-supervised models across all the metrics. However, PEinng uses a fraction of the training FLOPs for language alignment tuning, while significantly outperforming all vision encoders by large margin (an average +3.5 points for and +2.0 points for L).
In Tab. 31, we compare PEçore and PE with dynamic resolution setting [75, 80]. More specifically, we use up to 4 tiles, following after a thumbnail, which is a whole image resized into 448 448. With the maximum number of tiles of 4, the model can cover {1 x 1, 1 x 2, 1 x 3, 1 x 4, 2 x 1, x 2,3 x 1, 1} tile ratios. Similar to the Tab. 5, we show that PE largely outperforms the baseline vision encoders by large margins across all categories of MLLM tasks. Note that PEjing has been alignment-tunee with native resolution input, as opposed to e.g., Intern 2.5, which has been midtrained with dynamic tiling, which shows P[ing strong generality for different input formats.
Transferability. As PE is aligned with Llama 3.2-instruct 3B, we conduct a separate set of experiments to check if our model performs well with a different base LLM. In Tab. 32 we repeat the native resolution comparison with QwenLM 2.5 7B [153]. Interestingly, PEiagg not only outperforms all vision encoders in this setting, but it also outperforms Intern [18]. which is specifically aligned to QwenLM 2 [152] throughout midtraining. fact, PEnang with QwenLM even improves its performance with Llama in some cases like with OCR Q&A and video benchmarks, emphasizing the generality of our language alignment.
Grounding Breakdown. Next, in Tab. 33, 34, 35, we show full RefCOCO/+/g [55] results across all setups. Overall, PE L or show the best performance across all RefCOCO splits, except with Qwen2.5 LM. This is because (1) Intern ViT 2.5 6B is midtrained with Qwen2 LM, and (2) during te/midd-triinng the training data of Re/COCO/+/g are seen.
40

---

OCR/Chart Doart/ Quit
Visual (244
Caprioning
BuC0 NocoCo 195)
Video
Arg. OCR QA
Avg. Ground
Encoder
Ressldion
ClarlQA
Do4VQA
An [Ng
Irfo. QA
An. (98)
Arg: VQ4
TesVQA
OK--VQA
Acc (1:5)
POPE
VOAv2
Avi 199
Arg. Cops
Flicker
(88)
COCO
CD0 (74)
No Cap
croer
Arg. Viduo
VideoMME
Ar. (37)
STAR
An. (145)
TGIF-DA
An 153)
EgeS
(80)
MV RVBohh
An. (60)
Perception
Model
255 Tokens per Tile
MesCLIP-P L [1:50]
0.30
224/14
61.8
71.1
62.5
40.2
73.3
74.6
65.3
64.9
89.5
79.8
113.4
90.4
133.5
116.2
67.1
48.0
44.8
47.1
62.7
39.0
46.0
48.3
MeaCLIP-G [1500
24
60.3
134.4
46.5
44.7
PFin G
1,78"
224/14
70.2
79.8
79.1
47.5
74.6
76.0
70.6
64.3
88.3
90.6
116.3
92.0
136.4
120.5
69.5
56.6
49,0
55.9
69.9
61.2
50.0
53.6
576 Tokens per Tile
[103]
33
76.2
76.3
134.4
50.8
46.6
65.0
46.3
1361
0.38
336/14
06.7
74.1
74.9
45.2
72.4
77.4
73.5
65.5
89.0
51.7
116.4
92.5
137.1
119.5
66.6
54.1
43.4
54.3
70.6
56.0
47.3
52.7
SigLiP2 so (135)
0.4B
384/16
55.5
61.4
51.9
33.3
72.3
76.5
70.1
66.0
88.6
81.2
118.0
95.8
136.3
119.
66.5
51.3
44.9
52.8
66.8
58.6
19.6
53.3
Sig2-122-g-op [135]
1.1B
384/16
56.2
63.1
55.3
34.0
72.4
77.0
70.3
66.7
89.6
R1.6
117.7
94.9
137.8
120.3
66.5
53.9
46.2
53.9
66.6
53.8
48.5
54.7
Pri G1
336/14
88.5
66.4
120.3
140.2
49.4
74.1
53,1
7024 Tolous per life
SigLIP2-s0 [135]
0.4B
512/16
56.9
66.0
56.5
34.3
70.9
76.4
69.9
66.2
88.4
81.2
117.8
94.7
137.8
120.9
67.8
46.2
47.0
44,9
66.7
39.1
34.5
45.1
P.
0.38
448/14
67.1
72.4
78.3
46.4
71.2
76.4
74.0
63.7
88.8
79.0
113.9
91.5
134.5
115.7
62.9
51.4
47.0
51.2
62.7
49.6
47.8
50.1
P.
0.38
448/14
78.3
82.8
89.3
65.2
75.9
78.5
78.8
64.4
99.6
81.3
117.8
94.7
138.1
120.7
71.6
56.5
47.0
57.2
68.0
59.8
52.3
54.7
AIMv2 (36)
2.7B
448/14
67.5
75.5
66.2
139.7
19.9
ImeenViT2.55
5.58
448/14
67.4
74.6
74.3
47.6
72.9
75.9
71.3
64.8
87.7
79.7
110.4
85.3
132.5
113.5
56.8
52.0
46.0
49.6
65.0
50.6
49.6
51.3
68 [15]
G
199 448/14
68.0
75.4
81.2
09.7
76.4
62.5
99.1
79.6
115.0
91.6
134.5
67.6
53.2
46.0
67.0
51.3
48.7
52.0
PEiang G
L.78" 448/14
78.6
81.8
89.8
67.8
75.0
80.3
82.3
66.7
89.6
82.8
119.6
95.2
140.3
1234
7188
59.0
49.6
61.8
73,9
60,9
526
56.3
Table 31: 4+1 Tile Llama 8B MLLM Results. Llama 3.1 --instructt 8B [80] is used as a language model. aPEjang has 1.7B parameters since we discard the last 3 layers during language alignment. All MLLMs are trained with dynamic tiling for different image sizes and aspect ratio. We use up to image tiles of 448 x 448 (or the corresponding resolution for each encoder). The image tiles follow after a thumbnail input, similar to prior work [75]. 'Evaluation on an model that was interpolated without additional training (i.e., zero-shot resolution).
Enooder Parms
Arg OCR QA
OCR/ /Chrrt/oo Q44
Visual Q4A
Captioning
Ground
Ret secoggee (55)
Video
Reselation
Stack
Charkyy
[e])
DoeYQA
Arc (80)
Info QA
AID
Arc. (S6)
Arg. VO4
TARVQA
1221
Alc (1h)
POPE
(T))
VQN2
Arc. [90]
Arg. Cop.
Flicker
CID (15)
COCO
CIO (P])
No
C300
Ang.
VileMMME
Alc 1971
STAR
TGIR-AA
Arc (82)
EgosSchem Schemma
MVBBench
Alc. (67)
Porceptiont 1003)
Model
576 Todens per Image
SigLLPP--0 [135] SigLIP2 g (p1))
0.48
384/16
60.5
72.0
59.1
36.7
74.3
76.2
69.0
65.4
89.2
81.1
116.3
137.3
70.0
57.0
51.3
66.0
51.9
55.7
1.1 384/16
60.8
71.0
00.4
36.7
75.2
76.8
70.3
65.6
89.5
81.8
118.8
96.4
139.0
121.1
69.9
58.3
52.0
57.6
68.1
62.0
52.8
57.4
1,7
336/14
66.8
77.5
72.4
41.1
76.4
76.0
67.9
65.4
89.1
$1.5
118.8
94.6
139.5
122.3
70.1
60.2
54.6
61.7
69.8
63.6
54.3
57.2
Tokens per Image
VntenVir25 1181
0.38
448/14
60.3
75.4
61.1
36.2
69.4
74.2
65.6
63.7
87.8
79.5
112.1
88.5
133.5
114.1
68.1
55.8
50.3
54.7
66.6
29.0
10.6
53.8
Sipl-P2--a [135]
512/16
66.3
77.9
117.1
158.0
120.3
50.3
L.
0.38
448/14
63.5
73.9
67.4
40.5
72.2
35.7
69.2
64.0
39.4
30.2
113.3
88.7
135.2
115.9
66.5
57.3
49.6
57.8
67.7
00.8
52.3
55.5
PEimg
0.38
448/14
70.2
30.7
46.0
73.5
36.8
72.8
64.1
$1.0
116.4
93.4
137.6
70.4
51.6
67.4
53.4
DINOV2 [96]
1.1B
448/14
31.3
21.7
14.7
24.6
64.3
61.0
18.9
59.5
58.9
76.9
110.1
87.3
132.1
110.8
69.3
54.3
46.9
56.5
63.4
56.8
49.7
52.2
AIM22 38 [36]
2.78
448/14
66.0
76.7
70.5
41.4
75.2
77.9
74.2
66.2
89.4
$1.9
119.2
96.4
139.2
67.6
56.3
45.9
58.0
67.8
51.4
53.9
IntenVir2.5 [18]
5.58
448/14
64.2
782
05.3
39.6
73.6
76.4
70.1
64.5
89.3
81.7
117.6
95.9
138.
118.6
72.8
56.1
50.3
59.1
67.3
56.6
51.1
52.2
PE
1.9
448/14
61.8
75.9
41.6
72.9
75.2
62.4
113.1
135.2
57.0
48.7
52.9
PEtaag G
1.78
448/14
72.9
81.6
83.7
49.5
76.7
77.9
74.9
64.5
90.3
81.9
118.9
94.6
139.8
122
72.1
60.4
54.1
62.5
68.3
66.6
54.2
56.8
Table 32: MLLM Results with QwenLM 2.5 7B. Same setting Tab. 5, but with QwenLM2.55
7B [153] as the language model. Although PE is aligned to Llama3.2 2. 3B, the language alignment transfers well to a different language model.
41

---

Enenda Pazans
Grunding
Retolation
RdCCCC
RdCCCC
RdCDCCO
RdCC0CO+
RefCOCOx
ReCCOCOs
RorCUCC
Plant: Stor
(59)
aosi (33)
mnd (53)
enn (55)
Arg:
376 Tokons per Imsgee
CLIP [103]
0.38
336/14
65.0
66.7
61.4
71.6
57.6
62.5
54.5
13.2
72.8
A1Mv2-1.p9 [6]]
0.38
336/14
63.4
69.6
55.0
60.0
52.0
71.5
AIMV2-L Ditt [36]
0.38
336/14
62.6
64.8
61.0
69.4
54.4
59.0
51.3
70.8
70.0
SigLIP2-s0 [125]
0.4B
394/16
76.0
SigLIP2 goopt (135)
L.B
284/16
66.5
67.9
66.1
70.1
55.8
61.7
57.1
75.5
75.0
Plasug
236/14
77.3
1024 Image
V11.5 [188]
0.38
448/14
66.9
69.3
66.7
72.6
58.1
63.1
57.2
74.2
74.0
SigLIP2-40
0.48
512/16
71.4
69.2
74.4
61.3
77.2
PRoow L
0.31
448/14
59.7
61.7
55.3
66.9
53.1
58.8
48.0
68.5
67.5
Plaang
0.3B
148/14
70.5
71.
70.2
73.0
63.7
62.7
78.8
78.9
DINOW2 196]
1.18
448/14
64.9
67.2
62.5
70.5
57.0
61.0
54.5
73.1
73.1
AIMv2 (36]
27B
448/14
37.6
32.7
38.6
V122 6B (18]
5.5B
448/14
68.0
70.2
67.6
72.2
60.6
61.0
58.7
75.3
75.2
75.0
PRiuag G
1.78
448/14
71.3
71.9
69.9
75.1
64.2
67.3
63.0
79.4
79.2
Table 33: Llama ML.LM-Based Zeroshot RefCOCO. With Llama 3.1- 1in-instrttt 8B [80] as the I.I.M.
Eneoder Paras
Grounding
Reslution
Arg. Ground
ReCCOCD
RCCOCO
soal ReCOCD
ReCoCD.
ReCOCDD
ReCOCO-
ReCoCOg
RerCooog
Parch Siw
sol [33)
Aooi (SS)
sadl (SS
ne (53)
(83)
Model
576 Tilene Image
SigLIP225s0 [135]
0.48
384/16
73.6
59.9
78.4
77.2
SigLIP2 giopt [135]
1.18
284/16
69.9
73.3
72.4
73.6
003.
62.3
00.7
78.4
73.2
PRang
336/14
73.4
61.2
78.4
77.7
Image
IimemVI2..5 L. [18)
0.38
448/14
68.1
72.4
69.1
74.1
59.3
62.4
56.6
75.2
75.5
512/16
74.4
77.9
P.
0:38
448/14
66.5
70.4
67.8
71.5
57.7
61.1
56.2
75.8
75.3
0.38
448/14
34.4
74.6
78.7
DINOV2 [96]
L.B
448/14
69.3
73.4
71.1
73.9
60.0
63.9
39,0
76.4
76.7
AIMv2 [36]
277
448/14
67.6
71.4
56.3
76.4
Vitem 68 [181
5.38
448/11
77.7
76.5
PEoneG
199
448/14
70.5
74.0
71.8
75.8
61.5
61.8
60.1
78.5
77.3
PEany
1.78
448/14
72.1
75.4
72.9
76.
64.2
65.9
62.9
79.7
79.7
Table 34: Qwen MLLM- Zeroshot RefCOCO. With QwenLM 2.5 7B [153] as the LLM. All
MLLMS report zeroshot results RefCOCO/+/g datasets. *Trained with RefCOCO/+/g beforehand.
Eisodes Params
Geunndng
Resolation
Arg Ground.
ReC0000
RaCOCo
RaCOCD
RaC000
ReC0C00
ReC0000
ReCOO00
Planh Scor
[55)
Sood (85)
1od (59
Bnd 189)
Medel
256 Tokent per Tile
MensLLIP-L [150]
0.38
224/14
60.3
73.2
60.5
56.5
73.4
MetoCLIP.G [150]
188
234/14
66.11
67.9
63.2
71.9
59.2
62.9
55.8
73.8
73.1
Plangg
224/14
78.2
Tikens
CLIP [103]
0388
336/14
68.5
70.7
66.6
74.1
61.1
65.9
58.1
76.0
75.1
ATMv2-L. [66]
0.3B
336/14
66.6
63.4
65.5
71.4
59.3
63.4
56.5
74.2
74.2
Sigl IP2-51 [135]
0.48
384/16
67.9
66.1
58.8
57.1
75.0
SigLIF2 (-t.i/2---ppti- (135)
1.1
384/16
66.5
68.2
65.6
70.1
39.0
02.3
58.0
74,8
74.0
Plasg G¹
1.7B
336/14
71.9
73.6
71.5
74.9
64.8
67.3
63.9
80.4
80.6
Tobves per
SigLIP2-s0 [125]
0.48
512/16
67.8
09.2
67.8
71.2
59.9
62.5
59.0
76.9
76.0
PR
0.38
448/14
70.0
PEang L
0.38
148/14
71.6
73.0
70.8
74.3
65.2
67.2
62.9
79.7
79.7
AIMv2 38 [36]
2.78
448/
65.1
66.9
62.9
71.1
58.1
62.4
55.6
71.8
72.2
Interavir2. 6 [18]
5.58
448/14
58.9
1.98
448/ 14
67.6
60.2
65.8
72.4
50.9
64.1
58.1
75.
75.6
PEangg
72.6
64.6
80.3
Table 35: 4+1 Tile Llama 8B MLLM-Basee Zeroshot RefCOCO. With Llama 3.1-instruct 8B [80] as the LLM. We use up to 4 image tiles of the encoder's native resolution, with a thumbnail image in front, similar to prior work [75]. *Trained with RefCOCO/+/g beforehand.
42

---

E
Additional PEspatial Results
E.1
Alignment Method Derivation
Here we detail the analysis performed arrive at the spatial alignment method discussed in $5.
E.1.1 Core Feature Analysis
Featuure Similaity
We begin by analyzing the spa- tial properties of the features for PE G in the range of lay- ers where it performed opti- mally for zero-shot tracking in $3. In Fig. 18, we plot (1) the
Attention Map
Zero-She Shot-Shhtthe tracking
pairwise feature cosine similar- ity between the pink token and all others, (2) the head average attention map for that token, and (3) the full attention ma- trix (HW x HW).
Attention Matrix
Layer 30
Layer 31
Layer
Layer 35
Layer
Figure 18: PE G Feature Analysis. To understand the dichotomy between optimal Po features for spatial tasks observed in Fig. we analyze the spatial properties of the features between layers 30 and 34.
An 18 Layer Decoder. Re- markably, the cause for the tracking performance peak layer 32 is abundantly clear from observing the visualizations. Up until layer 32, the attention maps remain local. However, that changes abruptly at layer 33, at which point several tokens in the background of the image become "global" tokens. As shown by the vertical lines in the full attention matrix, starting from layer 33 every token attends to them. Thus, every layer 33 and up become part of a decoder for global information.
This is not transformers above scale. But notably these "global tokens" are not necessarily harmful. Given the optimal layer for most tasks in Fig. 4 lies within the global token region, the information they aggregate is useful downstream. However, tracking in $3 is zero-shot and relies purely on spatial correspondences, meaning it cannot make peaks right before their introduction, while tasks that rely on semantic understanding or have larger decoders that benefit from them do well with the later layers.
phenomenon. Recent work [23] shows this happening in all modern vision
of the global tokens. This explains why tracking
E.1.2 Spatial Alignment Method
Given the analysis in SE.1.I, we have two objectives in creating a spatial alignment method: (I) we must preserve the optimal semantic information of the model (including the global tokens) that peaks around layer 40, and (2) we must do so while emphasizing local alignment in service of spatial tasks with shallow decoders. The first be easily achieved by aligning with the model's own features (e.g.. with MaskFeat [144]), but the second is more challenging. To accomplish this, we employ the Segment Anything Model (SAM) 2.1 [108] in novel way enforce spatial correspondence information in PF.
Retaining Semantics. To retain the strong semantic features from PE we finetune the model with itself as a teacher. Specifically, we train the model to maximize the cosine similarity between its last layer and the frozen layer 41 features of its initialization (a layer around the peak for many tasks in Fig. 4). On its own this would be a tautology, so we apply heavy regularization to the student: DropPath [49] and LayerScale [132] similar to language alignment,
well performing MaskFeat [1 [144] with 75% masking. We keep the teacher fixed in contrast to other state-of-the-ar spatial models, which all employ an EMA teacher [96, 135]. This could potentially help, but we opt for simplicity.
Encouraging Locality. While could "retain" locality by self-distiling from layer 32 features, that may be less effective as we are already distilling another layer of the model. Instead, we turn to model that is explicitly tuned for locality: SAM [57, 108]. Notably, several works [107, 113, 116]
43

---

have shown SAM to nor be an effective teacher when distilling from multiple sources (though recently [44] has shown can help with some tricks). However, upon observation of the raw features of SAM 2.1-I. (Fig. 19), the main problem may be the same one we are currently trying to solve: SAM has
global tokens as well! In this case, they appear as dark spots in a grid-like arrangement across all examples in Fig. 19 raw features.
Using the features of model that itself has global tokens to mitigate the effect of global tokens is dubious best. But, we do not have to use SAM's features to learn locality. At its core, SAM model that transforms points into spatially contiguous masks of select object. If what we want is smooth, locally consistent features, we can use the mask predictions themselves. Specif- ically, we query SAM 2. I-L. with 1024 points ar- ranged in 32 32 grid. For each point, SAM returns a H x mask logit the size of the im- age, which it normally would threshold and NMS. However, we instead concatenate those logits into H x W x 1024 tensor and use that the feature map for alignment. This explicitly produces locally well-aligned features compared to the underlying feature space and has spatial artifacts caused by global tokens, as shown in Fig. 19.
SAM 2.1 Mask Logits
SAM Features
SAM 2.1 Feature Similarity. The cosine similarity between the pink marked token and all others for SAM 2. [108] features vs. our proposed mask logit features.
Figure 19:
Then to align, we distill the spatial correspondences between tokens by computing their pairwise cosine similarity for both the student and the teacher (creating a HW x HW matrix for each) and aligning them with MSE loss. Unlike SAM's underlying feature space (which [44] shows may be brittle to interpolation), the mask logit features are robust to interpolation, so we simply interpolate them down and train at the PE model's original 448px resolution. Finally, like for self-distillation we add the same masking and regularization. For both teachers, we apply loss to all tokens and add no extra parameters other than LayerScale.
E.2 Layer Analysis Details
In Tab. 36, we present the raw numbers for the layer analysis plots in Fig. 9. Note that layer analysis was not performed exhaustively on every layer.
Detretine
Depul
Teacking king
Segmentution
PB Teacher
SAM Teacher
PB Tracher
Teacher
PE Toadter
SAM leacheet
PB Toucher
SAM Teacher
Piaputal
PR Pilliia
Layer
50
35.0
44.4
33.4
44.5
0.31
0.26
0.47
0.28
42.8
57.4
70.3
61.5
38.6
46.1
21.4
48.9
44.8
44.8
0.26
0.28
57.4
61.4
46.2
48
38.3
44.8
35.7
45.2
0.28
0.27
0.44
0.28
43.5
57.5
70.7
61.5
40.4
46.3
25.3
49.0
47
39.3
44.7
37.0
45.2
0.28
0.26
0.43
0.27
46.8
57.7
71.2
61.3
40.9
46.6
28.6
49.0
46
39.8
45,0
45.6
0.26
0.26
57.8
71.3
61.1
46.3
31.9
49.1
45
40.8
45.0
39.4
45.5
0.26
0.27
0.40
0.27
50.7
57.9
71.5
61.1
41.5
45.8
34,3
48.9
45.3
459
0.36
0.38
0.27
51.7
58.1
71.1
60.7
45.6
48.7
43
41.8
45.4
45.9
0.26
0.26
58.1
70.4
60.5
41.3
45.5
48.0
42
42.1
45.4
41.9
46.1
0.26
0.27
0.35
0.26
53.1
58.2
69.8
59.8
41.4
45.1
40.4
47.6
41
42.6
460
0.79
0.36
0.27
58.2
69.2
44.6
41.3
46.8
40
45.4
46.1
57.8
44.4
46.6
39
42.6
45.1
43.4
45.9
0.25
0.25
0.35
0.26
$1.99
57.4
68.0
59.3
41.1
43.7
42.5
46.1
38
43.2
44.9
45.9
0.78
0.35
0.26
$4.9
56.8
67.6
58.9
43.8
43.3
45.5
37
44.4
45.3
0.26
0.28
56.4
41.4
44.9
36
42.9
41.0
43.7
44.5
0.29
0.30
0.35
0.30
55.8
56.0
66.7
$8.4
39.3
40.1
42.6
43.3
35
42.4
43.1
43.3
44.3
0.29
0.31
0.33
0.29
$5.6
55.8
66.3
58.2
38.4
38.9
42.1
42.1
34
42.0
42.5
43.1
0.31
0.36
0.29
55.7
55.6
65.8
57.9
38.3
37.5
41.3
41.2
33
41.1
41.2
42.0
42.3
0.30
0.32
0.35
0.32
56.0
55.6
65.4
58.0
36.8
36.3
40.4
40.1
32
40.5
40.4
41.4
0.34
0.35
0.33
56.8
55.5
64.9
57.7
36.4
34.6
39.6
38.6
31
38.8
39.1
39.9
0.39
0.37
0.37
56.4
55.2
64.3
57.4
34.7
33.2
37.7
36.9
21
24.7
26.9
27.3
27.3
0.51
0.52
0.49
0.52
52.1
52.8
55.2
53.8
16.0
168
19.4
19.2
11
19.6
20.5
20.7
20.5
0.60
0.56
0.60
40.6
41.7
41.6
7.9
8.9
123
119
0.70
16.4
16.2
3.1
Table 36: Raw Spatial Layer Analysis Results. The raw values for the plots in Fig. 9.
44

---

E.3 Smaller Models
We additionally distill smaller models from the original PEspaia checkpoint by applying the same strategy as in spatial alignment (Sec. 5): train jointly with a semantic teacher loss and a spatial teacher loss. To align PEspatin G, the semantic teach was an intermediate layer of the original PE model and the spatial teacher was SAM 2.1 [108].
For smaller spatial models, repeat this alignment by finetuning the corresponding Pi checkpoint with the two loss functions on a portion the pretraining data. However, unlike PEspatia we do not align the smaller models themselves and SAM 2.1. Instead, we both teachers PEspatial G. The semantic pairwise similarity teacher loss is applied directly on the last layer of each model, and the direct distillation semantic teacher loss is applied after linear layer (Io match the feature dimension).
Results for the distilled models are given below in Tab. 37. Each model trained with 1024 tokens input (448px for patch 14, 512px for patch 16).
Tracking
Segmentation
DAVIS (101
ADE22K (11 [164]
Reu/Puch
Ids
P
448/14
50/50
PBguiiaL
448/14
60.6
60.1
23/24
48.1
48.1
24/24
P
512/16
12/12
PP
512/16
57.5
57.5
12/12
37.5
37.5
12/12
512/16
11/12
12/12
Table 37: Distilled Spatial Models. Smaller spatial models compared to the original PEspatial G teacher checkpoint, Evaluation with the same settings as Tab.
45

---

E.4 Additional Qualitative Results
Figure 20: More Visualizations of the feature space following Fig. 10. After the image itself, column 1 is PE G last layer features, column 2 is GE aligned to its own layer 41, column 3 is PE G aligned to SAM 2. [108] mask logits, and column is PE aligned to both, denoted PE G. See $B.3.2 for visualization method. Example images from SA- [57].
46

---

F
Broader Impacts
This work covers several artifacts and techniques that may or may not have broader societal impact. In this section, we enumerate those components and discuss what positive and negative implications they may have for each.
PE The core PE model is a CLIP model that matches the given image or video and a string of text. This can used in data curation, image or video search and retrieval, as well as several downstream use cases where vision and language alignment are necessary. PE improves upon prior work in this area substantially in both robustness (see Tab. 3) and fairnsss (see Tab. 25), as well as extending to video (Tab. 4). This has the potential to improve standalone use cases such as search, but also it has the potential to improve downstream machine learning system, for instance by providing better data curation. However, Pi is not perfect and still makes errors (see benchmarks above). This has the potential for negative impact if core used without regard to the possibility of mistakes.
PFaung Language alignment extend PE to downstream applications using Multimodal Large Language Models (MILMMs). While, we do not release any complete MI..M artifacts ourselves in this work, the pang we intend to release is an important component of such system. While the usage of Pfaaag in these systems has the potential to increase performance (especially on tasks requiring OCR, see Tab. 5), all MLLM systems have potential to hallucinate and generate errors. A system developed with Pa would be no exception to this issue.
PE Similarly, spatial alignment extends PE downstream applications such tracking, segmentation, and detection. PEsparial improves performance these areas vs. prior models, which has the potential to improve security systems, image editing systems, and other traditional computer vision systems. However, that naturally also comes with the risk that these systems can be used for unintended purposes. To mitigate this, release only the PEspatal feature encoder and not any of the downstream application heads.
PE Video Dataset (PVD). Along with the models, also release novel annotated video-capp caleocn dataset consisting of high quality samples selected for high motion content. The captions are generated by MLLM and then refined by human annotators. Because of the quality of this dataset, this has the potential to improve downstream applications such as video generation, video-language alignment (which is what this work uses it for), and video benchmarking (with PVD Benchmark). However, it is important to note these captions were initially generated by a model. Even with human refinement, there may be mistakes hallucinations left in the captions, which could impact downstream use. Similarly, downstream uses of the dataset such as training video generators may have harmful implications. To address this, will control the access to the dataset well as include appropriate license terms.
47

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We show how we build state-of-the arrt CLIP model for images and video in $2, show how the resulting model has general intermediate features in $3, and show how we use alignment tuning to create state-of-the-ard models for language tasks in $4 and spatial tasks 55. In each case, we provide comprehensive analysis and results in both the main paper and appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We try discuss limitations when appropriate, for instance for video retrieval performance in $2.3 "zero-shot video results" and PEspatil tracking performance vs. the SAM aligned model in $5 "effects"" Otherwise, we tried be as comprehensive as possible with evaluations to show the generality of our approach. We also provide evaluation on Dollar [110] for fairness in the additional benchmarks in Appendix C.4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
48

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work does not contain any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification: We provide thorough experimental and implementation details in Appendix B as well as full descriptions of what and how hyperparameters were obtained in Appendix B. The only caveat is the pretraining data, which we not allowed to describe. However, describe the curation and quantity, which are the most important peices for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
49

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release evaluation code, checkpoints, and the dataset described in Appendix A.I upon de-anonymization. However, we cannot provide code for submission due to the difficulty anonymizing it, or any other private datasets used.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
httgg/dde/Codduuieie Sis.Summiiinoooicccyy..o for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/twpps.ccppubllla http:ccccppbc/guieieeeeee http://bbbiiguiiieee/eeeeeeaaaia /Cooeeubeeeeeeeeeeeeeeee ssi onnolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: To the best of our ability, we provide exact experimental details and hyper- parameters in Appendix B as well as ablations for obtaining those hyperparameters in the main paper and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
50

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the cost of training large models, repeating most of the experiments in the paper would be prohibitively expensive. Thus, we do provide error bars. However, we attempt to be as thorough as possible by testing on a wide variety of benchmarks.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Given the number of experiments conducted this work, we are unable to give exhaustive list of compute used for each experiment. However, batch sizes, hyper- parameters, and data quantities are provided for each experiment to facilitate reproduction and to give idea of the compute required. The exact compute cost will depend the efficiency of the implementation.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt////euripp...e cc wwtppp/bllctttitcccsiiieeeeeleee
Answer: [Yes]
Justification: We follow the ethical guidelines where applicable: annotators for PVD were compensated according to local laws and regulations and the PVD dataset data we release properly licensed and conforms to legal and privacy agreements.
51

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., dep.oymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The models release are feature extractors CLIP models that align images or videos and text. We do not release any generative or language models that have a high risk for misuse. The PE Video Dataset (PVD) we release originates from officially licensed sources and is not scraped from the web.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
52

---

12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For code, we cite the repository used and their licenses where applicable in Appendix B, and use private code for the remainder. Models other than our own are used for benchmarking purposes or data generation, and all licenses are followed therein, (c.g.. Llama 3 is used for data generation but
improve language model, following the Llama 3 license). Code and models released in this paper will be under Apache 2.0 and the PE Video Dataset will licensed under non-commercial research license. These licenses will accompany the artifacts when released.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, saperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The public release of the models and PE video dataset will include licensing information alongside them well the training details presented in this work. However, these assets are not included at submission time, so we mark this as non applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]
Justification: Instructions and compensation details are included in Appendix A.2.5.
Guidelines:
53

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: A note of the review process is included in Appendix A.2.5.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLMs are used generate data for our data engine, and the usage LLMs is thoroughly documented for that purpose (in $2.2 and Appendix A). Otherwise, LLMs were not used to generate any the techniques or methodology in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve L.LMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : //euriiiss....// wttpe/ccc/oococeeeeeeeeeeee ecc//oonereeccc/2225/////////// for what should or should not be described.
54


Reviews:
[{"review_id": "JDqOyPNkqT", "reviewer": "Reviewer_pctU", "summary": "The paper presents Perception Encoder (PE), a family of large-scale vision encoders, trained with vision-language contrastive objective. It first introduces an image-only pretraining recipe combining progressive resolution, large-batch training, LAMB optimizer, and RoPE encoding, significantly improving over OpenCLIP. The proposed PEcore, a single encoder, could achieve state-of-the-art performance on image and video benchmarks. An additional probing analysis reveals that the best features lie in intermediate layers of encoder. PElang and PEspatial also could achieve competitive results on VQA/OCR tasks and dense prediction tasks.", "questions": "This paper does not have new technical contribution, so I do not have specific technical questions at this time. However, the authors are recommended to address the key weaknesses identified above. In particular:\n1. How does the proposed approach fundamentally differ from existing open-source models such as OpenCLIP and ViCLIP? A clear comparison highlighting any unique design choices or training insights would help clarify the paper’s novelty.\n2. Are there specific empirical findings, insights, or general principles uncovered through large-scale experiments that advance the field beyond what existing models provide?\n3. The observation that intermediate layers yield the best embeddings is not new. Similar findings have been reported in recent works like \"Scaling 4D Representations\". Without deeper analysis, this cannot be considered a main contribution.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n1. The manuscript is well-organised and extensive experiments cover zero-shot image classification, zero-shot video classification, frozen linear probing, VQA and dense prediction tasks.\n2. The empirical finding that the “best” embeddings live in intermediate layers. \n\nWeaknesses\n1. This work lacks novelty. It does not propose a new model or training objective, and instead builds on established practices such as large-batch CLIP training, progressive resolution, and LLM-based alignment. The core contribution lies in engineering scale and careful recipe design rather than introducing new insightful ideas.\n2. The approach shows little differentiation from existing methods like OpenAI’s CLIP. The community already has access to strong open-source models such as OpenCLIP for images and ViCLIP (from InternVideo2) for video, many of which cover similar ground in terms of architecture and training strategies.\n3. While the technical execution is impressive and the results are strong, the paper reads more as a report. If the authors plan to release models and datasets, it could be valuable to the community. However, strong empirical results enabled by massive compute and data are not, on their own, sufficient grounds for acceptance at a venue like NeurIPS. Accepting such a paper without clear conceptual and technical contributions would be unfair to submissions that introduce novel methods or insights.", "was_revised": true, "final_justification": "Despite more clarifications provided by the authors in the rebuttal, some of my concerns remain. The claim of reproducibility is not fully substantiated without open-sourcing the entire pipeline and training dataset. The interpretability, as emphasized in the rebuttal, leans too heavily on empirical observations. Finally, the limited scale of the released training dataset undermines the ability of others to reproduce the results. Overall, I remain inclined to reject this manuscript.", "rebuttal": "> [W1] This work lacks novelty. It does not propose a new model or training objective, and instead builds on established practices such as large-batch CLIP training, progressive resolution, and LLM-based alignment. The core contribution lies in engineering scale and careful recipe design rather than introducing new insightful ideas.\n>\n> [Q2] Are there specific empirical findings, insights, or general principles uncovered through large-scale experiments that advance the field beyond what existing models provide?\n\nWe believe that the scope of novelty is broad and covers more than just new algorithms. It can also include extending existing paradigms to new problems, designing new experiments/ablations, drawing new observations/insights, and verifying expected hypotheses in new problems or tasks. From all reviewers’ comments, we have seen that many of these values of our work are recognized. We thank the reviewers for seeing value in our work from that perspective.\n\nIt is true that our work makes use of existing techniques. Yet, several high profile works such as GPT3 (NeurIPS 2020 Outstanding Paper Award), ViT (ICLR 2021), and CLIP (ICML 2021) used primarily existing techniques at the time, but offered significant technical and engineering novelty. We believe this work falls in the same category, with many technical and engineering contributions:\n\n1. PE is the first work to jointly obtain state-of-the-art image and video zero-shot classification and retrieval _with the same model_, primarily enabled by:\n   - A thoroughly ablated robust pretraining recipe (Section 2.1). While many of these techniques are used in other works, they are often just stated as being used without study (e.g., InternViT, EVA-CLIP, SigLIP2). The knowledge of which components are important at scale and their effect on robustness is a significant benefit to the community.\n   - A new video data engine, whose synthetic outputs lead to up to 10+% absolute gains on video benchmarks. Moreover, we will release the data we collected to construct it to the community (PE Video Dataset, see L128-129, Appendix A.1).\n   - A novel CLIP model distillation approach to produce very strong smaller models (L179-182, ablations in Appendix C.3).\n\n2. As described by reviewer AWet, PE _“demonstrates that a single, simple contrastive pretraining can yield intermediate features competitive with specialized captioning and self-supervised models”_. We expand on this in our response to [Q3] below, but this has not been shown by any CLIP-pretrained model until now, and we show this is a _direct result_ of our robust pretraining recipe in Section 2.1 with ablations in Appendix C.5.  The community has been searching for a single vision pretraining task with good performance on all downstream tasks for a while now, so we believe this finding will be particularly useful.\n\n3. We fully agree that LLM-based alignment is not new; however, our technique for spatial alignment is _entirely novel_. In fact, not only do we introduce a new loss for this alignment (described in L302-309, Eq. 3 in Appendix B.3.1), but we also use the SAM model for distillation in an _entirely new way_ (SAM mask logits, L306-307 and Figure 19). We expect this to provide significant benefit to future work, especially given existing work has found it hard to distill from a SAM model (as mentioned in L1460-1461). Regardless of the perspective on novelty, this at least qualifies as “a new training objective”, and we've demonstrated its effectiveness across several established benchmarks.\n\nWe hope that this helps to clarify our view on the novelty of our work.\n\n\n---\n> [W2] The approach shows little differentiation from existing methods like OpenAI’s CLIP. The community already has access to strong open-source models such as OpenCLIP for images and ViCLIP (from InternVideo2) for video, many of which cover similar ground in terms of architecture and training strategies.\n>\n> [Q1] How does the proposed approach fundamentally differ from existing open-source models such as OpenCLIP and ViCLIP? A clear comparison highlighting any unique design choices or training insights would help clarify the paper’s novelty.\n\n\nCorrect, PE follows in the CLIP line of work, meaning it will naturally share similarities with CLIP-based vision encoders. The main differentiating factors are:\n1. **Performance**: PE is the most powerful CLIP model to date, for both images and video. PE significantly outperforms other open models like OpenCLIP. The biggest OpenCLIP model (OpenCLIP ViT-bigG) obtains 80.1% IN-1k top-1, while PE core G obtains 85.4% (as well as significantly higher robustness metrics). ViCLIP 6B (InternVideo2) obtains 64.2% Kinetics 700 top-1, while PE core G obtains 69.1%. In both cases, a 5% performance improvement represents a substantial benefit to the community.\n   - Also note the starting point of our ablations (row 1 in Figure 2) _is_ OpenCLIP’s vanilla model trained on our data. Thus, PE’s strong performance is explained by our ablations, not hidden training tricks.\n2. **Joint Modeling**: PE obtains this strong performance on both image and video _with the same checkpoint_. Not only is this more convenient than using different models for images and videos, it also enables new capabilities that using OpenCLIP / ViCLIP today in tandem couldn’t offer. For instance, image→video or video→image retrieval.\n3. **General Features**: As shown in Section 3, PE core has strong, general features in its intermediate layers that can match or exceed the performance of encoders specialized to those domains (AIMv2 for MLLM, DINOv2 for spatial). This has not been shown before—in fact, CLIP models are notorious for being bad at spatial tasks. And as mentioned above this is a _direct result_ of robust pretraining strategy in Section 2.1, as shown in Appendix C.5.\n\n\n\n\n\n---\n> [Q3] The observation that intermediate layers yield the best embeddings is not new. Similar findings have been reported in recent works like \"Scaling 4D Representations\". Without deeper analysis, this cannot be considered a main contribution.\n\nThank you, this is a good point, and we would like to clarify that there are two aspects in our claim that “a well-tuned large-scale contrastive model can learn general embeddings in the process of fitting its objective, but it fails to output them”:\n\n1. Models can have intermediate features that are relevant to a given task but not output them.\n2. _Our model_ (contrastively pretrained) shows very strong general features, on par with the best models in each domain.\n\nThe first part (1), as you point out, is not new. In fact, we dedicate a section of the related work (L372-383) showing other works that share the same finding. What we add to the conversation is controlled, large-scale layer-wise ablations of large models across a diverse set of tasks. In this respect, we show surprising results that all models we test (DINOv2-g, AIMv2 3B, PE core G) have optimal layers for some tasks as deep as the very middle of the network (tracking for AIMv2, OCR for DINOv2), and the difference can be extreme; for example AIMv2’s middle layer has 54.7 J&F on DAVIS vs. 29.3 for the last layer.\n\nHowever, the main crux of our work is (2): that, when trained in the right way, a fairly vanilla CLIP model has intermediate features on par with the best pretrained models whose losses specialize for that task (AIMv2 for MLLM, DINOv2 for spatial tasks). We explicitly ablate this phenomenon in Appendix C.5, where we show these general features were a direct result of our robust pretraining recipe in Section 2.1 and that this general performance _scales_. As mentioned before, this is the first time this has ever been shown in a CLIP model.\n\nThis is an important point to clarify, and so we will update the introduction in the final draft to make this point clear. Thank you.\n\n\n---\n> [W3] While the technical execution is impressive and the results are strong, the paper reads more as a report. If the authors plan to release models and datasets, it could be valuable to the community. However, strong empirical results enabled by massive compute and data are not, on their own, sufficient grounds for acceptance at a venue like NeurIPS. Accepting such a paper without clear conceptual and technical contributions would be unfair to submissions that introduce novel methods or insights.\n\nWe will be releasing both the models, code, and PE video dataset. We are glad that the reviewer agrees that this work can be valuable to the community. We are happy this appears to be a consensus among the reviewers: _“open-source of this work would bring significant benefits to the research community”_ (Reviewer jvg2), _“could benefit a wide range of future research”_ (Reviewer KE9i), _“convincingly establish new SOTA in multiple categories”_ (Reviewer AWet), _“the paper is already top notch”_ (Reviewer KER3). We hope the technical contributions and novel methods listed above are convincing enough to reconsider meeting the bar of NeurIPS acceptance.\n\n---\nWe thank the reviewer for the constructive feedback and hope our response can inspire a reconsideration of the score. Please let us know if anything is unclear or needs further discussion."}, {"review_id": "5UQIM7tgHC", "reviewer": "Reviewer_KE9i", "summary": "The paper introduces Perception Encoder, a family of vision encoders trained solely with contrastive vision-language learning, demonstrating that this single pretraining strategy can match or surpass other pretraining objectives (e.g., captioning) across a wide range of image and video tasks.\n\nThe authors’ core research finding is that the best features for tasks like VQA, detection, and depth estimation typically emerge in different intermediate layers. Therefore, they propose language alignment and spatial alignment methods, for multimodal language modeling and dense prediction downstream use cases respectively, to lift the best features from the middle layers toward the end.\n\nPerception Encoder provides useful resources for general-purpose vision understanding, as the authors plan to release the models, code, and dataset.", "questions": "1. Line 277~282 mentions that the model training mix did not contain grounding data and it means that the significantly lifted grounding performance is entirely due to the strong intermediate grounding features now being aligned to the end of the network. However, the alignment method itself doesn’t intuitively or theoretically guarantee such an outcome. So what mechanism might explain this emergent grounding capability?\n2. Line 257~259 concludes that “a well-tuned large-scale contrastive model can learn general embeddings in the process of fitting its objective, but it fails to output them”. However, based on the results in Figure 4, this limitation seems to also apply to models like AIMv2 and DINOv2, which are not purely contrastive. Is there experimental evidence that this phenomenon is specific to contrastive models? Or does this insight actually more broadly apply to other well-tuned pretrained models regardless of their training objective?\n3. Why not design a Perception Encoder checkpoint that integrates both language alignment and spatial alignment? This could enable a single checkpoint to support both downstream multimodal language tasks and dense spatial/spatio-temporal prediction tasks. Was this direction considered, and what are the trade-offs or challenges in doing so?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Comprehensive, well-motivated, and large-scale experiments were conducted.\n2. The analysis is dense and insightful.\n3. The work makes a meaningful contribution by releasing powerful vision encoders, along with code and dataset, which could benefit a wide range of future research.\n\nWeaknesses:\n1. For downstream language modeling (e.g., MLLM development) and dense spatial tasks, separate perception encoder checkpoints are ultimately required. \n2. There is no theoretical guarantee that the proposed language alignment and spatial alignment methods will ensure the most informative features are propagated from the middle to the final layer. In fact, Figure 5 shows that performance plateaus toward the final layers, and may decline beyond the 50-layer mark (however, we cannot confirm this because the authors experimented with a model that contains only 50 layers); Similarly, Figure 6 indicates that for some tasks, such as detection, the best-performing layer is not the final one.", "was_revised": true, "final_justification": "I agree with reviewer pctU that the work has limited novelty from a scientific research perspective, as it does not introduce substantial new ideas or insights. Several limitations and flaws noted by all reviewers also reduce its scientific contribution. However, I also feel that developing such an encoder is a nontrivial effort (despite being largely engineering) and it is highly meaningful especially with the code, data, and model checkpoints open sourced. Considering these points, I agree with the other reviewers and recommend acceptance.", "rebuttal": "> [Q1] Line 277~282 mentions that the model training mix did not contain grounding data and it means that the significantly lifted grounding performance is entirely due to the strong intermediate grounding features now being aligned to the end of the network. However, the alignment method itself doesn’t intuitively or theoretically guarantee such an outcome. So what mechanism might explain this emergent grounding capability?\n\nThank you for the great question! While PE core’s grounding performance peaks at layer 38 and then trends downward, PE lang’s performance plateaus instead (with the last layer performing similarly to layer 38). That is why we suspect the alignment “brings out” those features, even though there is no data in the training mix for it. As for the mechanism, we have additional analysis in Appendix C.8.1 that shows the last ~18 layers of the PE core G model form a sort of “decoder” to produce the CLIP token. This is close to the optimal layer for grounding (38), so it’s possible that the reason PE core drops in grounding performance is because this “CLIP decoder” is not suited for grounding. Since MLLM alignment naturally replaces the “CLIP decoder” with an “MLLM decoder”, the relevant features may now be passed through. Of course, this is just a hypothesis based on the current evidence. More work would need to be done to show this concretely. \n\n\n\n\n\n---\n> [W2] There is no theoretical guarantee that the proposed language alignment and spatial alignment methods will ensure the most informative features are propagated from the middle to the final layer. In fact, Figure 5 shows that performance plateaus toward the final layers, and may decline beyond the 50-layer mark (however, we cannot confirm this because the authors experimented with a model that contains only 50 layers); Similarly, Figure 6 indicates that for some tasks, such as detection, the best-performing layer is not the final one.\n\nAbsolutely, this is true for spatial alignment since it’s a proxy task. Like you said, there’s no guarantee that training on this proxy task will align the performance of every downstream spatial task to the last layer, given each uses different and sometimes specialized decoders. However, language alignment is more concrete. Since all MLLM tasks share the same decoder (and LLM), it’s more likely that a single layer can be the best for all tasks. This would be a given if the decoder used at test time was the same as the decoder we aligned with, since the loss is directly minimized for that specific layer with that specific decoder. Given that we align with Llama 3.2 3B and test with Llama 3.1 8B, this is not guaranteed. But empirically, it seems to be the case. And we even see evidence that this alignment can transfer to entirely different LLM families, for instance to QwenLM 2.5 7B (see Appendix C.7 / Table 29).\n\n\n\n\n\n\n---\n> [Q2] Line 257~259 concludes that “a well-tuned large-scale contrastive model can learn general embeddings in the process of fitting its objective, but it fails to output them”. However, based on the results in Figure 4, this limitation seems to also apply to models like AIMv2 and DINOv2, which are not purely contrastive. Is there experimental evidence that this phenomenon is specific to contrastive models? Or does this insight actually more broadly apply to other well-tuned pretrained models regardless of their training objective?\n\nThank you, this is a great point and is important to clarify! We make two separate claims about that in our work:\n1. Models can have features that are relevant to a given task but not output them.\n2. _Our model_ (contrastively pretrained) shows very strong general features, on par with the best models in each domain.\n\nFor (1), you are absolutely correct: AIMv2 shows exactly the same phenomenon for tracking and DINOv2 for OCR and VQA. This is something that has already been widely shown (though, not to the extreme extent of the middle layers being the best): e.g., InternViT uses the 4th to last layer, DINOv2 itself evaluates with intermediate layers. However, the crux of our work is (2), that, when trained in the right way, a fairly vanilla CLIP model has intermediate features on par with the best pretrained models whose losses specialize for that task (AIMv2 for MLLM, DINOv2 for spatial). We more explicitly ablate this phenomenon in Appendix C.5, where we show these general features were a direct result of our robust pretraining recipe in Section 2.1 and that this general performance _scales_. To our knowledge, this is the first time this has ever been shown in a CLIP model.\n\n\n\n\n---\n> [W1] For downstream language modeling (e.g., MLLM development) and dense spatial tasks, separate perception encoder checkpoints are ultimately required.\n> \n> [Q3] Why not design a Perception Encoder checkpoint that integrates both language alignment and spatial alignment? This could enable a single checkpoint to support both downstream multimodal language tasks and dense spatial/spatio-temporal prediction tasks. Was this direction considered, and what are the trade-offs or challenges in doing so?\n\nYes, this is something we considered! Unfortunately, getting the performance of the model aligned to the end for all tasks and all decoders is very challenging (as you pointed out earlier with the spatial tasks not being fully aligned). We chose the separate checkpoints direction simply because we found this to be the best course of action in the near term, given our layer and alignment analysis. In fact, concurrently with this work, SigLIP2 attempted to support all the MLLM and dense prediction tasks with a single model by mixing together losses during pretraining. However, as we show in Table 6, this only seemed to improve alignment by a few layers v.s. SigLIP1 if at all. It seems like this is an area that may take the research community multiple iterations to get right. We hope that our analysis and layer-wise evaluation techniques can help accelerate that process.\n\n\n\n\n\n---\nFinally, we would like to thank the reviewer for their thorough comments and questions. We also thank the reviewer for their kind words that we conduct _“comprehensive, well-motivated, and large-scale experiments”_, have _”dense and insightful”_ analysis, and make _“a meaningful contribution by releasing powerful vision encoders, along with code and dataset, which could benefit a wide range of future research”_. Please let us know if anything is unclear or needs further elaboration, and we will try to answer."}, {"review_id": "Nj1o8Q2odz", "reviewer": "Reviewer_KER3", "summary": "The paper „Perception Encoder: The best visual embeddings are not at the output of the network“ proposes a multi-step training strategy to create a new vision foundation model, effective for various tasks. First, the authors train a base image model with a vision-language CLIP-like contrastive objective, using carefully ablated components. Next, they employ a video data engine to source captioned videos, with which their train their all-purpose Core model. Furthermore, they discover that different layers within the network are most effective for different tasks, and with that in mind, perform language and spatial alignment of their core model, to train specialized spatial and language models.", "questions": "Overall, the paper is already top notch, without minor room for improvement. However, I’m still asking myself:\n- Why do you think the video data engine moves the performance mainly for COCO-like dataset and not ImageNet object-centric datasets? Has the performance of the model already plateaued for ImageNet at that point?\n- It would be nice to have to separate alignment trainings nicely displayed in a supplementary table (see weakness)\n\nI really like the paper, and rate it with „Strong Accept“. I would appreciate if the authors would address my points in the rebuttal.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The proposed method for the base image model is very systematically designed and each component’s effect is well investigated\n- Training a unified video and image VFM is a step in the right direction, since both forms of data can benefit from each other\n- Employing a data engine to move to video data provides a very targeted way to source data that is effective at improving the model’s performance\n- The design components of the video data engine are well ablated and thoroughly investigated\n- The results presented in Tables 3 (image) and 4 show consistent across many different datasets and tasks.\n- The most interesting finding is presented in Sec. 3, where the authors discover that different layers of the network are best suited for the various different tasks, such as language and spatial tasks. This serves as a nice motivation for the following alignment trainings.\n- The spatial alignment to layer 41 AND SAM logits is an interesting design choice, and the visualizations show that it preserves a good mix of details and semantics in the last layer. \n- Overall, the results of the alignment models show a strong performance and underline the effectiveness of the respective alignment trainings.\n\nWeaknesses:\n- The video data engine ablation shows that it moves the needle only a little bit for the object-centric image datasets like ImageNet, but the gains are much greater for the more complex image datasets like COCO and, obviously, all video datasets. It would be interesting to learn about why especially COCO got such a boost, but not ImageNet. Perhaps the video scenes are more similar to COCO scenes?\n- While I appreciate Fig. 6 with the graphs showing the effects of aligning just to layer 41 or just to SAM logits, I think it would nice to have these numbers in a Table somewhere in the supplementary.\n- In the Table 3 evaluation on image tasks, without specialized alignment of the model, the performance many times just marginally exceeds that of SigLIP2, if at all. Overall in this table, these two models are relatively close together.", "was_revised": true, "final_justification": "The authors have addressed my remaining weaknesses. I therefore keep my rating of Strong Accept.", "rebuttal": "> [W1] The video data engine ablation shows that it moves the needle only a little bit for the object-centric image datasets like ImageNet, but the gains are much greater for the more complex image datasets like COCO and, obviously, all video datasets. It would be interesting to learn about why especially COCO got such a boost, but not ImageNet. Perhaps the video scenes are more similar to COCO scenes?\n>\n> [Q2] Why do you think the video data engine moves the performance mainly for COCO-like dataset and not ImageNet object-centric datasets? Has the performance of the model already plateaued for ImageNet at that point?\n\nThank you for the question; this is important to explain. We believe the video data helps image performance in two ways:\n1. The synthetic captions we generate (see Fig. 9 for examples) are often higher quality than the alt-text we use during pretraining. Several works [1,2,3] have shown this to be effective in improving _retrieval_ performance, which includes datasets such as COCO and Flickr (see Table 3).\n2. Synthetic captions typically _do not_ improve classification performance (ImageNet, ObjectNet, etc.), as the structured descriptions of synthetic captions do not capture the long-tail information necessary for fine-grained classification. However, the fact this new data consists of _videos_ means that the model now sees a diverse set of objects from many different angles and in many novel situations. This is particularly noticeable in the improvement of classification robustness metrics, particularly ObjectNet and ImageNet-A (see Table 3).\n\nTaken together, the video data improves the robustness of classification and the effectiveness of retrieval.\n\n\n[1] “What If We Recaption Billions of Web Images with LLaMA-3?”  \n[2] “Revisit Large-Scale Image-Caption Data in Pretraining Multimodal Foundation Models”  \n[3] “Altogether: Image Captioning via Re-aligning Alt-text”  \n\n\n\n\n\n---\n> [W3] In the Table 3 evaluation on image tasks, without specialized alignment of the model, the performance many times just marginally exceeds that of SigLIP2, if at all. Overall in this table, these two models are relatively close together.\n\nThis is true! Though, we’d like to note that (1) SigLIP2 is concurrent work—we include it in the results because we are confident in our model and techniques; (2) PE core jointly performs image and video CLIP tasks. SigLIP2 doesn’t support this and thus does much worse on videos in Table 4; (3) models trained on WebLI (Coca, SigLIP, SigLIP2) have dominated the CLIP benchmarks for the past 3 years. This is the first time they’ve been outperformed without WebLI.\n\n\n\n\n---\n> [W2] While I appreciate Fig. 6 with the graphs showing the effects of aligning just to layer 41 or just to SAM logits, I think it would nice to have these numbers in a Table somewhere in the supplementary.\n>\n> [Q2] It would be nice to have to separate alignment trainings nicely displayed in a supplementary table\n\nThank you for the suggestion! We will add the raw data for Figures 5 and 6 in the appendix. In the meantime, here are the raw numbers. Note that the layer-wise analysis wasn’t done exhaustively for every layer.\n\nFigure 5:\n\n| Layer |  | OCR Core | OCR Lang |  | Cap Core | Cap Lang |  | VQA Core | VQA Lang |  | Grnd Core | Grnd Lang |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 50 |  | 48.6 |  |  | 114.0 |  |  | 74.1 |  |  | 39.1 |  |\n| 49 |  | 53.8 |  |  | 114.5 |  |  | 74.9 |  |  | 54.2 |  |\n| 48 |  | 56.3 |  |  | 114.8 |  |  | 74.8 |  |  | 55.6 |  |\n| 47 |  | 57.3 | 72.4 |  | 113.9 | 120.1 |  | 75.3 | 78.1 |  | 57.8 | 71.2 |\n| 46 |  | 58.3 | 72.2 |  | 114.1 | 120.0 |  | 75.1 | 78.2 |  | 58.4 | 70.8 |\n| 45 |  | 59.6 | 72.0 |  | 114.6 | 119.7 |  | 75.1 | 77.7 |  | 61.2 | 70.7 |\n| 43 |  | 59.9 | 71.5 |  | 113.8 | 117.7 |  | 74.8 | 77.4 |  | 63.1 | 70.5 |\n| 42 |  | 60.3 | 71.3 |  | 113.1 | 116.6 |  | 73.8 | 77.2 |  | 64.1 | 70.8 |\n| 41 |  | 60.8 | 70.7 |  | 112.5 | 115.4 |  | 73.3 | 76.7 |  | 66.6 | 71.2 |\n| 40 |  | 61.4 | 69.8 |  | 112.0 | 115.0 |  | 74.0 | 76.3 |  | 66.5 | 71.0 |\n| 39 |  | 61.8 | 70.0 |  | 111.0 | 113.5 |  | 74.0 | 75.6 |  | 67.1 | 71.1 |\n| 38 |  | 62.1 | 69.3 |  | 110.2 | 112.6 |  | 73.9 | 75.3 |  | 68.7 | 70.3 |\n| 36 |  | 61.3 | 67.9 |  | 108.5 | 109.4 |  | 73.0 | 74.0 |  | 67.8 | 70.4 |\n| 34 |  | 58.7 | 65.4 |  | 102.8 | 104.7 |  | 70.0 | 72.2 |  | 66.5 | 69.6 |\n| 33 |  | 57.3 | 64.1 |  | 100.1 | 102.7 |  | 69.0 | 71.3 |  | 65.9 | 68.9 |\n| 32 |  | 54.2 | 63.2 |  | 96.9 | 100.1 |  | 67.8 | 70.6 |  | 65.5 | 68.7 |\n| 31 |  | 50.9 | 60.7 |  | 93.2 | 96.7 |  | 65.5 | 68.7 |  | 63.0 | 67.3 |\n| 21 |  | 29.6 | 30.3 |  | 59.3 | 71.3 |  | 49.1 | 52.0 |  | 41.5 | 51.6 |\n| 11 |  | 28.8 | 28.9 |  | 47.4 | 59.0 |  | 47.0 | 49.2 |  | 30.9 | 43.3 |\n| 2 |  | 28.2 | 28.7 |  | 38.6 | 42.8 |  | 43.3 | 43.8 |  | 22.8 | 27.2 |\n\n\nFigure 6:\n\n| Layer |  | Det Core | Det PE | Det SAM | Det Spatial |   |   | Depth Core | Depth PE | Depth SAM | Depth Spatial |   |   | Track Core | Track PE | Track SAM | Track Spatial |   |   | Seg Core | Seg PE | Seg SAM | Seg Spatial |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 50 |  | 35.0 | 44.4 | 33.4 | 44.5 |  |  | 0.31 | 0.26 | 0.47 | 0.28 |  |  | 42.8 | 57.4 | 70.3 | 61.5 |  |  | 38.6 | 46.1 | 21.4 | 48.9 |\n| 49 |  | 37.3 | 44.8 | 34.3 | 44.8 |  |  | 0.29 | 0.26 | 0.49 | 0.28 |  |  | 44.8 | 57.4 | 70.2 | 61.4 |  |  | 39.8 | 46.2 | 23.6 | 49.3 |\n| 48 |  | 38.3 | 44.8 | 35.7 | 45.2 |  |  | 0.28 | 0.27 | 0.44 | 0.28 |  |  | 45.5 | 57.5 | 70.7 | 61.5 |  |  | 40.4 | 46.3 | 25.3 | 49.0 |\n| 47 |  | 39.3 | 44.7 | 37.0 | 45.2 |  |  | 0.28 | 0.26 | 0.43 | 0.27 |  |  | 46.8 | 57.7 | 71.2 | 61.3 |  |  | 40.9 | 46.6 | 28.6 | 49.0 |\n| 46 |  | 39.8 | 45.0 | 38.4 | 45.6 |  |  | 0.27 | 0.26 | 0.42 | 0.26 |  |  | 49.1 | 57.8 | 71.3 | 61.1 |  |  | 41.4 | 46.3 | 31.9 | 49.1 |\n| 45 |  | 40.8 | 45.0 | 39.4 | 45.5 |  |  | 0.26 | 0.27 | 0.40 | 0.27 |  |  | 50.7 | 57.9 | 71.5 | 61.1 |  |  | 41.5 | 45.8 | 34.3 | 48.9 |\n| 44 |  | 41.4 | 45.3 | 40.5 | 45.9 |  |  | 0.26 | 0.26 | 0.38 | 0.27 |  |  | 51.7 | 58.1 | 71.1 | 60.7 |  |  | 41.5 | 45.6 | 36.8 | 48.7 |\n| 43 |  | 41.8 | 45.4 | 41.2 | 45.9 |  |  | 0.26 | 0.26 | 0.34 | 0.26 |  |  | 52.4 | 58.1 | 70.4 | 60.5 |  |  | 41.3 | 45.5 | 38.7 | 48.0 |\n| 42 |  | 42.1 | 45.4 | 41.9 | 46.1 |  |  | 0.26 | 0.27 | 0.35 | 0.26 |  |  | 53.1 | 58.2 | 69.8 | 59.8 |  |  | 41.4 | 45.1 | 40.4 | 47.6 |\n| 41 |  | 42.6 | 45.4 | 42.6 | 46.0 |  |  | 0.26 | 0.29 | 0.36 | 0.27 |  |  | 54.2 | 58.2 | 69.2 | 59.6 |  |  | 41.1 | 44.6 | 41.3 | 46.8 |\n| 40 |  | 42.8 | 45.4 | 43.1 | 46.1 |  |  | 0.26 | 0.27 | 0.35 | 0.29 |  |  | 54.5 | 57.8 | 68.5 | 59.5 |  |  | 41.1 | 44.4 | 42.0 | 46.6 |\n| 39 |  | 42.6 | 45.1 | 43.4 | 45.9 |  |  | 0.25 | 0.25 | 0.35 | 0.26 |  |  | 54.9 | 57.4 | 68.0 | 59.3 |  |  | 41.1 | 43.7 | 42.5 | 46.1 |\n| 38 |  | 43.2 | 44.9 | 43.8 | 45.9 |  |  | 0.25 | 0.28 | 0.35 | 0.26 |  |  | 54.9 | 56.8 | 67.6 | 58.9 |  |  | 40.2 | 42.8 | 43.3 | 45.5 |\n| 37 |  | 43.0 | 44.4 | 43.7 | 45.3 |  |  | 0.28 | 0.29 | 0.34 | 0.28 |  |  | 55.3 | 56.4 | 67.2 | 58.7 |  |  | 40.1 | 41.4 | 42.8 | 44.9 |\n| 36 |  | 42.9 | 44.0 | 43.7 | 44.5 |  |  | 0.29 | 0.30 | 0.35 | 0.30 |  |  | 55.8 | 56.0 | 66.7 | 58.4 |  |  | 39.3 | 40.1 | 42.6 | 43.3 |\n| 35 |  | 42.4 | 43.1 | 43.3 | 44.3 |  |  | 0.29 | 0.31 | 0.33 | 0.29 |  |  | 55.6 | 55.8 | 66.3 | 58.2 |  |  | 38.4 | 38.9 | 42.1 | 42.1 |\n| 34 |  | 42.0 | 42.5 | 42.7 | 43.1 |  |  | 0.29 | 0.31 | 0.36 | 0.29 |  |  | 55.7 | 55.6 | 65.8 | 57.9 |  |  | 38.3 | 37.5 | 41.3 | 41.2 |\n| 33 |  | 41.1 | 41.2 | 42.0 | 42.3 |  |  | 0.30 | 0.32 | 0.35 | 0.32 |  |  | 56.0 | 55.6 | 65.4 | 58.0 |  |  | 36.8 | 36.3 | 40.4 | 40.1 |\n| 32 |  | 40.5 | 40.4 | 41.1 | 41.4 |  |  | 0.31 | 0.34 | 0.35 | 0.33 |  |  | 56.8 | 55.5 | 64.9 | 57.7 |  |  | 36.4 | 34.6 | 39.6 | 38.6 |\n| 31 |  | 38.8 | 39.1 | 40.0 | 39.9 |  |  | 0.34 | 0.39 | 0.37 | 0.37 |  |  | 56.4 | 55.2 | 64.3 | 57.4 |  |  | 34.7 | 33.2 | 37.7 | 36.9 |\n| 21 |  | 24.7 | 26.9 | 27.3 | 27.3 |  |  | 0.51 | 0.52 | 0.49 | 0.52 |  |  | 52.1 | 52.8 | 55.2 | 53.8 |  |  | 16.0 | 16.8 | 19.4 | 19.2 |\n| 11 |  | 19.6 | 20.5 | 20.7 | 20.5 |  |  | 0.56 | 0.60 | 0.56 | 0.60 |  |  | 43.7 | 40.6 | 41.7 | 41.6 |  |  | 8.6 | 7.9 | 8.9 | 8.7 |\n| 1 |  | 12.7 | 12.1 | 12.2 | 11.9 |  |  | 0.66 | 0.70 | 0.69 | 0.70 |  |  | 28.2 | 16.4 | 17.5 | 16.2 |  |  | 3.1 | 3.0 | 3.1 | 3.1 |\n\n\n\n\n---\nFinally, we’d like to extend our thanks to the reviewer for their glowing review and for describing our work as _“already top notch”_. We hope we could adequately answer your questions with this response. Feel free to let us know if anything still needs further elaboration."}, {"review_id": "Ox8ZivV1mo", "reviewer": "Reviewer_jvg2", "summary": "This paper first introduces a new data processing pipeline for obtaining high-quality image/video captions. Then, based on these data, the PE encoder is proposed. It adopts the simple contrastive pretraining objective, but achieves SOTA performance. Developing on this encoder, the authors argue that for different downstream tasks, the vision features should be extracted from different layers of the ViT, which is not surprising. Massive experiments are conducted to validate the proposed method. The authors also plan to release the model and dataset.", "questions": "For pretraining the PE encoder, the authors now just implement a simple contrastive objective. I am curious if we can implement it with more advanced ones like SigLIP2?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-formatted and easy to follow.\n2. Massive experiments are conducted, which makes the conclusion convincing.\n3. The open-source of this work would bring significant benefits to the research community.\n4. The method is simple but effective, and personally, I enjoy this kind of straightforward work.\n5. The data processing pipeline is useful. \n\nMinor Weaknesses: I don't see any major weaknesses, but would like to raise a minor concern.\n\nI feel like the finding that the best visual embeddings are from different intermediate layers is not surprising, and it is almost well-known in all kinds of previous works. For VLMs, InternVL uses -2/-4th visual feature from the ViT(the authors also mentioned this in Line 268). For common tasks like detection or segmentation, it is almost a standard protocol to use visual features from multiple layers of ViT. In this work, the authors seem to just unify them in one paper and just implement a widely-used contrastive loss to pretrain the ViT, which slightly diminishes the novelty of this paper.", "was_revised": true, "final_justification": "Thanks to the authors for the detailed reply. I would love to raise my score. I believe the conclusion in this paper is solid, and the findings are interesting, which will benefit the whole research community a lot.", "rebuttal": "> [W1]  I don't see any major weaknesses, but would like to raise a minor concern.\n>\n> I feel like the finding that the best visual embeddings are from different intermediate layers is not surprising, and it is almost well-known in all kinds of previous works. For VLMs, InternVL uses -2/-4th visual feature from the ViT(the authors also mentioned this in Line 268). For common tasks like detection or segmentation, it is almost a standard protocol to use visual features from multiple layers of ViT. In this work, the authors seem to just unify them in one paper and just implement a widely-used contrastive loss to pretrain the ViT, which slightly diminishes the novelty of this paper.\n\nThank you, this is a great point and important to clarify. There are two aspects of the main “best layer” claim:\n\n1. Models can have features that are relevant to a given task but not output them.\n2. _Our model_ (contrastively pretrained) shows very strong general features, on par with the best models in each domain.\n\nThe first part (1), as you point out, is not new. In fact, we dedicate a section of the related work (L372-383) showing other works that share the same finding. What we add to the conversation is controlled, large-scale layer-wise ablations of large models across a diverse set of tasks. In this respect, we show surprising results that all models we test (DINOv2-g, AIMv2 3B, PE core G) have optimal layers for some tasks as deep as the very middle of the network (tracking for AIMv2, OCR for DINOv2).\n\nHowever, the main crux of our work that we try to describe in Section 3 is (2): that, when trained in the right way, a fairly vanilla CLIP model has intermediate features on par with the best pretrained models whose losses specialize for that task (AIMv2 for MLLM, DINOv2 for spatial). We explicitly ablate this phenomenon in Appendix C.5, where we show these general features were a direct result of our robust pretraining recipe in Section 2.1 and that this general performance _scales_. To our knowledge, this is the first time this has ever been shown in a CLIP model.\n\nThank you again for catching this! We will update the introduction in the final draft to make this point clear.\n\n\n\n\n---\n> [Q1] For pretraining the PE encoder, the authors now just implement a simple contrastive objective. I am curious if we can implement it with more advanced ones like SigLIP2?\n\nThank you for the suggestion. In this work, we wanted to keep the pretraining as simple as possible. We did consider using multiple losses to achieve more broad alignment, but given our layer-wise analysis we found that producing one model with aligned features for classification, retrieval, MLLM, and dense tasks is challenging. Thus, we opted to create three separate models to maximize utility to the community and to study the effects of alignment for these domains individually.\n\nInterestingly, concurrently with this work, SigLIP2 chose the opposite direction by attempting to support all MLLM and dense prediction tasks with a single model using multiple losses during pretraining. However, as we show in Table 6, this only seemed to improve alignment by a few layers v.s. SigLIP1 if at all (at least for spatial tasks). Thus, it seems like SigLIP2’s loss mix is not the right answer for joint pretraining. We hope that our analysis and layer-wise evaluation techniques can help accelerate future work in this area.\n\n\n\n\n---\nFinally, we would like to thank their reviewer for their comprehensive review. We especially appreciate your comments that, _“the paper is well-formatted and easy to follow”_, _“massive experiments are conducted, which makes the conclusion convincing”_, and _“the open-source of this work would bring significant benefits to the research community”_. If you still have any thoughts or would like further elaboration, feel free to let us know."}, {"review_id": "xjVGg72klf", "reviewer": "Reviewer_AWet", "summary": "This paper presents Perception Encoder (PE), a family of vision encoders that achieve state-of-the-art performance across a wide range of image, video, and multimodal tasks using a unified contrastive pretraining strategy. Unlike previous work that often combines multiple pretraining objectives (e.g., contrastive, captioning, self-supervised spatial losses), the authors show that large-scale contrastive vision-language pretraining alone, when executed with a well-engineered data pipeline and training recipe, is sufficient to yield strong general-purpose features. These features emerge in intermediate layers of the model, rather than at the output. To make these features usable, the authors introduce two alignment strategies: language alignment (PElang) and spatial alignment (PEspatial), which bring the strong intermediate representations to the end of the network for effective use in downstream MLLM and spatial tasks respectively. The paper also contributes a novel video-text dataset and promises open-source release.", "questions": "1. Please see weakness section.\n\n2. In the PElang alignment method, the last 3 layers of PEcore are discarded before fine-tuning. If the alignment stage can bring useful features for mllm to the last layer, why still dropping layers? \n\n2. The language alignment is performned on llama3.2 3B, and the final performance is reported by training PElang with  Llama 3.1-\ninstruct 8B -- the same family of models. Will the language alignment still work for different class of models? For example, aligning PEcore with qwen-1B and then perform final training with llama-8B.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The empirical study is thorough, with extensive experiments across classification, retrieval, dense prediction, and MLLM benchmarks. The results convincingly establish new SOTA in multiple categories. Ablation studies clearly isolate the impact of each component (contrastive recipe modifications, video-data engine, alignment stages).\n\n2. It demonstrates that a single, simple contrastive pretraining can yield intermediate features competitive with specialized captioning and self-supervised models.\n\n3. It shows  that intermediate contrastive features can be “lifted” to the network output via short finetuning, unlocking broad downstream utility. The paper also introduces a two-teacher spatial alignment combining self-distillation and SAM-derived locality.\n\n**Weakness** \n\n1. The comparison in Table 5 evaluates PElang against other frozen vision encoders, but does not unfreeze them during MLLM fine-tuning. This creates a potential bias in favor of PElang. The authors should ideally also report performance for these baselines when the image encoder is unfrozen during fine-tuning.\n\n2. The paper does not include a result for directly plugging PEcore (without PElang alignment) into an MLLM and unfreezing it. It remains unclear whether an explicit language alignment stage is strictly necessary or whether one-stage fine-tuning with unfrozen image encoder would suffice.", "was_revised": true, "final_justification": "The authors response has clearly addressed my concerns. I will keep the rating for this paper as accept and increase my confidence rating.", "rebuttal": "> [Q3] The language alignment is performned on llama3.2 3B, and the final performance is reported by training PElang with Llama 3.1- instruct 8B -- the same family of models. Will the language alignment still work for different class of models? For example, aligning PEcore with qwen-1B and then perform final training with llama-8B.\n\nThis is a good question! It’s very important for the community that PE lang’s performance is transferable to other LLMs, so we tested PE lang (aligned to Llama 3.2 3B) using _QwenLM 2.5 7B_ as the LLM in Appendix C.7 / Table 29 with the encoder frozen. Despite being aligned to a different family of models, PE lang G still significantly outperforms other models on almost every benchmark when benchmarked using QwenLM.\n\n\n\n\n\n---\n> [W1] The comparison in Table 5 evaluates PElang against other frozen vision encoders, but does not unfreeze them during MLLM fine-tuning. This creates a potential bias in favor of PElang. The authors should ideally also report performance for these baselines when the image encoder is unfrozen during fine-tuning.\n\nThis is a great point, thank you for the suggestion. It could be unfair to other models that weren’t language tuned to compare to PE lang, which is a language tuned model. Several of the models we compared to are already language-aligned with an LLM (AIMv2, SigLIP2 pretrained with captioning, InternViT 2.5 extracted from an MLLM), but your proposed experiment makes this comparison a lot more concrete.\n\nWe ran this experiment (same as Table 5, but with the encoder unfrozen) on AIMv2 3B, SigLIP2 g-opt, and PE lang G, all with 1024 tokens per image:\n\n|  | Chart QA | Doc VQA | Info. QA | AI 2D | Text VQA | OK VQA | POPE | VQA v2 | Flickr | COCO | No Cap | Ref COCO | Ref COCO+ | Ref COCOg | Video MME | STAR | TGIF QA | Ego Schema | MV Bench | Perception Test |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| AIMv2 3B | 77.4 | 68.8 | 39.4 | 74.4 | 73.1 | 64.4 | 88.4 | 81.7 | 97.1 | 139.7 | 120.7 | 69.3 | 62.3 | 77.5 | 46.8 | 53.2 | 64.5 | 58.4 | 48.9 | 53.9 |\n| SigLIP2 gopt | 77.3 | 68.6 | 39.6 | 74.6 | **74.9** | **67.7** | 89.5 | **83.2** | **97.9** | 140.7 | 121.5 | **74.8** | 65.0 | 80.3 | 43.8 | 52.7 | 66.8 | 59.0 | **51.2** | 52.9 |\n| PE lang G | **81.6** | **84.7** | **48.1** | **76.8** | 74.1 | 66.9 | **89.6** | 82.9 | 97.4 | **141.7** | **123.4** | 73.4 | **65.9** | **80.8** | **48.0** | **58.9** | **70.4** | **63.0** | 50.9 | **56.2** |\n\n\nIt seems all models, including PE lang, benefit from unlocking the encoder. However, PE lang G still outperforms the other models overall, often by a significant margin. Again, thank you for the suggestion—we will include this table in the Appendix of the final draft!\n\n\n\n\n\n\n---\n> [W2] The paper does not include a result for directly plugging PEcore (without PElang alignment) into an MLLM and unfreezing it. It remains unclear whether an explicit language alignment stage is strictly necessary or whether one-stage fine-tuning with unfrozen image encoder would suffice.\n\nAnother good point, thank you for mentioning this. We repeated the experiment above (Table 5 with encoder unfrozen, 1024 tokens) with PE core v.s. PE lang for both L and G, reported below:\n\n\n|  | Chart QA | Doc VQA | Info. QA | AI 2D | Text VQA | OK VQA | POPE | VQA v2 | Flickr | COCO | No Cap | Ref COCO | Ref COCO+ | Ref COCOg | Video MME | STAR | TGIF QA | Ego Schema | MV Bench | Perception Test |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| PE core L | 76.9 | 70.6 | 39.1 | 70.8 | 69.7 | **64.5** | 88.5 | 80.3 | **93.7** | 136.8 | 117.1 | 71.4 | 63.5 | 77.7 | 45.3 | 52.6 | 65.4 | 53.4 | **50.3** | 52.8 |\n| PE lang L | **81.2** | **82.5** | **45.6** | **75.3** | **71.7** | 64.4 | **89.2** | **80.9** | 85.1 | **138.7** | **119.7** | **72.8** | **65.7** | **80.5** | **47.1** | **55.1** | **68.4** | **58.4** | 49.9 | **54.3** |\n|   |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n| PE core G | 73.6 | 67.2 | 38.5 | 71.7 | 68.9 | 65.6 | 88.4 | 80.9 | 94.0 | 137.9 | 119.7 | 70.7 | 61.4 | 75.5 | 44.2 | 56.0 | 66.3 | 60.6 | 50.5 | 52.5 |\n| PE lang G | **81.6** | **84.7** | **48.1** | **76.8** | **74.1** | **66.9** | **89.6** | **82.9** | **97.4** | **141.7** | **123.4** | **73.4** | **65.9** | **80.8** | **48.0** | **58.9** | **70.4** | **63.0** | **50.9** | **56.2** |\n\n\nBoth PE lang models significantly outperform the PE core ones in this unfrozen setup, especially for the larger G size. Thus, it seems that a language alignment step is still necessary even when the encoder is unfrozen during MLLM construction. This is important to show, so thank you again for your suggestion! We’ll add this table to the Appendix as well.\n\n\n\n\n\n\n---\n> [Q2] In the PElang alignment method, the last 3 layers of PEcore are discarded before fine-tuning. If the alignment stage can bring useful features for mllm to the last layer, why still dropping layers?\n\nGood catch! This was an artifact of our early experimentation and the order we performed our ablations. In Appendix C.6 / Table 27, we detail how we designed the language alignment recipe. And there, we found aligning to layer 47 was better overall than layer 50. However, later in development we found regularization (drop path and layerscale, last two rows of Table 27) to be particularly useful for robust alignment and transferability. If we would repeat all the ablation experiments with regularization included first, it’s likely that layer 50 alignment would outperform or at least be on par to layer 47.\n\n\n\n\n---\nWe would like to thank the reviewer for their thoughtful comments and great suggestions. We particularly appreciate your comments that our _“empirical study is thorough”_, _“the results convincingly establish new SOTA in multiple categories”_, and that our _“ablation studies clearly isolate the impact of each component”_. We are also grateful for your suggested experiments, which we believe have greatly strengthened the language alignment section of the paper. Please feel free to let us know if anything needs further clarification and we will try to answer."}]

Author rebuttal:

Meta-review: This paper was reviewed by 5 experts in the field, and 4 of them are enthusiastic about the paper. Reviewer pctU gave borderline reject and is concerned about the technical contribution of the paper. 

The AC reads the paper, the reviews, and the authors' responses carefully, especially focusing on the review from pctU and the discussions. The AC appreciates the reviewer's detailed comments and the follow-up discussions that help to clarify the contribution of the paper. The AC also agrees with the weakness pointed out by KE9i in W1. The AC understands the concerns from the reviewer pctU and agrees that the study in the paper may not be sufficient to initiate a paradigm shift in the computer vision community. However, the current paper made significant contributions to the community, providing strong vision encoders with models and data to be released, and great insights about how to better unify the vision encoder for diverse downstream vision tasks. The AC believes the paper made a solid step forward, and the findings in the paper should be presented to a broader audience for better visibility.
Decision: Accept (oral)