Dynamic Reflections: Probing Video Representations with Text Alignment
Authors: Maks Ovsjanikov
Viorica Patraucean
Leonidas Guibas
Tyler Zhu
Tengda Han
Venue: ICLR 2026 Poster (2026)
Keywords: Platonic Representation hypothesis
video understanding
video-text alignment
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Our study of video-text representation alignment demonstrates that alignment is dramatically improved by using richer test-time data, such as multiple video frames and diverse captions.

Abstract: The alignment of representations from different modalities has recently been shown to provide insights on the structural similarities and downstream capabilities of different encoders across diverse data types.
While significant progress has been made in aligning images with text, the temporal nature of _video_ data remains largely unexplored in this context. 
In this work, we conduct the first comprehensive study of video-text representation alignment, probing the capabilities of modern video and language encoders. 
Our findings reveal several key insights. 
First, we demonstrate that cross-modal alignment highly depends on the richness of both visual (static images vs. multi-frame videos) and text (single caption vs. a collection) data _provided at test time_, especially when using state-of-the-art video encoders. 
We propose parametric test-time scaling laws that capture this behavior and show remarkable predictive power against empirical observations.
Secondly, we investigate the correlation between semantic alignment and performance on both semantic and non-semantic downstream tasks, providing initial evidence that strong alignment against text encoders may be linked to _general-purpose_ video representation and understanding.
Finally, we correlate temporal reasoning with cross-modal alignment providing a challenging test-bed for vision and language models. 
Overall, our work introduces video-text alignment as an informative zero-shot way to probe the representation power of different encoders for spatio-temporal data.

Published as a conference paper at ICLR 2026
DYNAMIC REFLECTIONS: PROBING VIDEO REPRE- SENTATIONS WITH TEXT ALIGNMENT
Tyler Zhu Tengda Han*
Leonidas Guibast Viorica Pătrăuccan*
Maks Ovsjanikov
Princeton University Google DeepMind
ABSTRACT
The alignment of representations from different modalities has recently been shown to provide insights on the structural similarities and downstream capabil- ities of different encoders across diverse data types. While significant progress has been made in aligning images with text, the temporal nature of video data remains largely unexplored in this context, In this work, we conduct the first com- prehensive study of video-text representation alignment, probing the capabillitess of modern video and language encoders. Our findings reveal several key insights. First, we demonstrate that cross-modal alignment highly depends on the richness of both visual (static images vs. multi-frame videos) and text (single caption vs. collection) data provided test time, especially when using state-of-the -he-- art video encoders. We propose parametric test-time scaling laws that capture this behavior and show remarkable predictive power against empirical observations. Secondly, we investigate the correlation between semantic alignment and perfor- mance both semantic and non-semantic downstream tasks, providing initial evidence that strong alignment against text encoders may be linked to general- purpose video representation and understanding. Finally, we correlate temporal reasoning with cross-modal alignment providing a challenging test- bed for vision and language models. Overall, our work introduces video-text alignment as an informative zero-shot way to probe the representation power of different encoders for tempoal data.
INTRODUCTION
A hallmark of general intelligence is the ability to reason about the world in its different manifes- tations and integrate different sensory data into unified mental representations (Ernst & Bülthoff, 2004: Kudithipudi et al., 2022). The pursuit of such unified representations is also fundamental to the development of capable multimodal AI systems and for the advancement of embodied agents that must operate within a complex world (Xie et al., 2024; Fung et al., 2025).
In recent years, significant progress has been made towards this goal. The success of multimodal learning approaches has highlighted the potential of joint training paradigms (Radford et al., 2021; Yin et al., 2024; Liu et al., 2024b; Dubey et al., 2024; Team et al., 2023; 2025), while a parallel line of research has provided evidence that even unimodal models, when trained at scale, develop representations with strong structural similarities (Huh et al., 2024; Maniparambil et al., 2024a; Tjandrasuwita et al., 2025). This observation has given rise to the Platonic Representation Hypothe- sis (PRH), which posits that neural networks converge towards a shared statistical model of reality in their latent spaces (Huh et al., 2024). Building this, several recent works have established wide range of techniques to evaluate and exploit cross-modal alignment, in both zero-shot and few-shot scenarios (Maniparambil al., 2024a; Jha et al., 2025; Schnaus al., 2025; Hadgi et al., 2025).
However, prior work has focused almost exclusively on the alignment of static modalities: primarily images and text. In other words, although the Platonic Representation Hypothesis has been stated in full generality, so far, its validity has only been evaluated on static data, leaving its applicability dynamic. temporal domains an open question. Consequently, the rich information contained in video: motion, causality and temporal dependencies, exhibited, e.g., in intricate human interactions
Work performed while the author Google DeepMind.

---

Published as a conference paper at ICLR 2026
(Gu et al., 2018; Wang et al., 2023), has been largely overlooked in the context of representation alignment.
A notable limitation of previous studies focusing on static modalities was raised in Huh et al. (2024), where the authors pointed out: "the maximum theoretical value for the alignment metric is 1. Is a score of 0.16 indicative of strong alignment [...] does it signify poor alignment with major differ- ences left to explain? We leave this as an open question." In this work, we provide a partial answer to this open question by showing that the limited alignment observed previously is, in large part, due to the impoverished data given al test time. Specifically, we demonstrate that by considering mul- tiple video frames instead of a single image, as well as a diverse set of captions instead of a single annotation, the alignment score can be improved significantly, achieving close to 0.4 in some cases, without modifying the underlying trained models. This result highlights, for the first time, that large improvements in alignment can be achieved through efforts at test time, which is complementary to the training-timn -aminng-timeete resources (model size, amount of training data, etc.) considered in prior work. also suggests that multi-frame approaches can capture different aspects of a scene better, and points to the possibility of a strong zero-shot evaluation metric capable of probing the representation power of both image and video encoders.
More broadly, in this work we extend previous cross-modal alignment studies into the temporal domain by conducting the first comprchensive investigation of video-text representation similarity. We introduce a robust evaluation framework to probe and compare the capabilities of 121 modern video and language models. Our findings reveal that the rich, temporal nature of video provides powerful signal for semantic grounding. Specifically, we demonstrate that (1) state-of-the-ee self- supervised video encoders, e.g., VideoMAEv2 (Wang al., 2023), achieve competitive alignment with text compared to top-performing image encoders, e.g., DINOv2 (Oquab et al., 2023); (2) we observe that alignment quality is highly sensitive to the richness of the provided visual and textual data, and (3) there is non- conviaiiial relation between the video-text alignment and the performance of video models on downstream tasks. Finally, we highlight challenging situations with hard negative examples, in which current video and text models struggle, leaving room for future improvement.
2 RELATED WORK
Our work is situated at the intersection of three domains: the study of emergent representation align- ment, self-supervised video understanding. and multimodal learning. We first review the recent work in representation convergence, then discuss the most prominent paradigms for learning from video data, and finally, contextualize our work by highlighting the specific gap probing the alignment of unimodal trained video models.
The Platonic Representation Hypothesis and Emergent Aligmment in Static Modalities A foundational concept for our investigation is the Platonic Representation Hypothesis (PRH), intro- duced by Huh et al. (2024). The PRH posits that as neural networks are scaled terms of capacity, data diversity, and task variety, their learned internal representations converge toward shared, uni- versal statistical model of reality. This converged latent structure is termed the "Platonic representa- tion," drawing an analogy to Plato's 's notion of an ideal reality that underlies our sensory observations. This hypothesis has been empirically tested by measuring the geometric similarity of representation spaces, obtained using a wide range of encoders. The primary method for this is comparing the representational kernels, which characterize how models measure dissimilarity between data points (Kornblith et al., 2019; Maniparambil et al., 2024b; Huh et al., 2024). The key finding supporting the PRH is that as unimodal vision and language models become more capable, the structure of their latent spaces becomes increasingly similar, even without explicit cross-modke supervision.
This convergence has been most robustly demonstrated in the alignment of static modalities, primar- ily between images and text (Maniparambil al., 2024b; Tjandrasuwita et al., 2025). For example, the work of Maniparambil et al. (2024b) showed that powerful, independently trained vision and language encoders develop representations with a surprisingly high degree of semantic similarity. Furthermore, following prior work (Merullo et al., 2022), the authors of Maniparambil et al. (2024b) showed that these disparate latent spaces may differ only by a simple, learnable linear transforma- tion. A more fundamental principle behind these and related results was offered in Huh et al. (2024),
Our project page with code is https: -videe--hh tit....btt is.
2

---

Published as a conference paper at ICLR 2026
which posited that since all modalities are projections of the same physical world, models trained at sufficient scale will inevitably converge on similar latent structures that reflect this shared reality (Huh et al., 2024; Tjandrasuwita et al., 2025). The vision encoders used to validate these claims are typically state-of-the-ar self-superrvsed models like DINOv2 (Oquab et al., 2023). Recent work has also examined the conditions and methods related to this emergent alignment (Tjandrasuwita al., 2025). Cross-modal alignment has also been assessed via "alignment probing," " which correlates emergent alignment potential with the representation's 's clustering quality (k-NN performance) over linear separability (Zhang et al., 2025). The underlying universal structure has even been shown to be sufficiently strong to enable both unsupervised and weakly supervised translation between dis- parate embedding spaces (Jha et al., 2025; Zhang et al., 2025; Schnaus et al., 2025). While these studies investigate and exploit emergent alignment, they focus almost exclusively on static modal- ities. Our work complements these findings by conducting the first systematic probe of emergent alignment in the temporal domain, and shedding light on the dependence of alignment on test-time data in both text and visual domains.
Learning Representations from Video Learning from video data presents unique challenges and opportunities due to the temporal dimension. Research in this area is divided into unimodal self-supervised learning, which focuses on spatiotemporal structure, and joint ladgu-la pre- training. which focuses explicit semantic alignment, e.g., Xu et al. (2021); Zhao et al. (2024). In the former category, masked autoencoding has proven highly effective (Tong et al., 2022; Bardes et al., 2024; Wang al., 2023). This paradigm, inspired by its success in language and image do- mains, involves masking portions of the input video and training a model to reconstruct the missing content in either pixel or feature spaces. Although initial approaches in masked video represen- tation learning have focused on modest model sizes, recent efforts have also been made to scale such models to the billion parameter regime (Wang et al.. 2023; Carreira al., 2024; Assran al., 2025). Critically, such models are pre-trained on vast, unlabeled video datasets with self- self-supervised (e.g., reconstruction) objectives. For example, VideoMAEv2 has no exposure to textual supervision, yet it learns powerful spatiotemporal features that achieve top performance on downstream action understanding tasks (Wang et al., 2023). This makes it the idcal subject for probing the existence of emergent semantic alignment. Furthermore, there is recent evidence that even unimodal video models, when trained at scale, tend to learn features that are useful in a broad range of tasks (Wu et al., 2025; Carreira al., 2024). We also note that evaluating self-supervised video representations is challenging, and current approaches rely on expensive task-specific training ( (Wang et al., 2023; Carreira et al., 2024; Hasson al., 2025). There is thus strong need for zero-shot metrics with strong predictive power for video models.
Bridging Unimodal Video and Text: Prohing for Emergent Alignment Given the success of powerful unimodal encoders in both static (Oquab et al., 2023) and temporal (Wang et al., 2023) domains, and the evidence for emergent alignment in static data (Maniparambil natural question arises: does the Platonic Representation Hypothesis extend to the temporal domain? While our work is the first to conduct a systematic probe of this question, we acknowledge related efforts that bridge video and text representations, e.g., (Kim et al., 2023; Liu et al., 2024a; Li et al., 2025). However, existing works this domain typically rely on an existing source of alignment rather than investigating the intrinsic properties of video encoders.
al., 2024b),
3 OUR APPROACH
At high-level, our approach follows the methodology introduced in Huh et al. (2024), which uses a mutual k-NN metric to measure the similarity across different modalities. We adapt this approach to our setting, extending it to measure the multi-frame (or multi-clip) and multi-caption similarity of video-text pairs. Our setting is illustrated in Figure 1.
At its core, the Platonic representation alignment is calculated in the following stages.
Dataset. We assume a test set of N video-caption pairs: S = (V, = {(v1, ci),.., ., (UN, eN) )- Here U; is video, and e, is a corresponding set of text captions for this video. Each caption cij in the set q; provides a textual description of the given video U,.
Encoding. We embed each video using a video encoder into some video embedding space Evia (1:) = v, E Rp. Similarly, we encode each set of captions e into a text embedding space using text encoder: ((e)((()) c, E R2. Importantly, the dimensionality of the embedding
3

---

Published as a conference paper at ICLR 2026
Vidco ambokding space
video model
Sample zy Eramics uniformly
Video Finhekling
Video Vi
mutual N-NN
large language
Text ambolding space
Captions
caption
Sample Dz captions and concatoratic
cap
cap
model
Text Embodding Cj
Figure 1: Scaling both the number of video frames and text captions at test time improves alignment. Given a paired video U; and set of captions Cr leveraging rich multi-frame and multi- caption information at test time leads improved alignment, measured in terms of mutual k-NN.
spaces, and y are typically different, making it difficult to directly compare distances. We de- fine X RN and Y Nx to be the matrices obtained by stacking all of the video and text embeddings respectively. For simplicity denote X = (v) and Y (tex(()).
Alignment metric. The Mutual k-NN (MKNN) metric introduced in Huh et al. (2024) measures the agreement between the nearest neighbor structure in two different embedding spaces. Given two feature sets X R Rgxx Nxp, P Y E R x2 we first construct two binary indicator matrices Mx and if element (row) is among the k-nearest neighbors of
My, both of size N x N. s.t., (Mx)ay element (row) i in space X, and 0 otherwise. The Mutual k-NN alignment is then calculated as:
N
A MRNN (X,Y)
Mxx C My)
KN
Here, denotes the Hadamard product (element-wise multiplication) of the two indicator matrices. Observe that the operation in Eq. (1) represents simply the mean overlap (alignment) in the sets of k nearest neighbors, normalized by a hyperparameter (typically set to 10 for a dataset of 1024 examples). Additionally, we also follow previous work and optimize over the choice of intermediate layers for both encoders, and pick the pair of layers that maximizes the alignment score.
Incorporating multi-instance data. As mentioned above, main goal is to extend previous ap- proaches, which focused on static instances to datasets of videos (treated as multi-frame sequences) paired with potentially multiple diverse captions; Sec. for detailed description of our data sources. For a video V; and a given video encoder Evid. which natively processes clips with n, frames, we extract the indices of y frames through uniform linear interpolation study the effect of using an increasing number of frames n at test-time. Using one frame (n, 1) corresponds the setup used prior works that focus on image-text alignment. For videos longer than no. we use nearest neighbor interpolation to extract increasing numbers of frames that are multiple of no. e.g. for n, = 16, we consider n [16, 32, 64, 80}. We pass the le-length sub-clips through the encoder and then average the representations over sub-clips.
Similarly, for set of captions Cty associated with given video v, we can use anywhere from one caption to the full set of captions. We concatenate the set of selected captions into single string and use text-based encoders (including LLM-based ones as discussed below) to extract their intermediate features. When encoder produces per-token embedding, we average the features along the token dimension to obtain a [layer, hidden connension-sshppe feature, associated with each video.
DATA AND MODELS
Data, We evaluate video-text alignment on several datasets that contain paired video/text data. Our main investigations use VATEX and the Perception Encoder Video Datasets (PVD) (Wang et al., 2019; Bolya et al., 2025). We construct two test sets using 1024 videos randomly sampled from VATEX and PVD. Each video in VATEX lasts around 10 seconds and is taken from unique YouTube video. They are captioned by 10 different annotators in English and Chinese (we only use the English captions). In other words, each video contains 10 different captions (each caption is produced by a different annotator and is typically a sentence with around 15 words on average). This provides rich source of diversity which
use to vary the amount of textual information used for
4

---

Published as a conference paper at ICLR 2026
LIM Mide
LUM Noselc
Aligmmett Scare Gemmaa Ji--T (Va (VaTex))
0.25
pemmaz Sb-t: 0.07
gemma (r 2,011
lame 1t = 0.871
drvik Matc
Woomast
gemmer 9: 1 6.111
gemme To y 0.821
lana 70 0.81)
platt
Alemmaeva Fuge
dree? glart
0.20
ralies diea mosel mage mosel video
Mode lpps
dhoup
driy sert
Grovs Ifgg
rattee diss mozel
Image mosel viteo Irage mocel image
Mode: Tapes
drodd hinge lag
droo2 gani
Molooryyti Torog
sicommaig Mage
mage mocel image
Ir32
witted 1oo1-20
Tagu $2x
da huge nizt
0.25
de lene
cipy iiroe
choed smal
pana ahouz
digge Tart ardi nize
cong jang
1862 100
Viput gu
Tatuu Highig
sugnng
dug Jarge
0.20
augma
Sugell Ding
pe,00m lang
0.05
maR
vidcomee Dar 2902
avg. video Image
2vg. video Image d: +0.014
0.225 5SV2 Retrieval Accuracy
0,250
K400 Retrieval Accuracy
Figure 2: Video representations are strongly aligned with text. We measure alignment between modern vision encoders and Gemma 2 9B-it (subset of all vision models shown for clarity) on the VATEX video dataset using a single caption for each video. In addition to the points, we plot linear regression of alignment scores to representation strength for three different LLMs. Our takeaways are threefold: (1) The strongest models for both alignment and retrieval are large video models
(2) Averaging over frames is a simple yet effective baseline for image models on video input (4), while image-only models are limited to scores below 22% (m), in line with Huh et al. (2024). (3) Recent text models have improved alignment with vision models, shown by the regression lines.
alignment. To achieve a similar effect with PVD, we use Gemini-2.5 Pro to split each fine-grained caption into 10 separate captions in similar style to VATEX; see See A.3 for more details.
Models. We experimented with wide range of both video and image encoders, as well as aggre- gation strategies. For image models, we consider two variants: first, we simply use a single (first) frame (A); second, to introduce naive temporal dynamics, we averaged the image features across 8 frames in the temporal dimension. We denote the resulting method as image model on video (m).
For video models (o), we consider a range of encoders, including sulf-supervised ones like Video- MAE and VideoMAEv2 (Tong et al., 2022; Wang et al., 2023), as well as partially text-supervised ones like VideoPrism (Zhao et al., 2024). We also evaluate recent vision models, such as the Percep- tion Encoder and DINOv3 (Bolya et al., 2025; Siméoni et al., 2025). We make a distinction between text-aligned and self-supervised models and use the former as an upper bound for alignment evaluation. For image models which are trained on video data, i.c., Perception Encoder (Bolya et al., 2025), denote them video models when tested video input. In total, test 85 different vision models and variants; see full list in the Appendix, Section A.2.
For language models, experimented with broad range of encoders, starting with the ones sidered in Huh et al. (2024). We also consider recent unimodal language models from the Gemma series (Team al., 2024b) and evaluate their potential as text encoders. Given caption, it with a text encoder and then average the representation over the token sequence. As mentioned above, we store intermediate representations across all layers of the encoder and select the best pair of layers for each pair of vision and text models when measuring alignment.
encode
5 VIDEO-TEXT ALIGNMENT RESULTS
Our first results on the VATEX dataset (Wang et al., 2019) are summarized in Figure 2. These results are obtained by using a randomly selected single (out of 10) caption for each video. We evaluate video understanding performance using nearest neighbour video retrieval of the Kinetics-400 and SSv2 test sets, respectively (Kay et al., 2017: Goyal et al., 2017). We use weighted N--N accuracy as retrieval metric following NPID and DINO (Wu et al., 2018; Caron et al., 2021); note that this is different from our Mutual k-NN alignment metric. We sweep over the number of nearest neighbours and find 8 to give good results for majority of the models for
10,000-video subsets
5

---

Published as a conference paper at ICLR 2026
Allgnment Scores (VaTeXX
0.40
Model Video lluge pivivz damm:
0.35
Captions |Rot 11 10 caption captiom
0.30
captions
captions
0.25
0.20
2
16
32 40
4 Number of Captions
8 10
Number of Frames
Figure 3: Vision-text alignment scales strongly with the amount of visual and textual data available at test time. (left) Providing more frames increases vision-text alignment for both image and video models, with the latter being able to take advantage of more frames more effectively. (right) Providing more captions to the text model also significantly boosts alignment with vision models, across all frame counts.
the retrieval task. We also report the alignment of the same vision models against wide range of language encoders on the VATEX dataset in the Appendix, Figure 11.
We make several observations: first, when considering pure image and text models that were stud- ied in prior work Huh et al. (2024), the alignment scores that we obtain closely follow previously reported values other datasets (e.g., the alignment score between the best image model DINOv2 and the best text encoder outside of the Gemma family is 0.18). Secondly, we note that more re- cent text models the Gemma-2 family lead better alignment with video models, even if trained purely for text generation, highlighting the utility of these models as text encoders. Simply us- ing a powerful
encoder already increases the best image-text alignment score to approximately 0.206. This is consistent with, e.g., Zhang et al. (2025) which has highlighted the importance of the language model for multimodal alignment. Third, we note that basic temporal averaging across multiple frames with powerful image models exhibits remarkably high video-text alignment reach- ing alignment of approximately 0.223. Finally, there is very wide range of text-alignment scores across video models. Nevertheless, the highest alignment is achieved with self-supervised Video- MAEv2 model. This last result notable, since shows, both that natively-traine video models outperform extremely strong image-based models (DINOv2) in text alignment, and that tempo- ral dynamics play an important role semantic grounding.
Overall, these results above point the potential, first, of using multi-frame video data for text align- ment, and second, of using language models for computing informative embeddings. Since in all of experiments, Gemma-based encoders achieved the best alignment, subsequent experiments focus on that particular text encoder and investigate the role of vision encoders in text alignment.
VIDEO-TEXT ALIGNMENT AND DATA DEPENDENCE
key observation of our work that the quality of the vision-text alignment depends strongly on the amount of visual and caption data given at test time. This is in direct complement to the focus of previous works that analyzed the dependence of the alignment on the size of the pre-training datasets or number of trainable parameters in different models.
Figure illustrates the dependence of vision-text alignment scores (against the Gemma 2-9b-it text encoder) on the number of frames from video and the number of associated captions in the VATEX dataset (Wang et al., 2019). Figure (left) demonstrates how the average alignment score changes with an increasing number of frames for different caption counts and two vision models: Video- MAEv2 (Wang et al., 2023) and DINOv2 (Oquab et al., 2023). Observe that across all settings, the alignment scores generally increases more frames are incorporated, suggesting that richer temporal context from the video leads to better alignment with textual descriptions.
Figure (right) complements the first by showing the alignment score's dependency the number of captions used, for the same vision models and a selection of frame counts. Increasing the number of captions leads significant improvement in alignment scores, particularly when starting with

---

Published as a conference paper at ICLR 2026
small number of captions. We note that since in the VATEX dataset each caption corresponds to a description of the same video provided by a different user, increasing the number of captions increases both the coverage of the captured visual concepts, as well as the diversity of perspectives on the same concept and event. In Section A, we show alignment scores against multiple Gemma models when using single vs all captions in VATEX, reinforcing this same observation (Figure 12).
These plots collectively underscore the importance of visual (frames) and textual (captions) data quantity in achieving high vision-text alignment, with distinct performance characteristics associ- ated with different vision models. We emphasize that our results strongly complement the prelim- inary experiments in Sec. 6 of Huh et al. (2024), relating information density to alignment. We investigate, for the first time, using both multi-frame video data and multiple complementary cap- tions, as opposed to a single short description distilled from longer one as done that work. We observe that having diverse captions, possibly describing the same visual concept in different ways, can boost alignment significantly. This effect quantified Figure 7, where a linear fit finds that going from to 10 captions improves alignment by 60% average.
To validate this behavior further, we demonstrate that even artificially synthesized text descriptions from single long video caption can obtain alignment that is higher than that of the source caption. Namely, in Section A.3, we find that our test time scaling approach improves alignment on the PVD dataset (Bolya et al., 2025), which does not naturally come with multiple diverse captions (Figure 8).
Test-time Scaling Laws Building on the empirical observations in Figure 3, we also quantify these dependencies using a predictive parametric model. We tested several formulations, but found that a saturation-based model (Eq. 2) provided the best fit by a significant margin:
score(ny, n.) = 5.x - (C,m7" + CCn,"))
(2)
Here ny and n, are the number of video frames and text captions given to a specific pair of vision and text encoders (as in Fig. 1). Soe represents the theoretical saturation score corresponding to ideal alignment of this vision/text model pair, whereas Cy, Ce a and |3 are fitted scalar parameters. This formulation achieves remarkably high coefficients of determination for both VideoMAEv2 (R² = 0.9791) and DINOv2 (R² = 0.9964), with strong predictive power for test-time data scaling against the Gemma-2 text model.
The fitted parameters are also highly informative: VideoMAEv2 (Sxx ~ 0.41, Cy = 0.15, C. = 0.13, a 0.75, 3 1.30) versus DINOv2 (5.0 0.37, C 0.05, C. 0.13, a 1.76, 1.4). Notably, the frame coefficient (Cy) for VideoMAEv2 is nearly triple that of DINOv2, while the caption coefficients (C₂) remain comparable, highlighting the video model's stronger ability to leverage temporal information from additional frames to improve aligmment.
This parametric analysis resembles compute-optimal scaling laws, such those identified for pre- training, e.g., in Hoffmann et al. (2022), which model the dependency of loss function (e.g., Negative LogeLikelihood) on compute or data size. Whereas those laws are formulated in terms of a loss metric (where lower is better), our metric is alignment score (where higher is better), making our additive saturation model conceptually equivalent. The saturation score So (e.g.. 0.41 for VideoMAEv2) can be interpreted the base accuracy of an ideal alignment process, while the subtracted terms are the respective eтror penalties incurred by having a finite approximation of the data at inference time. We provide additional justification for scaling laws in Appendix, Section B.
We note that such predictive models for test-time scaling can be useful to design strategies for multi- modal data acquisition (e.g., when collecting multiple high-quality annotations of video data, which can be costly), as well as to compare the ability of different encoders to incorporate diverse data modalities. We include test-time scaling laws for other vision models in the Appendix, Section A.4.
7 CROSS-MODAL ALIGNMENT AND DOWNSTREAM PERFORMANCE
Previous works, including Huh et al. (2024) and Maniparambil et al. (2024a), have highlighted that multimodal alignment between images and text is correlated with performance unimodal tasks, ie., image models that perform well on pure vision tasks (e.g.. depth estimation) tend to also align well with language. Our goal is to evaluate whether similar claim holds for video encoders.
7

---

Published as a conference paper at ICLR 2026
Pearson Carn: R- 0.883, p 2002
Pourson Corr. It 523, p- 0.000
Peargo CUT 0.256, 2 0.222
6.07
8.05
6:35
6:35
3.04
0.92
0.45
550
255
0.60
065
0.70
6:20
6.3:
3.00
6.92
0.55
0.76 5.71 076 6.79 200 cul 6.22 552
1e12 ecion dauiitatie pigher betiet
Entia stlln Siaal thgter belle)
peseation Ind pear, inoddn lligne boller
Fearac Car: R- 6,832. 2.022
Neurion Corr Rt 0.52. 0.000
Pearson Cor:: & 0.20 0.02
0:27
Alpeen Sare
0.35
2.05
504
5.70
6.75
5.70
n2l mutimttion Lower
ddeect tracking (nihtr buller
Figure 4: Correlation between video/text alignment and downstream video perception task perfor- mance, for SSL methods trained without text supervision.
We take advantage of recent work that trained large-scale unimodal video encoders and evaluate their performance various semantic and non-semantic downstream tasks. We focus video models trained without explicit text supervision, in order to better analyze the emergent alignment between representations, as opposed to models explicitly trained for video-language alignment (e.g. through video-text contrastive losses). In particular, we consider the best performing video models reported in Carreira et al. (2024) and correlate their vision-text featuur aligument against Gemma 2-9b-it on the VATEX dataset to their accuracy on video analysis tasks: point tracking on the Perception Test dataset (Patraucean et al., 2023), box tracking on the Waymo Open dataset (Sun al., 2020), camera pose estimation on RealEstate 10k (Zhou et al., 201800181 1b). ScanNet depth estimation (Dai et al., 2017), and action classification on the SSv2 (Goyal et al., 2017) and Kineties-700-200 Smaira et al., 2020) datasets. Downstream accuracy is computed by training a separate learnable attention- based decoder on top of the frozen video features for each task.
(Kay et al., 2017:
Figure 4 summarizes our key results. Observe that generally for self- self-supervised video models, there a strong positive correlation between the cross- modal alignment scores and semantic tasks perfor- mance such as action classification on SSv2 and Kinetics. Interestingly, there is also significant correlation between the alignment score and the accuracy on non-semantic perception tasks such as camera pose estimation, depth prediction, and object tracking. The point tracking task represents a notable exception, with weak correlation between text alignment and downstream performance. This can potentially be due to the highly local nature of the point tracking task, and also suggests room for improvement in terms of truly general purpose video encoders.
Overall, these results suggest that video-text alignment could potentially be used a powerful zero- shot metrie for probing the quality of video representations as an alternative or complementary to more expensive evaluation techniques that require repeatedly training multiple cross-modal decoders during sele-supe video model development.
8 TEMPORAL ANALYSIS & CROSS ALIGNMENT
Given that both modalities that we focus on follow the arrow of time, we can study how their repre- sentations encode temporal ordering via cross-modal alignment. We analyze temporal alignment on two datasets: the simple, synthetic Test of Time (Bagad et al., 2023) and the challenging long-form VideoComp (Kim et al., 2025).
Test of Time. The synthetic dataset in Test of Time was specifically designed to probe the temporal abilities of video models. It contains 180 synthetic (video, caption) pairs, with captions of the form "A C₁ circle appears (after, before} a C2 circle," " where Cı, 02 are selected colors, and the videos are 30 frame clips enacting the caption with the shapes appearing in the same region of black square (upper left, upper right, bottom left, bottom right). For each choice of shape and {cz, c2). there four pairs of related captions: two for each choice of preposition and color order. Within these, there are two sets of logically equivalent captions (c.g., ej before c2 and C2 after cj), for which their videos may be identical or different depending on the location of the shapes.
8

---

Published as a conference paper at ICLR 2026
Test of Time Ali Alignment arross different NNN
Alinmment againnt Fositive vs. Negative Captions (VideeComp Temporal Seorderf
0.30
Score Typs
cal
0.20
0:25
gnot vides
pe care, Vilcoman puge, Poeetunnt, kinvtess
vidsomaeen, Kiwntcs000
0:25
onertine ner Vialeceet
Fasterst, kentrt viturtreek vaurreed.nn ai tradlate y01
Figure 5: Video and text eventually are aligned, but encode temporal information differently. While the video text alignment is largely perfect for all of the models once k = 3, they differ heavily before that when looking at k = 1, 2. Text models are often bag-of-words, while video models differ.
Figure 6: Video embeddings are sensitive temporal caption reordering. Video-text alignment drops against the temporal reorder negatives in VideoComp, with the most aligned video representations having the largest such drop (models towards the right).
We use video-text alignment on this dataset understand how sensitive text and vision models are to this temporal reordering. We find that when we take k = 3 neighbors, many models get near perfect alignment as expected, since each example has distinetly closer neighbors (Figure 5). However, the order in which each modality ranks these three neighbors differs significantly, as shown by the differing = 1, 2 alignments.
We find that the language models tend to rank the neighbor which has the same words but ordered differently first, i.e. for "A 61 square appears after 62 square", the closest neighbor will be "A 02 square appears after C1 square." The other two related captions, "c1 before c2" and "cy before C1 the logically equivalent statement, are equally close to the original caption. Thus, in this case, LLMs measure closeness more akin to a bag-of-words than being temporally sensitive, at least at the shallower layers from where we are extracting the features.
VideoComp. We also test multimodal aligmment on VideoComp (Kim al., 2025). We use Video- Comp's 's test set, sourced from YouTube and based on ActivityNet Captions and YouCook2 (Krishna al., 2017; Zhou al., 20) We use the "temporal reorder" examples which compare caption of a video against a negative caption describing the same events in a different order. This results in 5 122 video-caption pairs, so we use lower k 5 for these experiments.
To test the sensitivity to such negatives, we first compute the standard alignment using the text embeddings of positíve captions. Then, we recompute neighbors in the text space, by considering the negative of each caption, say č; and computing its neighbors to the positive captions of other videos. Since negative captions are still from the same overall video, just temporally shuffled, this tests the model is still aligned with the content in temporally-agnostic manner. As shown in Figure 6, the alignment lower with the negative captions than the positive captions, but not significantly. Notably, models with larger alignment suffer more of drop, suggesting that these models may be learning temporally-awe
structures which are somewhat perturbed by the negative. However, there is still room for further improvement in their temporal awareness.
Cross-model alignment. We also investigated the alignment of features across different video mod- els, and report the results in the Appendix Section A.5. We observe that text-supervised and self- supervised models, predictably, form clusters in pairwise alignment. However, interestingly, we note that some models (such DINOv2) able span multiple clusters, and we hypothesize that such alignment against multiple models provides a strong indicator of the versatility of a given vision model. We leave a complete investigation of this phenomenon as interesting future work.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
In this work, we conduct the first comprehensive study extending the Platonic Representation Hy- pothesis into the temporal domain. We demonstrate that alignment scores dramatically improve doubling in some cases simply by utilizing multiple video frames and diverse caption sets at infer- ence, without any retraining. We quantify this phenomenon with novel and accurate (R² > 0.98) saturation-baat based scaling law, which quantitatively confirms that native video models (like Video- MAEy2) are intrinsically better at using temporal information than static encoders (like DINOv2).

---

Published as a conference paper at ICLR 2026
Through an extensive analysis involving 121 vision and language models, show that alignment against text encoders strongly correlates with downstream performance on both semantic and non- semantic tasks, suggesting that vision-text alignment can be used as informative zero-shot metric to guide video model development. At the same time, our analysis sheds light on some limitations of existing pure video foundation models, showing that many video models outperformed by image models applied frame-by trame-b--rame. Finally, while generative models are a promising direction for video models, remains an open question of how we can best harness their latent representations for understanding, as currently their alignment to text is quite weak.
10 ACKNOWLEDGEMENTS
The authors would like to thank Joe Heyward, Dilara Gokay, Yana Hasson, Morgane Riviere, and Pauline Luc (Google DeepMind) for many useful comments and discussions, as well as for help in running experiments presented this work, and David Fan for cross referencing the WebSSL. results.
REPRODUCIBILITY STATEMENT
All our results can be reproduced using models and datasets hosted on HuggingFace or open-sourced by their authors; see Section A.2 in the Appendix for full list of models.
REFERENCES
Kelsey Allen, Carl Doersch, Guangyao Zhou, Mohammed Suhail, Danny Driess, Ignacio Rocco, Yulia Rubanova, Thomas Kipf, Mehdi SM Sajjadi, Kevin Murphy, et al. Direct motion models for assessing generated videos. arXiv preprint arXiv:2505.0024 2025. 19
Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. Proceedings of the IEEE/CVF international conference on computer vision, pp. 6836-6846, 2021. 16
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Am- mar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.0990 2025. 3,
Piyush Bagad, Makarand Tapaswi, and Cees G. M. Snoek. Test of Time: Instilling Video- Video-Language Models with a Sense of Time. In CVPR, 2023. 8
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXi::2102.067000 2021. 22, 23, 24.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.084F 084771 2024.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588):1788-1828, 2013. 23
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, pp. 813 824. PMLR, 2021.
16
Daniel Bolya, Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint 2504.1330,,,.11,, 2025, 4, 5, 7, 17
10

---

Published as a conference paper at ICLR 2026
Ramon Ferrer i Cancho and Ricard V Solé. The least effort and the origins of scaling in human language. Proceedings of the National Academy of Sciences, 100(3):788 199119, 2003. 24
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervieed vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021. 5
João Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahendran, Thomas Albert Keck, Joseph Heyward, Skanda Koppula, Etienne Pot, et al. Scaling 4d represen- tations. arXiv preprint arXiv:2412.15212 2024. 3, 8, 19
Joel Cohen and Meng Xu. Random sampling of skewed distributions implies taylor's power law of fluctuation scaling. Proceedings of the National Academy of Sciences, 112(25)7749-7754, 2015. 22
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017. 8
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Mare O Ernst and Heinrich H Bülthoff. Merging the senses into robust percept. Trends in cognitive sciences, 8(4):162-169, 2004.
David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rab- bat, Nicolas Ballas, Yann LeCun, Amir Bar, al. Scaling language- visual representation learning. arXiv preprint 25Xiv:2504.00017, 2025. 17
Pascale Fung, Yoram Bachrach, Asli Celikyilmaz, Kamalika Chaudhuri, Delong Chen, Willy Chung, Emmanuel Dupoux, Hongyu Gong, Hervé Jégou, Alessandro Lazaric, et al. Embodied ai agents: Modeling the world. arXiv preprint 2rXiv:2506.22355 2025.
Xinyang Geng and Hao Liu.
Openllama: An open reproduction of llama. arXiv preprint
arXiv:2307.12928, 2023. 17
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phat, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017. 5, 8
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et Ava: A video dataset of teatio-temporally localized atomic visual actions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6047-6056, 2018. 2
Souhail Hadgi, Luca Moschella, Andrea Santilli, Diego Gomez, Qixing Huang, Emanuele Rodolà, Simone Melzi, and Maks Ovsjanikov. Escaping plato's cave: Towards the alignment of 3d and text latent spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19825-19835, 2025. 1
Yana Hasson, Pauline Luc, Liliane Momeni, Maks Ovsjanikov, Guillaume Le Moing, Alina Kuznetsova, Ira Ktena, Jennifer Sun. Skanda Koppula, Dilara Gokay, et al. Scivid: Cross- domain evaluation of video models in scientific applications. arXiv preprint arXiv:2507.03578,
2025. 3
Trevor Hastie, Robert Tibshirani, Jerome Friedman, et al. The elements of statistical learning, 2009. 21
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022. 16

---

Published as a conference paper at ICLR 2026
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: arxiv:1712.004000 2017. 22
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. 7, 21, 22
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic represen- tation hypothesis. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20617-20642. PMLR, 21-27 Jul 2024. URL btttss:/prroeeeeee .ml. .tt.press././sp////........... v235 / hhth//hh2//// 4a html. 2, 3, 4, 5, 6, 21, 27
Rishi Jha, Collin Zhang, Vitaly Shmatikov, and John Morris. Harnessing the universal geometry of embeddings. arXiv preprint arXivv2505./2550, 2025. 1,3
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020. 22
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint (eo::/705.0900 2017. 5,8
Dahun Kim, AJ Piergiovanni, Ganesh Mallya, and Anelia Angelova. Videocomp: Advancing fine- grained compositional and temporal alignment in video-text models. In Proceedings of the Com- puter Vision and Pattern Recognition Conference, pp. 29060-29070, 2025. 8,9
Dahye Kim, Jungin Park, Jiyoung Lee, Seongheon Park, and Kwanghoon Sohn. Language-fres free training for zero-shot video grounding. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), pp. 1114-1122 202.2.3.......
Simon Korbllith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019. 2
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In International Conference on Computer Vision (ICCV), 2017. 9
Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Black- iston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196~ 210. 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221 1.X0v:221..05100, 2022. 17
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004. 23
Chau-Yi Yi Li, Chia-Hao Li, Pin- Yu Chen, and Chen-Kuo Chiang. ELIOT: A zero-shot generative video-to-texx retrieval framework, 2025. 3
Han Liu, Yinwei Wei, and Liqiang Nic. Dreaming user multimodal representation for micro-video recommendation, 2024a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference computer vision, pp. 216-233. Springer, 2024b.
12

---

Published as a conference paper at ICLR 2026
Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel C'Conor. Do vision and language encoders represent the world similarly? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14334-14333, 2024a. 1.7
Mayug Maniparambil. Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E O' O'Connor. Do vision and language encoders represent the world similarly? In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14334- 14343, 2024b. 2,3
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXi::200./51162, 2022. 2
Meta. Introducing llama 3.1; Our most capable models to date, 2024. URL https: aeim......... com/blog/ 11ama- aaaaama--ll-. 3-11. 17
Mark EJ Newman. Power laws, pareto distributions and Zipf's law. Contemporary physics, 46(5): 323-351, 2005. 22
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang. Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick I Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision, 2023. 2, 3, 6, 17
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Mateusz Malinowski, Yi Yang. Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing 36:427s,,6:4244885
2023. 8
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PmLR, 2021. 16
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140):1-61) 10000 17
2020.
Dominik Schnaus, Nikita Araslanov, and Daniel Cremers. It's a (blind) match! towards vision- language correspondence without parallel data. In Proceedings of the Computer Vision and Pat- tern Recognition Conference, pp. 24983-24992 2025. 1,3
Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint 2004v:2004. 10802, 2020. 22
Oriane Siméoni, Huy Vo, Maximilian Seitzer, Federico Baldassarre, and Maxime et al. Oquab. Dinov3, arXiv preprint 2rXiv:2508.101000 3(4), 2025. 5, 17
Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. A short note on the kinetics-700-2020 human action dataset, arXiv preprint arXiv:2010.10860 2020. 8
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.1022 2021. 16
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on com- pufer vision and pattern recognition, pp. 2446-2454 2020. 8
13

---

Published as a conference paper at ICLR 2026
Gemini Team, Rohan Anil, Sebastian Borgeaud, Sean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23/2.11805, 2023. 1
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprini urXiv:2033082295 2024a. 17
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint Xi1:24008.00116 2024b.
5, 17
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma 3 technical report. arXiv preprint arXiv:2503. 2025. 1, 17
Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin, and Paul Pu Liang. Understanding the emer- gence of multimodal representation alignment. arXiv preprint arXhv:2502.16282, 2025. 2,
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances neural information process- ing systems, 35:10078-10093, 2022. 3, 5, 17,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1rXi::2302.13977 2023 17
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. VideoMAE v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14549 14549-14560, 2023. 2, 3, 5, 6, 17, 19
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4581-4591, 2019. 4, 5,
Yao-Chih-Leee Wu, Yi-Ting Chen, Yi-Hsuan Tsai, and Wen-Hsiang Liao. VideoREPA: A video- based representation alignment framework for physics-enhanced video generation, 2025. 3
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3733-3742, 2018. 5
Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large multimodal agents: A survey, arXiv preprint 1511: 2024.
Hu Xu, Gargi Ghosh, Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pp. 6787-6800. Association for Computational Linguistics, 2021. URL https: aallatttanoooooooooo org/ / 2021. www.p-maaaeeemm.mmm 544.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11 11(12)nwae4033 2024.
Le Zhang, Qian Yang, and Aishwarya Agrawal. Assessing and learning alignment of unimodal vision and language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14604-14614, 2025. 3,
Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian. Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. arXiv preprint ar2iv: 13217, 2024. 3, 5, 19
14

---

Published as a conference paper at ICLR 2026
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018a. 9
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.0000 2018b. 8
George Kingsley Zipf. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio books, 2016. 22
A APPENDIX
The supplementary materials below provide additional results and illustrations that support and ex- tend the claims made in the main paper.
APPENDIX OVERVIEW
The Appendix is structured follows:
Section A.1: Additional Results. We provide additional results on the VATEX dataset, visualizing alignment between range of language and vision models. We show alignment when using both one caption and all captions from the VATEX dataset, highlighting the improvement across all vision and language models.
Section A.2: List of models. We give full list of models that we used in our study, including details on both vision and language models.
Section A.3: Details caption synthesis and PVD alignment scores. We provide the details and qualitative examples of our approach for creating synthetic captions on the PVD) dataset, and our resulting alignment scores.
Section A.4: Additional test-time scaling laws. We provide additional examples of our test-time scaling laws for other models and datasets.
Section A.5: Video-video model alignment. We illustrate alignment within different video models.
Section B: Theoretical justification for scaling laws. We justify our framework of scaling laws from first principles using two separate theoretical models drawn from literature.
15

---

Published as a conference paper at ICLR 2026
A.1 ADDITIONAL RESULTS
We include in Figure 11 the alignment scores of all the vision and text encoders considered in our analysis on the VATEX dataset; see the full list in Section A.2. The rows correspond to vision encoders and the columns correspond to language models. The label for each model has the for- Sruumlayeera
Each ij entry in the matrix contains the alignment score (also colour-coded: lighter shades correspond to higher align- ment scores) and a tuple (layer; layer,) indicating from which layers in vis ceiiinnmon masicnmmodell. 1_i and language sow.lageemmee1j..... respectively, the representations were extracted to get the best alignment for that pair; we recommend zooming in for better readability. We used a single caption for each video (out of 10) sampled randomly.
mat: <model <model_name>>>> __name>> (<max_a paaalignnn inmenttsscoree
As mentioned in Section 6, the alignment scores improve significantly when considering more video frames and/or more text captions. In Figure 12, we show additional re- sults that confirm this observation. We calculate align- ment scores between visual representations produced by subset of the vision encoders considered above and language representations given by different variants of text models when processing of the captions from the VATEX dataset. As before, the rows correspond to vision encoders and the columns correspond to language mod- consider: (I) image models
0.40
pwm.mn..l 21- 0.550)
LLN
video MacvA
SOEe gland
Score
0.35
Visinn Mode: Types rathe video mocel
rrage model in viceo
dinoe Huge
disvv gont
Multi Caption AligneentSon
0.30
irage modil mage
vicesmi webul 224
"AANTTEEE Jage
0.25
die arre
dravz Part
dinor Arrll
0.20
sugre
co ent
Whicks4400
0:15
els. For vision models, applied on a single frame of each video, (2) image mod- els applied every frame of the video and the represen- tations averaged over time, and (3) video models. When using a single caption, we observe that the best alignment scores are generally obtained by video models, which are able to encode not only features about the scene appear- ance, but also details about the scene dynamics, leading better alignment with caption representations. A no- table exception is the Perception Encoder Core Giant im- age model applied on all the frames of the video, which obtains the best alignment score, but is also contrastively trained on images, videos, and text.
pe ingss
0.10
drov smat"
0.05 auga
videomac
mat
0. 00
0.00
0.05
0.10
0.15
0.20
0.25
Single Caption Alignment Score
Figure Multiple captions provides consistent improvement in alignment. Going from to 10 captions improves alignment by 60%, based the best fit estimate.
However, when we consider all the ten captions available for each video, the alignment scores im- prove significantly overall, and several image models applied on the full videos or even on single frames outperform many video models. This is further reinforced when plot the alignment with one vs. ten captions in Figure 7. The line of best is y = 1.6x - 0.018 with r = 0.998, indicating that multi-caption data consistently improves alignment.
A.2 FULL LIST OF MODELS
In total, we analyze 121 models. We use the following 85 open-sourced visual models and their variants for our main analysis. All the models are available on HuggingFace or from the GitHub repositories of the authors. We use the class token when available, and fall back to averaging over all tokens as a backup. This performed best experimentally. Key results are shown in Figure 2.
AugReg [m, a] (Steiner et al., 2021): 8 variants of the image model across Tiny, Small, Base, and Large sizes, evaluated on image and video settings.
CLIP [m, a] (Radford et al., 2021): 12 variants of the image model across Base, Large, and
Huge sizes (including IN12k pre-training), evaluated on image and video settings.
MAE [m, a] (He et al., 2022): 6 variants of the image model across Base, Large, and Huge sizes, evaluated on image and video settings.
Timesformer [o] (Bertasius et al., 2021): video variants of ViT-Base, finetuned on either K400 or SSv2.
VIVIT [+] (Amab al., 2021): video variant (ViViT-B-16x2) finetuned Kinetics-400.
16

---

Published as a conference paper at ICLR 2026
VideoMAE [o] (Tong al., 2022): 3 video variants, including Base (on SSv2 and Kinetics) and Huge (on Kinetics).
VideoMAEv2 [o] (Wang et al., 2023): video variants across Base, Large, and Huge sizes.
DINOV2 [.. Δ] (Oquab et al., 2023): 8 variants of the image model across Small, Base, Large, and Giant sizes, evaluated on image and video settings.
WebSSL [, A] (Fan et al., 2025): 6 variants of the image model across IB, 5B, and 7B parameter sizes, evaluated on image and video settings.
PE (Perception Encoder) [m. e] (Bolya et al., 2025): 24 variants of the image model across Core, Language, and Spatial types with sizes from Tiny to Giant, evaluated on image and video settings.
V-JEPA [o] (Assran et al.. 2025): video variants covering Large, Huge, and Giant sizes with a 384 x 384 resolution Giant model as well.
DINOv3 [m, a (Siméoni al., 2025): 8 variants of the image model across Base, Large, Huge, and 7B sizes, evaluated on image and video settings.
We also look into 6 additional video models for their video-video alignment with each other: 4DS-c, TRAIAN, VideoMAEv2 k710 finetuned, V-JEPA v1, WALT, and VideoPrism (see Section A.5).
For language models, we experiment with the following 30 open-sourced models and their variants:
T5 (Raffel et al., 2020): variants including Small, Base, Large, 3B, and 11B.
BLOOM (Le Scao et al., 2022): 5 variants including 560M, 1.1B, 1.7B, 3B, and 7.1B.
Llama (Touvron et al., 2023): 4 variants including 7B, 13B, 30B, and 65B.
OpenI.lama (Geng Liu, 2023): variants of an open-source reproduction of Llama in- cluding 3B, 7B, and 13B.
Llama (Meta, 2024): variants in total: Llama 3.2-1B, Llama 3.2-3B, Llama 3.1-8B, and Llama 3.3-70B.
Gemma (Team al., 2024a): variants including 2B and 7B.
Gemma 2 (Team al., 2024b): instruction-tund variants including 2B-it, 9B-it, and 27B-it.
Gemma 3 (Team et al., 2025): instruction-tur variants including 1B-it, 4B-it, 12B-it, and 27B-it.
A.3 SYNTHETIZING CAPTIONS FOR PVD DATASET USING LLMS
To study the effect of including an increasing number of captions on the alignment score between video representations and the language representation of the associated captions, synthesized multiple captions for videos in the PVD dataset using LLMs, starting from the detailed caption provided in the original dataset. Importantly, encouraged the model only use details present in the original caption, and not hallucinate new possible descriptions. We relied on Gemini 2.5 Pro.
For reproducibility purposes, we provide below the prompt used generate these captions, which was obtained from the prompt given to the VLM model when collecting the annotations for the original PVD dataset. We also include qualitative example in Figure 9.
17

---

Published as a conference paper at ICLR 2026
Prompt for Caption Generation
Create ten concise captions a vidae using the provided detailed captions. Each 61 ferrenttt detaila present. the informaticn 10 different ways, but to get 10 different deseriptions all potentially containing fferent information.
caption shou
goal 1a summarize all of
TASK: Extract key information: from the capcions and combine it into set of ten captions, eech of which is single phrese or set of pheeses that includes some subset of the relevant details in alt text format. Steps to Follow:
caption
general
2. Extract the most selevant and concise information.
Don't include all the key information every captior. some information
to leave out between captions.
Combine extracted information into a alt text format using short phrase or set of phrases with approximately :20 takens, considering special characters like part of the token count.
special characters including everything each caption. 6. Pick only few things to includa in aach caption, it's okay to leave detat la out. What Avoid:
Minimize the
Avoid adding inferring information
in the original metadate. and
captions.
structures prioritizing sentence flow. Return 1ist separated only by new Lines, with only the captiona, nothing else.
Avoid using complex
We show the effect of adding more frames or using multiple captions at test time in Figure We re-iterate that Perception Encoder is trained on this dataset and sees text and videos together during training, but we include it as reference point regardless.
On the left, we plot the alignment beddings. We find that image-based models benefit slightly from this, but plateau quickly. Video models improve strongly with frames however, demonstrating their ability utilize more visual information. On the right, we plot the alignment scores as we add more synthesized captions. The dashed lines indicate the baseline alignment score of each vision model when considering the orig- inal detailed caption. We can observe that for all models, using even just 3 captions leads to better alignment compared to the baseline score. This saturates around 6 captions for most models, point- ing to potential redundancy between the synthesized captions beyond this number.
add more frames and average the resulting
0.40
Model
2e glart
Alinmmett Score (PVD)
d8062. gant
0.35
dhov2
ruge
0.30
Type
framee
Best score glven
0.55
Number of Captions
Figure 8: Test time scaling is effective for other datasets. We apply our test time scaling for PVD. (left) Scaling with more frames is generally helpful, but is especially important for video models to improve. Here, frames are sampled linearly spaced. (right) Scaling the number of captions at test time, even synthesized ones, outperforms using the original provided caption (dotted lines).
A.4 TEST-TIME SCALING LAWS FOR OTHER MODELS AND DATASETS
We include test time scaling laws for other models on VaTeX and PVD in Table 1, extending the results from Section 6. Overall, most of the models had high R²
18

---

Published as a conference paper at ICLR 2026
Detailed caption
The person is adding mayonnaise and vegetables to the burger. The person is also using wonden board make the burger. The video is tutorial on how to make burger. The person is wearing red and white checkered shirt. The video well lit and the colors are vibrant. The video great tutorial people who want learn how make a burger.
Synthesized captions
1. Person making burger.
2. Adding mayonnaise and vegetables to burger.
3. Burger assembled on wooden board.
4. Tutorial showing how to make burger.
5. Person a and white checkered shirt making burger.
6. Well-lit video with vibrant colors showing burger preparation.
7. A person makes a burger using a wooden board.
8. Tutorial adding mayonnaisc and vegetables a burger.
9. Person a red and white checkered shirt preparing burger.
10. Great tutorial for learning how to make burger.
Figure 9: Example of video frames, the original fine-grained model caption, and LLM-synthesized captions for a video in the PVD dataset.
A.5 CROSS-MODER (PAIRWISE VIDEO-VIDEO) ALIGNMENT)
Figure 10 shows the pairwise feature alignment between selection of video models, well as the Gemma2 text encoder. We include both MAF-based models (Tong et al., 2022), a Kinetics-finetuned VideoMABv2-G (Wang et al., 2023), self-supervised 4DS-e model (Carreira et al., 2024) well as language-supervised model (Zhao al., 2024), and a recent pure motion encoder TRAJAN (Allen al., 2025). We observe that there several clusters appearing across video models, which seem reflect baaguage-alignmann and spatio-tempon
awareness (especially among MAE-based models). We hypothesize that although language alignment along is correlated with downstream performance, cross-model alignment might hold more predictive power. Specifically, strong, general-purpee models must align across multiple clusters to be useful in broad range of downstream (including pixel or patch-level) applications.
19

---

Published as a conference paper at ICLR 2026
Table I: Comparison of Scaling Law Parameters for VaTeX and PVD
Model
Type
R²
Sec
Cy
a
Ce
B
VaTeX
0.9928
0.287
0.043
2.302
0.095
1.426
dinov2.base
PVD
0.9950
0.311
0.019
0.877
0.101
1.939
VaTeX
0.9964
0.365
0.046
1.762
0.132
1.400
dinov2.giant
PVD
0.9979
0.343
0.017
1.415
0.114
1.931
VaTeX
0.9955
0.339
0.046
1.852
0.120
1.354
dinov2.large
PVD
0.9970
0.329
0.013
3.704
0.114
2.046
VaTeX
0.9965
0.405
0.029
1.853
0.150
1.316
pe core giant
PVD
0.9976
0.417
0.015
2.238
0.148
1.454
pe core large
VaTeX 0.9710
0.109
0.014
1.695
0.027
1.396
VaTeX 0.9442
0.297
0.118
1.792
0.093
1.357
timesformer.base
PVD
0.9871
0.293
0.063
1.838
0.092
1.867
VaTeX
0.9788
0.334
0.156
0.451
0.088
1.221
videomaev2.base
PVD
0.9856
0.410
0.254
0.075
0.058
1.619
VaTeX
0.9791
0.410
0.146
0.748
0.128
1.302
videomaev2.huge
PVD
0.9878
0.296
0.092
0.479
0.079
1.939
videomaev2.large
PVD
0.9850
0.297
0.145
0.185
0.058
1.673
videoprism
VaTeX
0.9832
0.400
0.090
0.740
0.140
1.210
VaTeX PVD
0.9789
0.209
0.094
1.163
0.060
1.391
vivit b 16x2
0.9845
0.243
0.067
1.157
0.070
2.011
Alignment
0.40
425
1,00
0.06
0.56
0.56
0.23
0.30
0.34
0:10
0.00
0.55
0300
MideoMABV
0.56
1.00
0.28
0.4
0.07
Viteoet PuP 471. fivetuedd
0.56
0.06
0.60
1.00
0.21
0.27
0.03
0.22
0.14
WWTT
0.30
0.24
9.06
0.24
0.03
DROk2
0.56
0:34
0.15
1.00
0.57
0.10
205
Gemma?--ae #b-1t
0.20
0.00
0.07
0.34
0.06
0.03
0.37
1,00
0.00
405+
TRAAA
NEN
WULT
Ounmu 0b-E
Figure 10: Video encoders cluster into semantic and geometric specialities. We find that video- video alignment exhibits two clusters of models: those which are closer aligned to language and semantic tasks (bottom right), and those which are closer to geometric abilities (top left). However, some models are able span both axes, notably DINOv2 and VideoMAEv2 K710 finetuned.
20

---

Published as a conference paper at ICLR 2026
B THEORETICAL JUSTIFICATION FOR TEST-TIME SCALING LAWS
B.1 INTRODUCTION
Our paper observes close relationship between the amount of data provided test time and the resulting cross-modal alignment score between vision and text encoders, as measured by the Mu- tual k-NN metric introduced in Huh al. (2024). This relationship is captured by the following
parametric model:
(ny, ne) = S50 - mC + Come
(3)
This formulation achieves remarkably high coefficients of determination (A²² 0.98). In the paper, state this relationship empirical observation and call it "test-time scaling" law, taking inspiration from empirical training- time scaling laws, e.g., Hoffmann et al. (2022).
In this document, goal is provide theoretical justification for these scaling laws. For this, we proceed in the three major stages:
1. First, provide interpretation of the observed scaling laws, in terms of different con- stants involved, and their conceptual meaning.
2. We then provide a justification the particular form of the scaling law (why subtractive, why power-law, etc.).
3. Lastly, we deduce the empirical test-time scaling laws from theoretical models of how different encoders process data.
B.2 STAGE 1: INTERPRETATION OF THE SCALING LAW AND ITS CONSTANTS
The scaling law in Eq. equation 3 models the alignment score saturation process. As test- time data increases, performance approaches an asymptotic limit. The constants characterize the inherent limitations of the vision and language encoders and the efficiency with which they integrate information.
1. Saturation Score (:o See represents the theoretical maximum alignment score achiev- able by the specific pair of vision and text encoders, assuming access to infinite test-time information (ny - 30, n. + oc). This constant reflects the structural similarity of the latent spaces learned during training, related to the concept of "Platonic alignment" (Huh al., 2024). It analogous the irreducible error in standard learning theory (e.g., Hastie et al. (2009) Chap. 2.9).
2. Reducible Error: The term (Cyn7" + C.n.²) represents the "alignment gap." This is the penalty incurred due to the impoverished information provided by finite number of samples at inference time.
(a) Error Coefficients:
(C, and C.) These constants quantify the magnitude the re- ducible error attributable to the visual (Cy) and textual (C₂) modalities, respectively. larger Cy suggests the model has a greater capacity to utilize information from that modality (e.g., VideoMAEv2's utilization of temporal information compared to DINOv2). Another way of looking at these coefficients, is that they capture the de- pendence of approaching the irreducible error, using information from a particular modality. Note that when given one caption and one frame, our test-time scaling law becomes: S. Cc-
(b) Scaling Exponents: (a and (3) These exponents determine the rate at which the align- ment score approaches S.c. They reflect how efficiently the encoders integrate new information. A larger exponent implies faster saturation. For example, our finding that DINOv2 (image model) has higher a than VideoMAEv2 (video model) suggests that static models quickly extract relevant visual features but gain little from subsequent frames, whereas native video models integrate temporal information more effectively over time.
21

---

Published as a conference paper at ICLR 2026
B.3 STAGE 2: JUSTIFICATION FOR THE FORM OF THE SCALING LAW
The specific mathematical form of Equation 3 s-ttuuration, subtractive error terms, and power-law dependence-is consistent with established principles statistical learning and the broader litera- ture on neural network scaling laws (Hestness et al., 2017; Kaplan et al., 2020).
B.3.1 WHY SUBTRACTIVE FORM (SATURATION)?
The formulation Performance = Max Potential Error is necessary because the alignment score (Mutual k-NN) is increasing (higher is better) and bounded. As the aggregated representation be- comes richer, the eπror must decrease, and the score saturates at Sec. This structure is directly anal- ogous the scaling laws observed for training dynamics, notably the Chinchilla laws (Hoffmann et al., 2022), which model loss L as a function of model size N and dataset size D:
B D
D) = = No +
(4)
Here, E is the irreducible Since the alignment score is accuracy metric (higher is better) rather than a loss metric (lower is better), the subtractive form in our Equation 3 is the appropriate adaptation.
B.3.2 WHY POWER L.W??
The power-law relationship (n ") between error and resources (data size, model size, or, here, test- time samples) is ubiquitous in deep learning (Kaplan al., 2020; Bahri al., 2021). This arises due to several interconnected factors:
Diminishing Returns and Redundancy: In natural data (videos and language), samples are highly correlated. Initial frames or captions capture the most salient aspects, while subsequent samples often provide redundant information. power law effectively captures this smooth decay in marginal information gain (Bahri et al., 2021; Cohen & Xu, 2015).
2. Statistical Structure of Natural Data: Natural signals often exhibit heavy-tailed distri- butions of feature importance (e.g., Zipf's Law) (Zipf. 2016; Newman, 2005). Power-law scaling naturally emerges when learning from such distributions, shown in Section B.4 below.
3. Data Manifold Geometry: The complexity of the data manifold dictates how many sam- ples are required to approximate it accurately. Approximation theory suggests error scales power of the sample size, inversely related the intrinsic dimensionality (Sharma & Kaplan, 2020).
B.3.3 WHY ADDITIVE ERROR TERMS (Cf C.)?
Our model assumes the total error the sum of the error from limited frames and the error from limited captions: Etotal = (n ) + (n This implies that the information gained from adding more frames (e.g., capturing dynamics) is largely independent of the information gained from adding more captions (e.g., semantic diversity), different (vision and language) encoders. The remarkably high R² values reported the paper (> 0.98) validate that this additive model is an excellent empirical approximation.
these two sources of information are captured by two
B.4 STAGE 3: DEDUCTION FROM THEORETICAL MODELS
In addition to the interpretation provided above, we deduce the observed scaling laws from theoreti- cal models of how encoders acquire and process information. We note that observed power-law behavior is closely related to Tresolution-limitet
scaling described previous scaling law analy- (Bahri al., 2021). In this regime, performance is limited by how well the available resources (here, test-time samples) can resolve the details the underlying data structure. We present two complementary frameworks:
22

---

Published as a conference paper at ICLR 2026
B.4.1 STATISTICAL ESTIMATION SETUP
Idealized Representations and Alignment: For a given event, we assume there exist idealized, "true" representations in the vision (V*) and text (T*) embedding spaces. In the context of alignment, we assume that (V*) and text (T*) represent the idcalized alignable latent subspaces of the learned latent spaces of the two encoders, i.e., V = TT.
2. Noisy Estimation from Finite Samples: The representations produced by the two coders from a finite number of samples are noisy estimates of the ideal representation:
V(ny) = V + (n() T(ne) T (n,(nn)
(5)
Here, ny and rrr are zero-mean noise vectors (TE 0) representing the estimation error due to finite sampling.
3. Independence of Noise Processes: We assume that the noise process in the vision modality (πv) statistically independent of the noise process in the text modality (nT). This is plausible as the sources of variation (e.g., specific temporal sampling of frames or visual occlusion vs. linguistic ambiguity or phrasing variations captions) are distinct.
4. Alignment Metric and Mean Squared Error (MSE): We assume the alignment error E(n,, nz)) (the gap between the best possible alignment for the two encoders Soc and the achieved score) proportional to the Mean Squared Error (MSE) between the noisy es- timates. While the empirical metric is Mutual k-NN, MSE provides a tractable analytical proxy for representation distance.
E(ny, n.) = S - -Score(nn,,,) x E (m)) T(
5. Submanifold Hypothesis: Following previous literature on scaling law analysis, we sume that the data corresponding to an event (e.g., the temporal evolution of the video), as perceived by particular encoder lies on a d-dimensional submanifold within the high- dimensional embedding space (Bengio et al., 2013). This encoder-specific dimensionality reflects the variability the structure perceived by the encoder. Test-time data (n frames or captions) provides samples from this submanifold. The trained encoder aggregates these samples to estimate the precise representation or the local structure of the manifold corre- sponding to that specific event.
Deduction Part 1: Derivation of Additivity
The additive structure of the error terms follows from the independence of the noise processes. Specifically, decomposing the MSE and using the definition of the ideal alignment (V* T"),, we have:
[[V(())) - T(ne) E [I[ [I( + mv) (T' + mr |1
(8)
= E [[ 1|2 + || 1| 2n mt
(9)
E || ||]] + E |||||| - 2E nTTT
(10)
Since γL and yrr are assumed to be independent and zero-mean, the expectation of the cross-term zero: B[nEnn]] = |[m]]" (']nrl'E(r 0. Therefore, the total alignment error is the sum of the variances of the estimation errors in each modality:
E [[)) [[ [In(nn + [l)rr())))1]
(11)
Ev(ay)
Er(n.)
Deduction Part 2: Deriving the Power Law via Manifold Approximation
According approximation theory on manifolds, the error in estimating the structure of the man- ifold from samples typically scales as n -1/d, where the proportionality constant is related to the smoothness of the manifold or the specific task (see Levina & Bickel (2004), and Theorems 2 and 3 in Bahri et al. (2021) relating this behavior explicitly to neural network scaling laws). When using the Mean Squared Error (an analytic loss function minimized at the target), the error scales quadrat- ically with the distance. As detailed in Section 2.2.3 of Bahri et al. (2021), ) the estimation variance
23

---

Published as a conference paper at ICLR 2026
(population loss) scales E(n) & (Ar)² n 2/d Applying this result to the variance terms in our decomposition yields:
En & n 2/d, and (n x n. n/2//
(12)
where du and de are the effective dimensions as perceived by the vision and text encoders respec- tively. Substituting these back into our definition of the alignment metric, Eq. equation 7, and the additive error model, Eq. equation 11, we obtain the final scaling law:
Score(ny, ne) S
(Cmm + Cone
(13)
where the scaling exponents are given by a 2/d,, and B 2/d2.
Final Form and Interpretation:
Combining the additive structure derived from noise independence (Eq. 11) and the power-law vari- ance decay, we recover the empirical scaling law:
Score(ny; me) = 5.x - (Cyn + C)
(14)
Image Models (e.g., DINOv2): Image models largely ignore temporal relationships. The effective temporal dimensionality they perceive (d) is low. This results large a (fast saturation), additional frames quickly become redundant for approximating the static scene.
Video Models (e-g., ViveoMAEv2): Video models capture spatiotemporal features, per- ceiving higher effective temporal dimensionality (d). This results in smaller (slower saturation), as more frames are needed to map the temporal dynamics.
Another way to interpret this model is that the scaling exponents directly relate the redundancy of the information source as perceived by the encoder. For example, image models (e.g.. DINOv2) pri- marily capture static features; additional frames quickly become correlated regarding static content, leading to a relatively large a (faster saturation). Native video models (c.g., VidcoMAEv2) capture dynamic, spatiotemporal features where information unfolds time; this results in smaller (slower saturation), as more frames are required to reduce the variance related to temporal dynamics.
B.4.2 MODEL 2: MODALITY-SPECIRE FEATURE DYNAMICS
We also provide an alternative probabilistic model based on the statistics of natural data.
Assumptions:
1. Modality-Speeiffe Zipfian Distributions: this model, we assume countable universe of features indexed by k. Furthermore, the importance (weight) of these features depends on the modality, and follows a Zipfian distribution, characteristic of natural data (Cancho & Solé, 2003):
Vision Modality: Features have weights w following a Zipfian distribution w(v) oc -
Text Modality: Features have weights w following a Zipfian distribution (n(*) oc km
Crucially, the ranking of features may differ between modalities (c.g., visually salient background color may have high (²) but low (i())).
2. Independent Acquisition Priority: within their modality. Given n, frames, the video encoder captures the set of features Kv(n;) corresponding to the top-ranked visual features. We assume that the effective number of features captured in n, frames scales as [Kvl | x ny- Similarly, given n. cap- tions, the text encoder captures the set (r(n.) corresponding to the top-ranked semantic features, scaling as [Kr] & n., Note that this is analogous to assuming data points resolve the top D eigenfunctions in kernel analysis (Bahri al., 2021).
Encoders capture features in the order of importance
24

---

Published as a conference paper at ICLR 2026
3. Alignment Definition: The alignment score A(ny, n2)) is defined the total weight of features captured by both encoders, given n, frames and ne captions. Because alignment reflects the joint information capacity, the contribution of a matched feature k is function
its weights in both modalities, Su (wg w ) For the alignment error to be bounded by the tail of the distributions, we assume the total potential alignment So Σx 2g is finite.
Deduction:
The total Alignment error Enn, ne) represents the potential alignment mass that was missed be- cause either the video or text encoder failed to capture the necessary features. Formally:
nl, S00
Σ
S4
Σ
(15)
REKvnKт
RęKynKr
Using the Principle of InclusionExxlusion, the set of missed features is the union of features missed by Vision (My K&)) and features missed by Text (MT Kf). Thus:
Σ O
Σ Ok
Σ
(16)
KEMV
KEMP
KEM
Euid(ny)
Beaze( (ma)
Bovertop
Now, consider the vision term (n)) This represents the alignment potential lost because the video encoder did not capture the tail features. Even if we assume only loose correlation between visual and text weights, the total value of features in the "visual tail" is dominated by the power-law decay of the visual weights wo()) We approximate this tail sum using integral, which standard practice for analyzing the asymptotics of heavy-tailed distributions (formalized by the Suler-Maclaurin formula) and is commonly used in scaling law derivations (e.g., Bahri et (2021), Sec 2.3.3):
Evia(1s)) ( p dx x ny
- (1v-1)
[ny.
(17)
This yields the power law Cyn where (14o-1). 1). Applying the same logic, the text error term follows (ne)t(ne) Rs Cen where 3 = (Pe-11
The interaction term Eoverlap in Eq. 16 represents the mass of features missed by both encoders. In the asymptotic regime (large n),, we assume that the probability of a feature being in the deep tail of both distributions simultaneously of lower order than being in the tail of just one. Thus, Eowerlop < Emii + Erezt, and the additive approximation holds:
Score(n,, S.o (Cmnj" + ))
(18)
Final Form and Discussion:
Since E'ny, Re) = Soc A(my, ne), substituting the expression of E(ny, ne) obtained above, we
get: A(ny, ne) = Soe - (C + Con=)) &), which replicates the empirical scaling law described
in our paper.
Observe that there were several explicit assumptions made above, both in relation to the indepen- dence of feature acquisition processes, and associated with the negligeable weight corresponding to the interaction term Eoverlap- While these assumptions are consistent with prior work, a more care- ful investigation would shed better light on the exact ideal form (potentially involving more careful model of the interactions between modalities, suggested by this model) as interesting future work.
B.5 CONCLUSION
All three models provide justification for the observed test-time scaling laws. They demonstrate that the power-law form arises naturally from the statistical properties of natural data (heavy tails) and the geometric properties of the representation space (intrinsic dimensionality). consistent with the resolution-limited framework (Bahri et al., 2021). These models also allow us to connect the empir- ical parameters (Soo, a, 3) the fundamental capabilities of the encoders processing the data.
25

---

Published as a conference paper at ICLR 2026
Cloball Averoge D 0186
45
835
13
131 11,
175
53-
12
22 205 12%
115
120
18
121
2
28.
254
125
151 1.%
38
101 12%
201
21 121 (74
27 161
55%
11.
161
511 32%
11.
115.
242
2
27%
321
201
37%
371
372
117.55
21
0.5
34%
54%
35%
541
225
841
200
501
221
115.
Figure 11: We measure alignment between a large number of vision and text encoders on the VATEX dataset, using only one caption for alignment.
26

---

Published as a conference paper at ICLR 2026
Global Average: 0.2944
websii_dine1b 224 _ideo (Max: 0.2768) 1:40
8277
6217
E2TS
0258
E243
0247
0228
0223
dinov2_ base video (Max: 0.2756) 1:12
8228
8273
6274
0258
E3ET
E24
D223
C228
webssi webssi_dinolb_ 224ae (Max: 0.2796) 1:40
8280
6278
6275
0214
0313
8360
0360
B340
clip, hipeh in12k, video (Max: 0.2846) 1:32
8285
6204
6278
0278
0276
527L
037L
videomaev2, base (Max: 0.2952) 1:12
2285
5228
5203
0256
EATA
CASE
BADE
timesformer, base .4000 (Max: 0.2941] 1:12
6244
0200
E303
02M
02M
dinova large video (Max: 0.2977)
01.20
02.271
02.47
21
dinova large image (Max:
03.00
02.381
02.47
21
21
C3
2230
videomaev2_ Jarge
1:24
02.00
$7.00
50.301
U2
22.0
webssi_ webssi_dino7b di__ddnnon_222 video 0.3071) 1:32
(4.41
(9.411
11.41
11
$1.71
webssi_ dicol_ddno___2___2_ 224002222____22222 jmage (Max: 0.3067) 1:32
05.30
00.311
05A71
29
mebssi_dino705 22472622222____ 224_mmaee (Max: 0.3069) 1:32
$1,011
8387
13.10
0307
11.11
0365
0254
0248
CO4E
webssi, dinecl__dncbb___co 224, video (Max: 0.3187) 1:32
8326
0313
8329
0314
0356
6265
0200
0258
C255
dinov3_77 7b jmage (Max: 0.3268) 1:40
8227
6:21
8:27
012
0315
030T
020
dinov3_7 7b, video (Max: 0.3279) 1:40
8328
6325
6324
5324
SILE
E366
8260
EXEE
dinov2 large video (Max: 0.3313) 1:24
A321
5320
5329
5327
0322
BILA
E20L
$250
8268
dinova_hipp www.huueeeeeeeeee (Max: 0.3368)
1:32
6441
(1.08
(1.51
(1.21
$1.30
0273
dinov3 Linov3_huge___e immgee (Max: 0.3371)
1:32
01.00
(1.20
01.31
21.211
01.21
(4.30
$1.001
$1.00
01.20
01.01
01.11
dinov2_giant_ video 0.3536)
06.00
06.211
06331
09.211
08.00
98.711
(9.20
08.21
(8
06.11
videomaev2_Mee Suugeehhhheeeee 1:32
08.30
06.311
09.00
09.21
09.00
(6.1)
01.00
08.21
00.11
pe_core_gias ter_ggnnnttveeeeeoo 0.4007) 1:50
50.30
(9.30)
gemmaz-2711 27b-t ddd- (Avg: 3.711
gemmaz 30m-maa--b-....m cd- (Avg: 0.15) 1:43
fiama faama-55b, db ci-9 (Avg 0.312)
gemmaz 2-b-t cid. (Avg: 3.307
(Avg: 0.313 1:81
llama 7b cid- (Aug 0,3041 1:33
bloom- 560m (Avg (Avg: 0
(Avg: 0.302) 1:19
1:27
(Avg: 0.2971
(Avg: 0,2801 0.278)
1:31
1:25
0.252)
1:25
1:25
farge cid-3 (Avg:
liama3
gemma 20 761, 1b1,
bloom-
bloom-1b
We measure alignment between subset of our highest scoring vision and text encoders in Figure 11 on the VATEX dataset, using all ten captions for alignment. We observe a significant boost when integrating information across multiple captions. Notably, our best results significantly exceed those reported in Huh et al. (2024), highlighting that the paucity of annotations in both visual space (images videos) and text space (single caption vs. multiple descriptions) can help to explain the limited alignment observed in prior work.
Figure 12:
27


Reviews:
[{"review_id": "bCVJnTmUZK", "reviewer": "Reviewer_aU2m", "summary": "This work conducts a study of video-text representation alignment, probing the capabilities of modern video and language encoders.The authors propose a systematic probing framework using mutual k-NN alignment, extending previous static image–text  to videos. The study is comprehensive especially in video-text alignment study.", "questions": "1, What's the main difference or challenge in adapting mutual k-NN from  Huh et al. (2024) to the video domain? It seems the novelty is incremental. \n2, L484-485, what could the potential reason? This question may help with future directions", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, The study of the video-text alignment is meaningful and important.\n2, Provides comprehensive experiments, which requires a lot of hardwork.\n3, the idea of test-time scaling seems sound\n4, Obervation provided in L182-186,L484-485 is informative.", "weaknesses": "1, Limited novelty in methodology.\nThe approach seems  a empirical report, might not  meet ICLR’s innovation threshold.\n2, abstract is different from the paper, could be misleading.\n3, the improvement over previous image based methods seems limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1HxCv1IiWR", "reviewer": "Reviewer_AP4E", "summary": "The paper investigates video-text representation alignment, focusing on modern video and language encoders. It demonstrates how cross-modal alignment depends on the richness of visual and textual data provided at test time. The study introduces test-time scaling laws, showing how adjustments to the number of frames and captions can improve alignment scores. The paper further explores the relationship between alignment quality and downstream task performance, including temporal reasoning and general video understanding. The findings suggest that alignment could serve as a valuable zero-shot metric for evaluating video models.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "**Comprehensive Approach**: The paper provides the first comprehensive study of video-text representation alignment, extending the Platonic Representation Hypothesis to the temporal domain, making it a significant contribution.\n\n**Correlation with Downstream Tasks**: The correlation between alignment scores and performance on semantic and non-semantic tasks demonstrates the practical value of alignment as a metric.", "weaknesses": "The idea of probing visual representation with video-text alignment is not such convincing. This evaluation is fair for models proposed on cross-modal tasks, but visual ability is not only cross-modal alignment.   \nFor example, in tasks such as video object detection and video object tracking, the vision model only need to detect pixel-level difference in the picture, without the need to be aware of textual semantics.   \nThe DINO-series [1], SAM-seris [2], I-JEPA [3] and V-JEPA [4] are some evidence that model can excel in visual tasks without the need of textual semantic. \n\n[1] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P. and Joulin, A., 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9650-9660).    \n[2] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao et al. \"Segment anything.\" In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015-4026. 2023.  \n[3] Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y. and Ballas, N., 2023. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15619-15629).   \n[4] Assran, Mido, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi et al. \"V-jepa 2: Self-supervised video models enable understanding, prediction and planning.\" arXiv preprint arXiv:2506.09985 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QNcdMHDcwj", "reviewer": "Reviewer_h9ZS", "summary": "This paper presents a large-scale empirical study on the emergent alignment between unimodal video and text representations, extending the Platonic Representation Hypothesis to the temporal domain. The authors' central thesis is that the alignment is not a static property but is highly dependent on the amount of information provided at test-time. This observation is formalized through a proposed \"test-time scaling law,\" a parametric model that achieves a high degree of fit with the empirical data. Finally, the authors show that this alignment score strongly correlates with downstream performance on a variety of video understanding tasks, proposing it as a computationally efficient, zero-shot proxy for evaluating video model capabilities.", "questions": "1.\tWhat is the true significance of scaling data to boost alignment scores?\n\n2.\tIsn't the high alignment achieved this way just an artificial way to minimize error? Fundamentally, isn't this just about providing more information to reduce the randomness of the MkNN metric and make the metric itself more robust? \n\n3.\tIf a model is incapable of producing a robust, comprehensive, and unambiguous representation from a single piece of data, then it naturally deserves a lower alignment score. Isn't that precisely what's supposed to happen? Why do we need to artificially intervene to change this score at all?\n\n4.\tWhether alignment is a true predictive cause or simply another correlated effect of a \"better model.\"? \n\n5.\tLacking theoretical analysis on Scaling Law: The is presented as an empirical fit. While the fit is excellent, Is there any theoretical justification/guarantee for its specific mathematical form of the proposed scaling law. \n\nOther questions please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tLarge-Scale Empirical Study: The primary strength of this work lies in its experimental rigor. The authors conduct a comprehensive analysis across a vast suite of 63 vision models and 30 language models on multiple datasets.\n\n2.\tCritical Baseline: The paper's systematic use of a powerful image-encoder-plus-frame-averaging baseline is a significant contribution. The fact that this simple baseline outperforms many purpose-built video models is a critical finding for the community.\n\n3.\tNovel Phenomenon: The introduction of the test-time scaling law (Eq. 2) is a valuable contribution. The high coefficient of determination (R² > 0.98) indicates its descriptive power, and the interpretability of its parameters provides a principled way to compare how different architectures leverage temporal information.", "weaknesses": "1.\tThe Scaling Law: I am concerning about the scaling law proposed in the paper. What is the true significance of scaling data to boost alignment scores? Isn't the high alignment achieved this way just an artificial way to minimize error? Fundamentally, isn't this just about providing more information to reduce the randomness of the MkNN metric and make the metric itself more robust? But if a model is incapable of producing a robust, comprehensive, and unambiguous representation from a single piece of data, then it naturally deserves a lower alignment score. Isn't that precisely what's supposed to happen? Why do we need to artificially intervene to change this score at all?\n\n2.\tInsufficient Control for Confounding Variables: The paper's core claim is that video-text alignment is a predictive proxy for downstream performance. However, the analysis does not control for obvious confounding variables like model scale, pre-training data volume, and architecture. It is unclear whether alignment is a true predictive cause or simply another correlated effect of a \"better model.\" The study would be significantly stronger if it could demonstrate this correlation holds even when controlling for these factors (e.g., within a single model family of varying sizes).\n\n3.\tThe statement of \"General-Purpose\" Applicability: The claim that the alignment metric serves as a probe for \"general-purpose video representation and understanding\" is not fully substantiated. The reported weak correlation with the point tracking task is a direct counterexample, suggesting the metric's predictive power may be limited to a specific class of tasks. \n\n4.\tLacking theoretical analysis on Scaling Law: The proposed scaling law is presented as an empirical fit. While the fit is excellent, the paper offers no theoretical justification for its specific mathematical form. Without a basis in information theory, learning theory, or another principled framework, the law remains an observation specific to this experimental setup, and its generalizability to other datasets, modalities, or alignment metrics is not guaranteed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XBmFdwlGM6", "reviewer": "Reviewer_ZGK1", "summary": "This paper extends the static cross-modal alignment evaluation scheme to the dynamic temporal domain via the Mutual k-NN metric. It explores the laws of video-text alignment using the VATEX and PVD datasets; validates the correlation between alignment and downstream tasks using Kinetics-400 and SSv2; and verifies the sensitivity of alignment to temporal information using VideoComp. Its core contributions are: 1. Proposes an alignment evaluation scheme tailored for the temporal domain; 2. Confirms that multiple video frames and multiple captions during testing can enhance alignment performance; 3. Proposes the Test-time Scaling Laws formulation, which can guide data collection and model capability comparison; 4. Preliminarily verifies that video-text alignment can serve as a zero-shot metric, enabling replacement of expensive cross-modal decoder-based evaluation without additional training; 5. Verifies the temporal sensitivity of alignment using the Test of Time and VideoComp datasets.", "questions": "Reference Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.For the first time, static alignment evaluation strategies are extended to the dynamic temporal domain, enabling alignment assessment between video and text.\n2.It clarifies the law that \"multiple frames/multiple captions can improve alignment but exhibit a saturation effect,\" providing a directly reusable quantitative basis for \"data collection scale optimization\" and \"rapid model selection\" in practical applications, thus avoiding redundant resource investment.\n3.Covering multiple downstream tasks, the zero-shot evaluation metric features a low-cost advantage: it enables the assessment of model representation quality without the need to train additional expensive cross-modal decoders, significantly simplifying the evaluation process and reducing computational and annotation costs, which adapts to the demand for efficient model screening in production.\n4.It empirically verifies temporal sensitivity, pointing out directions for the optimization of video models.", "weaknesses": "1.The core theory relies solely on experimental observations and lacks solid support from theoretical derivation.\n2.The Test-time Scaling Laws are essentially derived from data fitting, lack theoretical support, and their generalizability across scenarios remains limited.\n3.The downstream tasks covered in the preliminary exploration of the zero-shot metric are incomplete. In practical scenarios, more focus is placed on the actual performance of downstream tasks, and relying solely on relative scores may not meet the practical needs of video model selection.\n4.The temporal analysis lacks validation on large datasets, casting doubt on the applicability of its conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)