VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models
Authors: Jesimon Barreto
Carlos Caetano
Andre Araujo
William Robson Schwartz
Venue: NeurIPS 2025 poster ()
Keywords: Vision Foundation Models
Self-Supervised Fine-Tuning
Video-Based Adaptation
Parameter-Efficient Tuning
Primary area: deep_learning
TL;DR: We show that adapting vision foundation models using self-supervised fine-tuning with simple object-centric videos substantially improves representation quality without labels.

Abstract: Foundation models have advanced computer vision by enabling strong performance across diverse tasks through large-scale pretraining and supervised fine-tuning. However, they may underperform in domains with distribution shifts and scarce labels, where supervised fine-tuning may be infeasible. While continued self-supervised learning for model adaptation is common for generative language models, this strategy has not proven effective for vision-centric encoder models. To address this challenge, we introduce a novel formulation of self-supervised fine-tuning for vision foundation models, where the model is adapted to a new domain without requiring annotations, leveraging only short multi-view object-centric videos. Our method is referred to as VESSA: **V**ideo-based obj**E**ct-centric **S**elf-**S**upervised **A**daptation for visual foundation models. VESSA's training technique is based on a self-distillation paradigm, where it is critical to carefully tune prediction heads and deploy parameter-efficient adaptation techniques – otherwise, the model may quickly forget its pretrained knowledge and reach a degraded state. VESSA benefits significantly from multi-view object observations sourced from different frames in an object-centric video, efficiently learning robustness to varied capture conditions, without the need of annotations. Through comprehensive experiments with 3 vision foundation models on 2 datasets, VESSA demonstrates consistent improvements in downstream classification tasks, compared to the base models and previous adaptation methods. Code is publicly available at https://github.com/jesimonbarreto/VESSA.

VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models
Jesimon Barreto
Carlos Caetano
André Araujo
William Robson Schwartz'
'Departamento de Ciência da Computação, Universidade Federal de Minas Gerais (UFMG) Recodaai, Instituto de Computação, Universidade Estadual de Campinas (UNICAMP)
Google DeepMind
(jesimonbarreto william ufng. br, caetanocoumicamp.be br, andrearaujo@goggee.com
Abstract
Foundation models have advanced computer vision by enabling strong perfor- mance across diverse tasks through large-scale pretraining and supervised fine- tuning. However, they may underperform in domains with distribution shifts and scarce labels, where supervised fine-tuning may be infeasible. While continued self-supervised learning for model adaptation common for generative language models, this strategy has not proven effective for vision-centric encoder models. To address this challenge, we introduce novel formulation of supervisdd fine-tuning for vision foundation models, where the model is adapted to new domain without requiring annotations, leveraging only short multi-view object- centric videos. Our method is referred to VESSA: Video-based objEcctcentric Self-Supervised Adaptation for visual foundation models. VESSA's training tech- nique is based on a self-distillation paradigm, where it is critical to carefully tune prediction heads and deploy parameter-efficien adaptation techniques otherwise, the model may quickly forget its pretrained knowledge and reach degraded state. VESSA A benefits significantly from multi-view object observations sourced from different frames in an object-centric video, efficiently learning robustness to var- ied capture conditions, without the need of annotations. Through comprehensive experiments with 3 vision foundation models on 2 datasets, VESSA demon- strates consistent improvements in downstream classification tasks, compared to the base models and previous adaptation methods. Code is publicly available https: gi thub. com/j jes http/jjssmoonann iwwimmmrretooo//eooeeoeoeooeooet /VESSA.
Introduction
Visual foundation models (VFMs) trained with self-supervised learning large image datasets have become a powerful tool for wide range of computer vision tasks [1, 2]. Techniques such as contrastive learning and self-distillation allow these models to learn high-quality visual representations without manual labels [3, 4]. Despite their generality, performance can suffer when applied to specialized domains with different characteristics from the pre-tras mre-training data. For this reason, after the VFM is pre-trained, fine-tuning is commonly employed before applying it to downstream tasks. Supervised fine-tuning, in particular, has been the dominant approach, with impressive results across a variety of datasets and applications [2, 5] such as remote sensing [6, 7]. medical imaging [8, 9] and place recognition [10, 11]. These successes demonstrate the adaptability of pre-trained models, but also highlight their reliance on labeled data, which can be expensive or impractical to obtain in many real-world scenarios.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Classification with Pretrained Foundation Model
Incorrect Classification
Input image
Pretrained Foundalion Model
Feeture Embedding
Noighbors (KNN)
Classification with improved Model
VESSA
Haffetn
Corract
Video-besed Emfiien
Seif-S Sup.r-iapddidddddee Adaplation
dataset
Improwad
Feature Embeddiig
K-Nearest Noighbors
Foundation Model
(KNN)
Figure 1: We present VESSA, novel and efficient method for adapting vision foundation models using self-supervised fine-tuning with videos. Starting from pretrained foundation model applied to classification problem in target domain, VESSA adapts the model without using labels by leveraging simple, object-centric videos. The resulting model learns improved representations that better structure the feature space in the target domain, boosting downstream classification accuracy.
Significant challenges may arise in scenarios where labeled data are unavailable for supervised fine-tuning. Despite its potential in cases where collecting annotations is costly, time-consuming, or even infeasible, unsupervised fine-tuning for foundation models remains largely underexplored in vision [12, 13]. By contrast, the natural language processing (NLP) community has long adopted unsupervised fine-tuning as a standard method to specialize large language models to new data distributions, typically via continued pretraining on unlabeled in-domain text [14, 15, 16]. While this strategy has proven successful for generative language models, adaptation visual data remains an open and challenging problem. For this reason, a few natural questions arise: how can adapt pre-trained vision model to a specific context without supervision? What forms of unlabeled visual data are best suited for adapting vision foundation models to new data distributions? What type of learning technique can effectively adapt pre-trained visual representations under the constraints in this scenario?
To answer the aforementioned questions, we propose VESSA (Video-based objEct--cntric Self- Supervised Adaptation), self- celf-supervised fine funnng method for VFMs that is both simple and effective, leveraging only short object-centric videos. A conceptual overview of the application of model is presented in Fig. 1. VESSA employs self-distillation training algorithm with critical adaptations to make it work in fine-tuning setup. We show that naive application self-distillation to the fine-tuning stage may lead to degraded model state, but this can be avoided with the careful adjustments proposed in this work. In particular, we introduce a training schedule which adjusts the self-distillatio prediction head before unfreezing the rest of the model. Then, efficient method is used to gently tune the backbone parameters towards the new domain without disrupting the encoded pre-trained knowledge, also leveraging uncertainty weighting prioritize harder training examples. Finally, we propose to source observations of target objects in the new domain from short videos, which are easy capture and require no labeling, but enhance the model performance significantly.
Experimentally, we leverage three existing foundation models and two downstream classification applications to comprehensively assess the proposed VESSA A technique. Our results demonstrate that the proposed video-base based self-supe fine-tunt fine-tuning significantly outperforms base foundation models or other fine-tuning strategies.

---

2 Related Work
Recent advances in visual foundation models have reshaped the landscape of computer vision by enabling scalable, general-purpose representations trained on massive datasets. To tailor these representations to specific downstream tasks, task-adaptive fine-tuning strategies have emerged as solution for many applications, aiming to bridge the gap between foundation model generality and task-specific performance. Complementary to this, approaches in video-to-image knowledge transfer explore how temporal and multimodal supervision in video models can be distilled into stronger static image representations. In this section, we provide structural overview of these areas, highlighting their connections to our proposed formulation and identifying key gaps our method addresses.
Self-supervised Visual Foundation Models. Self-supervised transformer- bassed foundation models have achieved state-of-the- results across a wide range of computer vision tasks [17, 18, 19]. For
image classification, image-level self-supprvised models such as DINO [3], DINOv2 [4], iBOT [18], and SimCLR [19] have shown strong performance, with DINO and DINOv2 notably relying on label-free self-distillation for scalable pretraining. The widely adopted Masked Autoencoders (MAB)
[17] build on the idea of reconstructive pretraining by randomly masking a large portion of input patches and training the model to reconstruct them, enabling efficient learning of high-capacity visual representations that scale well with data and model size. A key advantage shared by all these methods is their independence from human annotations, which allows them to leverage large-scale unlabeled datasets without the constraints and costs associated with manual labeling. This characteristic makes them especially powerful in scenarios where labeled data is scarce unavailable, enabling the use of virtually all accessible visual data for representation learning.
Task-Adaptive Fine-Tuming, aask-adaptive or continual pretraining has yielded notable gains in natural language processing, enabling foundation models to specialize for downstream domains [14, 15, 16]. In computer vision, the use of self-supervised learning to adapt models to different domains has been widely studied [20, 21, 22, 23], where the goal is to bridge the gap between a labeled source domain and an unlabeled target domain. In contrast, we aim to adapt pretrained VFM to a specific domain using only unlabeled data. Many approaches for adapting vision foundation models have been proposed, such as AdaptFormer [24] and Visual Prompt Tuning [25], but they rely on supervised learning and often involve more complex adaptation pipelines. Recent works such as [26, 27] employ continual pretraining to adapt image- based foundation models to satellite imagery. These approaches aim to construct new domain-specific foundation models, which are subsequently fine-tuned with supervision for downstream tasks. ExPLoRA [28]. the most similar to our approach in terms of weight adaptation, builds a new foundation model for satellite images by adapting self-supervised models trained on other domains-such as DINOv2 [4] Masked Autoencoders and]-and incorporates LoRA [29] for efficient continual self-supervised learning. It reuses the training procedures and heads of the base models, but ultimately still relies on supervised fine-tuning to complete the adaptation. In contrast, our method does not propose new foundation model. We reuse the original DINO architecture [3] and perform direct adaptation to downstream tasks without requiring labeled data. We adapt it not only with a new loss function, but also with a new training method, with careful tuning of different parts and the integration of an efficient parameter learning component. This enables effective representation learning even in settings with limited domain-specific data.
Video to Image Knowledge Transfer: Videos provide rich supervisory signals due to their inherent spatio-temporal consistency, natural motion, and realistic object transformations [30, 31, 32, 33, 34, 35]. Recently, several works have investigated transferring knowledge from video to image representations [36]. Methods such as VITO [37] and ViC-MAE [38] build joint models for video and image tasks, involving costly frame selection pipelines and brbdid loss formulations (c.g.. masking and contrastive objectives). These models often require substantial architectural modifications and exhibit training instability, with heavy reliance on masking to guide object learning. Time Does Tell [39] introduces a dense frame-wise module to extract visual features from all frames. This highlights the potential of video data and its valuable contribution to unsupervised representation learning. In contrast, we propose lightweight approach using object-centric videos with minimal visual clutter. Our method emphasizes learning unified and robust representations from simple videos, enabling improved generalization without extensive frame curation or resource consumption. Importantly, most prior video-based methods target level-level tasks (c.g., segmentation, detection) and show
3

---

Frame/View Selection
Preprocessing Augmentation
Model
Fine-tuning
R
Teacher model
Grap
View
UWSD Loss
Vieue Pains
model
rown:
Lofoturingg
Uniruson
View
Fli
Pairs
viow
poin
LoRA
Grayscalo
Video
detaset
Norm
Attontion
Pairs
Color
Embeddings
VIT biock
Figure 2: The proposed training pipeline. The model input consists of videos, which first undergo Frume Selection stage, where n pairs of frames are sampled from each video. These pairs are then passed through the Preprocessing and Augmentation stage, where distinct transformations are applied to the first and second images of each pair. The resulting views are fed into teacher and student networks, both initialized from the same foundation model; LoRA is applied their architectures for parameter-efficient fine-tuning. Finally, the uncertainty-weighree self- deff-distillation loss (UWSD) is applied to align their representations.
limited improvements for frame-level classification. These methods also typically require fine-tuning to be competitive.
3 VESSA
In this section, we describe the methodological foundations of our work and introduce our novel formulation of self-supervised fine-tuning for vision foundation models. We begin by revisiting prior methods that serve as the basis for our formulation, highlighting their key mechanisms. Building on this foundation, we then present our novel self superviie fine-tuning strategy for vision foundation models, designed to enhance adaptation.
3.1 Background
Our method builds on recent advances in self-supervised learning and parameter-efficient adaptation. We focus on two core components: DINO [3], a self-distillation framework for label-free represen- tation learning, and LoRA [29], lightweight technique for adapting large models with minimal trainable parameters. We review them briefly in the following.
DINO [3] is self-supervised learning framework that trains student network to match the output of a teacher network, with both networks receiving different augmented views the same image. The teacher's parameters updated using exponential moving average (EMA) of the student's weights, ensuring stable training and consistent representations. The method aligns the output probability distributions of the student and teacher using cross-entropy loss:
LINOO Le(e)) log fa(
where at, and Za,i denote the augmented views for the i-th image, and fe(-) and f.(-) represent the outputs of the teacher and student networks, respectively. This objective minimizes the discrepancy between their normalized output distributions, encouraging the student network to learn meaningful representations without the need for labeled data. DINO has demonstrated strong performance across variety of visual tasks, producing robust and transferable features.

---

Low-Rank Adaptation (LoRA) [29] is -eficceeeeeficient fine-tuning technique that enables the adaptation of pre-trained models by injecting trainable low-rank matrices into their weight structure. Rather than updating the full weight matrix W E Rdxk of linear layer during training. LoRA keeps W frozen and instead learns an additive update in the form low-rank decomposition:
AW = AB, AERUX, BER
(2)
where r < min k). Here, and are the newly introduced trainable matrices, and AW the low-rank adaptation added to the original weight matrix. This approach significantly reduces the number of trainable parameters and memory requirements while preserving model performance, making particularly effective for adapting large-scale models in resource-constrained environments.
3.2 Video-based objEct-centric Self-Supervised Adaptation (VESSA)
The VESSA pipeline is illustrated in Figure 2, and consists of three main modules: Frame Selection, Preprocessing and Augmentation, and Model Fine-tuning. Each input video V, with T frames (F.)== is first processed by the Frame Selection module, which samples frame pairs per video. For each pair of frames that we aim to construct, we first randomly sample starting frame index t ~ U(,, maxx. where mmax is a predefined maximum temporal offset, and U denotes the uniform distribution. Then, we sample frames based on a temporal gap å ~ (11,, mmxx), ensuring a minimum offset of one frame. Each selected pair of frames is then formed according to the rule:
8 E [1, Ōmax];
LE [ 66
This randomized strategy introduces temporal diversity by allowing variable distances between frames, which helps the model learn more robust representations across different viewpoints. At the end of this module, obtain batch composed of frame pairs sampled from different videos, represented as:
= ((Fe. Fo == 1,...,u),
where each (Fr,, +5,+ ) is a pair of frames from the k-th video in the batch, and v is the batch size. The temporal index l and offset 6% are sampled independently for each pair. This setup ensures that the batch contains diverse video content and temporal gaps.
In the Preprocessing and Augmentation module, each frame in the pair k is transformed indepen- dently using two distinct pipelines of random augmentations. This results in two augmented views:
p(b) r!()) and 1+5. which are designed to promote appearance diversity and representation robustness. This module also generates the local crops used our adaptation. Inspired by the reference method, which employs multiple local crops per image stabilize fine-tuning, we adapt this strategy by sampling local crops as pairs-one one from each frame. This pairing ensures temporal consistency while preserving local variability, contributing to more robust and transferable representations. At the end of this module, we have
= { {(((e)
F(() 14 Fattor ) kk= |k=1,,... .).
{ (p(e),
where:
+ two temporally separated frames from the k-tth video, with distinct global transformations and b applied;
F(o) and Fx()
ple)) p(a) )" te ==1
set of u pairs of local crops, where e; denotes distinct transforma-
tion involving small crop on the main frame: and u the number of local crop pairs.
In the Model Fine-fur Fine-tuning stage, training is performed in batches; however, for clarity, we describe the process using single pair of images and associated local crops. The pair k; given by (F("). p(b) is passed through both the student and teacher networks, while the local crops are processed only by the student. Both networks are initialized from the same pretrained vision foundation model. The outputs of the student and teacher are denoted as follows:
5

---

s = s=f((((
= f(
Stcl = f(((()) Ste2 = fa((()
=122=
s. q denote the outputs of the projection head applied to the global frame representations. The final representations of all local crops (i.e., Stel and Stc2). along with are also compared to as part of the DINO loss computation. The fine-tuning training objective the formulation presented in Section 3.1. To prioritize uncertain teacher outputs, we introduce an Uncertainty- Weighted Self-Distillation (UWSD) loss, which modulates the contribution of each sample to the loss based on the estimated uncertainty of the teacher's 's predictions. The entropy of the teacher's distribution is computed. The entropy is used to compute the weight:
weighted form of
w(q)= = 1 + H(g);
where y is hyperparameter controlling the influence of uncertainty. The final training objective becomes:
LUWSD
Σ
w((g) LDINO 8, Ste;)
N
(9.8.8te, JBB
Critical optimization considerations. Continuing self-supervised training from a pretrained founda- tion model requires careful handling due potential gradient instabilities. These are often caused by shifts in data distribution and the use of a randomly initialized projection head, which introduces abrupt gradient changes early in training. In standard training regimes, all network parameters typically updated simultaneously, which can lead to unbalanced gradient flow and degrade the pre- trained representations. To mitigate this, we initially freeze the backbone and train only the projection head for a few epochs, allowing it to adapt to the existing embedding space. As another strategy to mitigate this issue, we gradually unfreeze the backbone, applying different strategies to different parts of the network. Specifically, we enable fine-tuning of the first H layers using LoRA [29], which restricts updates to low-rank adaptations of the attention Value projections of each self attentttnn wailr keeping the normalization layers trainable. This design helps preserve the low-level visual features encoded in early layers, such as edges and textures, which are generally transferable across domains. In contrast, the last T. layers of the backbone are fully unfrozen and updated normally, allowing the model to adapt high-level semantic representations to the new domain. This staged unfreezing strategy mitigates representational drift while preserving stability and efficiency during fine-tuning.
sigghts--peeifialllly in the Query. Key, and
Experiments
4.1 Experimental Setup
Datasets. MVImageNet[40] and CO3D [41] are large-scale video datasets offering multi-view images. MVImageNet comprises 6.5 million frames from over 219,000 videos across 238 object categories. while CO3D includes 1.5 million frames from 19,000 videas spanning 50 categories. Captured from various viewpoints under real-world conditions, these datasets enable learning of viewpoint-consistent representations. To adapt them for classification, we designed protocol that splits each class into training and testing sets (75%-25% selecting one frame per video and using k- <-Nearest Neighbors (KNN) to evaluate the quality of learned embeddings across different views and instances.
Implementation details. Our experiments were performed on TPU v3-8, featuring 8 cores and 128 GB of high-bbndwidth memory. All implementations used the scenic library [42] in JAX. We adopted the ViT-Base architecture to balance performance and efficiency, as preliminary tests showed that the Small model underfit and the Large model offered minimal gains at higher cost. Our training followed the base hyperparameter configuration of the DINO protocol [3], except for the specific settings detailed below. As reference, we adopted 10 training epochs for both the initial projection head adaptation and the subsequent full model training, using batch size of 256 and an input image resolution of 224x224. For each video, we sampled frame pairs. The hyperparameter Y, which controls the weight of the distillation loss, was set to 1. For the image-based baseline, we used the

---

first frame of each pair as the reference image. This frame was then processed through the subsequent steps as a single-image imput, following the standard image-based pipeline.
Statistical significance test. To assess the significance of the observed differences, we performed statistical tests using three independent runs for each experimental configuration. We employed an unpaired Student's 's 1-test with a 90% confidence level. In supplementary material we report confi- dence intervals for the main comparisons to highlight cases where the differences were statistically significant.
Visual Foundation Models Our experiments were conducted using widely adopted and kate-of-thee art VFMs, including DINO [3]. DIÑOv2 [4], and TIPS [43]. These models represent a strong set of visual backbones commonly used in computer vision tasks. In order to standardize the experiments, an input size of 224 was used for all VFMs. However, please note that TIPS can achieve improved results with input size of 448, which would match its pretraining setup.
We compare our method with ExPLoRA [28], a recent approach for improving transfer learning of pretrained vision transformers (ViTs) under domain shifts. To ensure a fair comparison with our video-based approach, we extend ExPLoRA to operate on short object-centric videos. Experiments using TIPS ExPLoRA were not reported since the original work ExPLoRA clearly specifies that it is designed for continual self- self-sueereieed learning using training heads from self-supervised models (e.g., the projection head from DINO for self-distillation autoencoder- baseed modeling such as MAE [44]). TIPS, however, is not a sele--suerriiedd model.
4.2 Results
We begin our evaluation by analyzing the individual contributions of each component of the VESSA approach through a comprehensive ablation study. This analysis provides insights into the effective- ness of leveraging video data for self-supervised adaptation and highlights the critical design choices that enable our approach to outperform existing alternatives. In what follows, we systematically isolate and compare different configuran configurations, followed by comparisons state-of-the-arr baselines across multiple datasets and model architectures.
A series of experiments conducted on the MVImageNet dataset using the vision transformer-small (ViT-S) to isolate and evaluate the contributions of each component of our method are shown in Table I. Given the challenge of adapting
new domain with limited and unlabeled data, we first trained DINO from scratch using both image and video data, As expected, the results were suboptimal due to the limited data, with final accuracy of 33.86%. However, training with video (i.e., using different views of the object from the video) consistently outperformed the image-based counterpart, achieving 39.39% accuracy-an improvement of 5.53 percentage points (p.p.) aligning with our motivation to leverage temporal fformation-thuugee the results remain relatively low overall. Subsequently, we applied the pretrained DINO model directly, which yielded solid performance and served as a strong baseline, achieving 89.69% accuracy. We also evaluated a naive continuation of training using only images, which similarly led to performance degradation, resulting in a slightly lower accuracy of 88.54%. Notably, switching from image input with local crops (88.54%) to video input without local crops (90.53%) already provided a larger gain (1.99 p.p.), indicating that temporal information plays a stronger role than local crops alone. Careful decision of the training head projection before fine-tuning had significant impact, improving performance by approximately 10 p.p.. For instance, using video input with local crops but without training the head achieved 80.87% 87% whereas enabling head training in the same setup increased accuracy to 91.87%, underscoring that head training the dominant factor. This result indicates the importance of carefully designed adaptations to achieve such improvements. Finally, we compared our full method against its individual variants, confirming that our complete approach achieves the best results, validating the importance of each design choice in the overall effectiveness of VESSA. The pest-performing configuration achieved 91.87% accuracy, representing an improvement of 2.18 p.p. over the pretrained DINO baseline, with the optimal setting obtained by unfreezing the last 2 layers during adaptation.
Across all experiments, leveraging video consistently outperforms frame- based alternatives. To better understand the source of these gains-wheeen conducted an additional experiment on the CO3D dataset using DINO and DINOv2. Specifically, we evaluated the impact of frame distance (6) during self-supervised fine-tuning, as detailed Table 2. The results show that varying the temporal gap between frames affects performance, with the highest
from motion cues or temporal continuity-we
7

---

Table 1: Ablation study components for video-based self-supervised ViT fine-tuning. All models use VIT-S/16 and are evaluated with k-NN (k=1) on the MV ImageNet dataset. We analyze the impact of architectural choices, local crops, training heads, and data modalities.
UWSD
Unfrozen
Local
Train Head
Method
Input
Accuracy (%)
Loss
Last Layers
Crops
2
Video
91.87
2
Video
90.53
2
Video
90.92
Video
87.14
VESSA
3
V
Video
90.80
4
V
Video
90.55
2
V
Video
80.87
2
Image
88.54
DINO [3] DINO [3] DINO [3] Pretrained
V
Image
33.86
Video
39.39
Image
89.69
accuracy achieved when was randomly sampled from the range [5, 10), yielding 85.03% with DINO and 91.85% with DINOv2. This suggests that exposing the model to diverse temporal relationships between frames contributes positively to representation learning.
Table 2: Top-1 accuracy (%) on the CO3D dataset using different frame distance strategies. We report k-Nearest Neighbors (k=1) classification accuracy for models pretrained with DINO and DINOv2. Each value defines the temporal distance between frames selected from videos during self-supervised fine-tuning.
Frame Distance (8)
DINO [3]
DINOv2 [4]
1
85.00
91.51
2
84.73
91.52
3
84.90
91.39
4
84.27
91.42
5
84.66
91.80
85.00
91.74
84.54
91.25
20
84.82
91.23
Random [5, 10]
85.03
91.85
Random [10, 30]
82.46
91.52
After analyzing the individual components of our approach, we now turn broader evaluation of VESSA applied to different backbone models across two datasets, As shown in Table 3, all other methods significantly outperform the base pretrained models without fine-tuning (the first row of the tables), except for a few cases involving baseline variant of our method that employs static images only, which we refer to as Static-baseline. In particular, when video data used, unsupervised fine-tuning generally leads to superior performance across both datasets and architectures, with the exception of Explora with video and DINO in MVImgNet, where performance decreases relative to the pretrained model. The performance gap between VESSA and Static-baseline, as well as between the ExPLoRA bascline and ExPLoRA + video, confirms the effectiveness of adapting models to the target domain using unlabeled video data in both CO3D and MVImageNet datasets. In contrast, the results of the Static-baseline indicate that a naive image-based self-supervised continual learning approach is not sufficient to achieve successful fine-tuning.
Considering the CO3D dataset, applying VESSA to DINOv2 yields the best result 91.85% + 0.56, which is 2.21 p.p. higher than ExPLoRA + video (89.64 + 0.47); this difference is statistically significant. On the other hand, for MVImageNet dataset, VESSA achieved 96.01 1.08 while EXPLORA video reached 96. 15 + 0.87; however, the difference is not statistically significant (see supplementary material for details). Nevertheless, when considering DINO, VESSA outperforms again the ExPLoRA results. Moreover, approach also performs better with TIPS against its

---

Ball
Baseball glove
Book
Apple
Carrot
Validation Image
Orange
Plant
Toy Truck
Orange
keyboard
IHBA ve
Dinov2
glove
VESSA (ours)
Figure 3: Qualitative examples of nearest neighbor retrieval = on the CO3D validation set. We selected some of the most challenging validation samples and retrieved the nearest neighbors using two methods, shown row-wise: the first row displays the query (validation) images; the second row presents the retrievals using raw DINOv2 features; the third row shows the retrievals produced by our method. Images with red borders indicate incorrect retrievals, while green-bordered images represent correct ones. These examples illustrate the effectiveness of leveraging multi-frame video information for representation learning. Notably, our method demonstrates greater focus on the object interest, whereas the baseline often retrieves matches dominated by background similarity.
pretrained version. It is important to present these results-alongside those Table I-as they demonstrate that simply continuing self-supervised training with domain-specific image data, while seemingly straightforward, does not yield consistent improvements. As the results show, this strategy often leads performance degradation or, at best, no noticeable gains.
Table 3: Top-1 accuracy (%) on CO3D and MV ImageNet datasets using k- k-Nearest Neighbors (k=1), We compare pretrained vision foundation models, an image-based baseline, and our proposed video-based fine-tuning method. All results are reported on the validation set using representations extracted from the backbone and evaluated via KNN. Our method achieves superior performance by leveraging object-centric videos for unsupervised adaptation.
CO3D
MVImageNet DINOv2 [4]
Method
DINO [3]
DINOv2 [4]
TIPS [43]
DINO [3]
TIPS [43]
Pretrained
78.86
87.86
60.02
90.44
95.75
78.65
ExPLoRA [28]
79.78
88.31
90.94
95.79
ExPLoRA [28]+video
83.64
89.64
87.74
96.15
Static-baseline
80.31
81.60
55.59
89.39
92.53
76.05
VESSA
85.03
91.85
70.56
92.51
96.01
80.54
To qualitatively illustrate the benefits of video-based self- self-supervised training, Figure 3 shows examples of top retrievals using KNN based on the learned embeddings. When comparing the pretrained DINOv2 model with our proposed VESSA method, we observe that DINOv2 produces embeddings that focus primarily on the background and broad scene structures. In contrast, VESSA clearly attends to the object of interest, even in challenging cases where the texture or color of the retrieved object differs from the query image. This demonstrates that VESSA learns more semantically meaningful and object-centric representations, which improves robustness and task relevance.

---

5 Conclusions
In this work, we presented VESSA (Video-based objEct-centric Sulf-Super Adaptation), simple and effective strategy for unsupervised fine-tuning of visual foundation models using object-centric videos. Our approach requires no labeled data and leverages temporal coherence by treating distinct frames from the same video as positive pairs in a contrastive setup. Inspired by advances in NLP, we showed that unsupervised fine-tuning in vision is both feasible and valuable. We demonstrated that method consistently improves classification performance datasets while remaining lightweight and easy to apply. This opens up new directions for adapting foundation models to visual contexts without additional supervision, making them more versatile and accessible in practice.
Nomain-speciine
Limitations A notable limitation of our approach is the tendency to forget previously acquired knowledge during fine-tuning -a known drawback of fine-tuning methods in general. Additionally, our experimental setup relies on video data that offer multiple viewpoints of the same object, characteristic that not commonly available in many real-world datasets. This may limit the applicability of the approach to scenarios where such structured multi-view data is not present.
Acknowledgments and Disclosure of Funding
The authors would like to thank the National Council for Scientific and Technological Develop- ment - CNPq (Grant 312565/2023-2), Fundação de Amparo Pesquisa do Estado de São Paulo FAPESP (Grant 2023/12086-9) Recod.ai and Google LLC..
References
[1] Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. survey on self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[2] Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation models defining new in vision: survey and outlook. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025. 1
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021. 1, 6, 8, 9, 14
[4] Maxime Oquah, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang-Wei Wen Li. Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023. 1. 1, 7, 8, 9,
[5] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87-110, 2022. 1
[6] Xin He, Yushi Chen, Lingbo Huang, Danfeng Hong, and Qian Du. Foundation model-based multimodal remote sensing data classification. IEEE Transactions on Geoscience and Remote Sensing, 62:1-17, 2023.
[7] Xuechao Zou, Shun Zhang, Kai Li, Shiying Wang, Junliang Xing, Lei Jin, Congyan Lang, and Pin Tao. Adapting vision foundation models for robust cloud segmentation in remote sensing images. arXiv preprint arXiv:2411.13127. 2024. 1
[8] Beilei Cui, Mobarakol Islam, Long Bai, and Hongliang Ren. Surgical-dino: adapter learning of foundation models for depth estimation in endoscopic surgery. International Journal of Computer Assisted Radiology and Surgery, 1913) 1020, 2024.
[9] Mohammed Baharoon, Waseem Qureshi. Jiahong Ouyang, Yanwu Xu, Abdulrhman Aljouie, and Wei Peng. Evaluating general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks. arXiv preprint arXi::2312.02366 2023. 2023.
10

---

[10] Sergio lzquierdo and Javier Civera. Optimal transport aggregation for visual place recognition. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 17658-17668 2024.
[11] Feng Lu, Xiangyuan Lan, Lijun Zhang. Dongmei Jiang, Yaowei Wang. and Chun Yuan. Cricavpr: Cross-image correlation-aware representation learning for visual place recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16772-16782, 2024.
[12] Hao Dong, Moru Liu, Kaiyang Zhou, Eleni Chatzi, Juho Kannala, Cyrill Stachniss, and Olga Fink. Advances in multimodal adaptation and generalization: From traditional approaches to foundation models. arXiv preprint arXiv:2501. 18592, 2025. 2
[13] Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi. Shuang Xu, and Bo Xu. Vip: A survey vision-language pre-training. Machine Intelligence Research, 20(1):38 56, 2023. 2
[14] Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models domains and tasks. CoRR, abs/2004 10964, 2020. 2.3
[15] Rujun Han, Xiang Ren, and Nanyun Peng. DEER: A data efficient language model for event temporal reasoning. CoRR, 20ss2002.11283,, 2020. 2, 3
[16] Zihan Liu, Genta Indra Winata, and Pascale Fung. Continual mixed- language pre-training for extremely low-resource neural machine translation. CoRR, abs/2105.0399 2021. 2,
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[18] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022. 3
[19] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pages 1597-1607. RmL.,, 2020.
[20] Jiaolong Xu, Liang Xiao, and Antonio López. Self-supervised domain adaptation for computer vision tasks. IEEE Access, 7:156694-155706, 2019.
[21] Xiangyu Yue, 7angwei Zheng, Shanghang Zhang, Yang Gao, Trevor Darrell, Kurt Keutzer, and Alherto San- giovanni Vincentelli. Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pages 13834-13844, 2021.
[22] Min- Hung Chen, Baopu Li. Yingze Bao, Ghassan AlRegib, and Zsolt Kira. Action segmentation with joint self-supervised temporal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9454-9463, 2020.
[23] Jinwoo Choi. Jia-Bin Huang, and Gaurav Sharma. Self-supervised cross-video temporal learning for unsupervised vidco domain adaptation. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 3464-344 3470. IEEE, 2022. 3
[24] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 35:16664-1666 2022. 3
[25] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. Visual prompt tuning. In European conference on computer vision, pages 709-727. Springer, 2022. 3
[26] Linus Scheihenreif, Michael Mommert, and Damian Borth. Parameter efficient self-supervised geospatial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27841-27851, 2024. 3
[27] Matías Mendieta, Boran Han, Xingjian Shi, Yi Zhu, and Chen Chen. Towards geospatial foundation models via continual pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11880616866,,11111 2023. 3

---

[28] Samar Khanna, Medhanie Irgau. David B Lobell, and Stefano Ermon. Explora: Parameter-efficient extended pre-training adapt vision transformers under domain shifts. arXiv preprini wrXiv:2406.10973 2024. 3, 7, 9, 14
[29] Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Wei Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR), 2021. 3, 3,4,5, 6
[30] Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Leamnnng to see by moving. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 37 45, 2015. 3
[31] Xiaolong Wang and Abhinav Gupta. Unsupervised learning visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2794-2802, 2015. 3
[32] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2701-2710, 2017. 3
[33] Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4086 4093, 2015. 3
[34] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In European Conference on Computer Vision (ECCV), pages 527-544, Springer, 2016. 3
[35] Tejas D Kulkarni, Ankush Gupta, Catalin lonescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. Advances in Neural Information Processing Systems (NeurlPS), 32, 2019. 3
[36] Arthur Aubret, Markus R Ernst, Céline Teulière, and Jochen Triesch. Time to augment self supervuper visual representation learning. In The Eleventh International Conference on Learning Representations (ICLR), 2023. 3
[37] Nikhil Parthasarathy, SM Eslami, Joao Carreira, and Olivier Henaff. Self-supervised vidco pretraining yields robust and more human- aamann visual representations. Advances in Neural Information Processing Systems, 36:65743-657777 2023. 3
[38] Jefferson Hemandezz, Ruben Villegas, and Vicente Ordonez. Vic-mae: Self-supervised representation learning from images and video with contrastive masked autoencoders. In European Conference on Computer Vision, pages 444-463. Springer, 2024. 3
[39] Mohammadreza Salchi, Efstratios Gavves, Cees G. M. Snock, and Yuki M. Asano. Time does tell: Self-supervised time-tuning of dense image representations. ICCV, 2023.
[40] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjic Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. Mvimgnet: A large-scalc dataset of multi-view images. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 12345-12354. IEEE, 2023. 21
[41] Jeremy Reizenstein, David Novotny, Deva Ramanan Sun, Federico Tombari, and Andrea Vedaldi. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 10901-10911, 2021. 6, 6.21
[42] Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay. Scenic: A jax library for computer vision research and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21393 21398, 2022. 6, 21
[43] Kevis-Kokitsi Maninis, Kaifeng Chen, Soham Ghosh, Arjun Karpur, Koert Chen, Ye Xia, Bingyi Cao, Daniel Salz, Guangxing Han, Jan Dlabal, Dan Gnanapragasam, Mojtaba Seyedhosseini, Howard Zhou, and André Araujo. TIPS: Imat- Pretraining with Spatial Awareness. Proc. ICLR, 2025. 7, 9, 14
[44] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Masked autoencoders as spatiotem- poral leameer, Advances Neural Information Processing Systems, 2022. 7
12

---

A Technical Appendices and Supplementary Material
In this Technical Appendix and Supplementary Material, we present the complementary experiments of the paper. These evaluations include the main experiments with confidence intervals for the primary results, demonstrating the statistical significance of the observed differences. We further analyze the input differences between DINO and VESSA, and investigate the impact of applying stronger image augmentations, aiming to reduce the gap between static images and the visual variability observed in video data. We also present an additional study examining the extent of catastrophic forgetting when adapting visual foundation models with VESSA, highlighting its impact on general-purpose performance. Finally, we provide detailed analysis of the training cost associated with VESSA, offering quantitative insights into its computational efficiency and practical feasibility.
The main results with confidence intervals are shown in Tables and which report the top- performing models along with confidence intervals to highlight the significance of the performance differences. The results suggest that VESSA achieves significantly better performance in several cases. We employed an unpaired Student's 1-test with a 90% confidence level. We report confidence intervals for the main comparisons highlight cases where the differences were statistically significant. For the CO3D dataset, the variation in accuracy across runs was 0.52 for DINO, 0.56 for DINOv2, and 1.03 for TIPS. In all cases, the differences were statistically significant when compared the second-best performing method, which in this case was ExPLoRA A method also based on video data. In contrast, on the MVImageNet dataset, the variations were 1.11 for DINO, 1.08 for DINOv2, and 1.71 for TIPS. In this scenario, non non-overlapping confidence intervals between the BINO-based baseline and VESSA (ours) indicate a statistically significant difference. In the case of DINOv2 pretrained on MVImageNet, noticeable overlap between the two video-based methods can be observed.
(ours))
DINO/DINOV2
Figure 4: Example frames from the MVImageNet dataset illustrating the differences between global crop input pairs used for the teacher and student networks during training with DINO and VESSA (ours). Our method, VESSA, introduces substantially greater variability in the appearance of the evaluated object. The temporal distance between the selected frames is 5 frames. The first image of each pair shows the global crop from the transformation of view 1, and the second image of each pair shows the global crop corresponding the transformations of view 2.
Approximate video-like image transformations. Motivated by the strong performance observed when training with videos, we investigated whether additional image cannsformations-beyyee used in the standard DINO pipeline-could simulate the benefits of camera motion. To this end, we applied set of motion-inspired augmentations to one the views during training, aiming to mimic the effect of slight viewpoint changes. Specifically, incorporated translations of up 10% of the
those
13

---

Table 4: Top-1 accuracy (%) on the CO3D dataset using k-Nearest Neighbors (k=1). We compare pretrained vision foundation models, an image-based based baseline, and our proposed video- based fine-tuning method. ExPLoRA and VESSA results are reported on the validation set using representations extracted from the backbone and evaluated via k-NN. We report confidence intervals to highlight the statistical significance of the improvements.
Method
DINO-B [3]
DINOv2 [4]
TIPS [43]
ExPLoRA [28] + video
83.64 + 0.84
89.64 + 0.47
VESSA (ours)
85.03 0.52
91.85 0.56
70.56 1.03
Table 5: Top-1 accuracy (%) on the MVImageNet dataset using k-Nearest Neighbors (k=1). We compare pretrained vision foundation models, an image-based baseline, and our proposed video- based fine-tuning method. ExPLoRA and VESSA results are reported on the validation set using representations extracted from the backbone and evaluated via k-NN. We report confidence intervals highlight the statistical significance the improvements.
Method
DINO-B [3]
DINOv2-B [4]
TIPS-B [43]
ExPLoRA [28] video
87.74 + 1.03
96.15 0.87
VESSA (ours)
92.51 + 1.11 96.01 1.08
80.54 + 1.71
image dimensions, rotations up 10 degrees, scaling variations up to 5%, brightness shifts of 0.1, and contrast adjustments in the range of 0.9 to 1.1. These transformations were carefully selected to approximate changes in camera perspective while avoiding the introduction of unrealistic artifacts. As shown in Table 6, these modifications did not yield significant performance improvements compared to the baseline using standard image augmentations, suggesting that the advantages observed with real videos may stem from cues beyond simple geometric or photometric variation.
The data augmentation pipeline consists of two global views and multiple local crops, each subjected to specific set of transformations, as detailed below.
Global crops: Two crops sampled with scale ranges between (0.4, 1.0). The transfor- mations applied to these global crops are as follows:
Transformation view 1: horizontal flip (probability 0.5), color jitter (strength 0.8), grayscale conversion (probability 0.2), and Gaussian blur (probability 1.0).
Transformation view 2: horizontal flip (probability 0.5), color jitter (strength 0.8), grayscale conversion (probability 0.2), Gaussian blur (probability 0.1), and solarization (probability 0.2).
Local crops: A set of u local crops per image are sampled with a scale range (0.05, 0.25) defined by the configuration and resized 96 96 pixels. Each local crop undergoes the following transformations independently:
Color jitter with parameters (strength 0.8, brightness 0.4, contrast 0.4, saturation 0.2,
-
hue 0.1).
GGaaysall conversion (probability 0.2).
Gaussian blur (probability 0.5).
Impact of using video-based inputs. To highlight the differences between the view generation strategies employed by VESSA and those used in DINO, we present illustrative examples of view pairs from both approaches. It is important to note that in both VESSA and DINO, the same groups of transformations are applied independently to each view. To assess the impact of video-basee inputs on representation learning, we analyze input variability by contrasting the frame selection strategy used in VESSA with the standard augmentation-baset Figure 4, frame pairs selected by VESSA- -asedd on a fixed temporal offset of 5 5 eeheeiie substantially greater visual diversity than those generated through standard augmentations, In contrast, the views generated by DINO/DINOv2 tend to be more visually homogeneous. This enhanced
sampling DINO and DINOv2. As shown in
14

---

Table 6: Performance comparison using our method and images and transformations to simulate camera movement in images with DINO and DINOv2 on the CO3D dataset with k:
Method
DINO B
DINOv2
VESSA
85.03
91.85
Static-bascline
80.31
81.60
Static-bascline + Transf. simulate video
80.60
81.49
variability introduced by real video frames is likely key factor in the performance differences observed when training with video data, as opposed to relying solely on static image augmentations.
Impact of catastrophic forgetting and cross-dataset generalization. We also investigate the impact of domain-speeifii adaptation with VESSA on the original pretraining task. To this end, we evaluate the performance of DINO, DINOv2, and TIPS on ImageNet classification using a k-nearest neighbors (KNN) classifier, both in their pretrained form and after adaptation on CO3D and MVImageNet. As shown in Table 7, while the pretrained models achieve competitive accuracy on ImageNet, their performance drops drastically once adapted with VESSA on either domain. This result confirms the presence of catastrophic forgetting and underscores the infeasibility of applying the adapted models seneral-purposs setting. Nonetheless, this outcome aligns with the intended design VESSA: the method is tailored to specialize visual foundation models for unsupervised adaptation in target domain, where delivers strong performance despite losing generality.
Table 7: Effect of catastrophic forgetting on ImageNet classification after VESSA adaptation.
Model
DINO
DINOv2
TIPS
Pretrained
76.10
82.10
80.00
VESSA (CO3D)
15.46
17.15
18.10
VESSA (MVImgnet)
15.68
16.78
17.10
Moreover, we conducted an experiment in which training was performed using VESSA exclusively on the MVImageNet dataset, followed by evaluation on held-out test set of the CO3D dataset. As shown in Table 8, this cross-dataset setting reveals marked drop in performance- approximately 5 percentage points-wher wwen compared the baseline results obtained by pretraining and evaluating on the same dataset. This performance degradation highlights the presence of catastrophic forgetting and limited generalization capabilities when the model is exposed to a distribution shift, even when trained on a diverse and temporally rich video corpus.
Table 8: Performance comparison between DINO and DINOv2 models using the pretrained base model, proposed VESSA method, and the cross dats-ett evaluation. The cross-datased model was trained on MVImageNet and tested on CO3D to analyze forgetting behavior, demonstrating the degradation experienced when model is trained experiments utilized the ViT-B architecture.
dataset and evaluated another. All
Method
DINO
DINOY2
Pretrained
78.86
87.86
Cross dataset
74.40
80.36
Training cost and computational efficiency. We also analyze the computational requirements of adapting visual foundation models with VESSA to assess its practical feasibility. For the Co3D dataset (20, 412 training pairs and 4, 535 test samples) using a ViT-Base backbone, the adaptation stage required only 1.97 hours of training. corresponding 7.04 core-hours on TPU v3-8, which consists of 4 TPUs, each with 2 cores (total of 8 cores). The process ran for 20 epochs, processing 407, 040 examples at throughput of 135 images per second (16.88 images per second per core), with total energy consumption of 13.86 kWh and an estimated carbon footprint of 5.55ky CO2, Importantly, VESSA adapts a pre-trained model rather than training from scratch, and the minor additional overhead introduced by processing paired video frames (or local crops) is negligible due to
15

---

offline video decoding. This efficiency contrasts sharply with the full pretraining of models such as DINO DINOv2, which require thousands of GPU-hours and result in CO2 emissions on the order of tons.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction claim the introduction of our novel method for self-supervised adaptation of vision foundation models using videos, showing improvements in downstream tasks. The method is introduced in detail in Section and experiments backing up results presented in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer the Limitations paragraph in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
There are no theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All of the implementation details are described in the submission, so all the information needed to reproduce all the experimental results is provided. The code will be made public upon acceptance of the paper, to additionally aid reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
The code used to perform the experiments of this work are not included as part of the submission, but it will be made available with Open Access in the case of the acceptance of the paper. The data we use is publicly available already.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //wipssc... wwtpccppcpgguupiieeecceeecpoceee gwwd//////////gg Saoaaee//oodSSmbaaacaa
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details regarding training and testing are provided in the "Experimental setup" section of the submission.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This is discussed in Section 4. We report confidence intervals in the main tables, to reflect the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: This is described in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neeut wttpehewwwss..p.p.eeeeeeeeeeeeee Ceuublic/Ethcssuiieeee
Answer:
[Yes]
Justification:
The research conducted in this paper conforms with the NeurlPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
The paper focuses on adapting discriminative vision foundation models to specific domains without annotations. This opens doors for a number of application with positive societal impact, that organizations without significant resources can adapt such models to their needs. We do not believe that there are significantly direct paths to negative applications, especially since model is not generative.
Justification:
20

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The code that is used to support the experiments are part of the Scenie framework repository [42], which is cited in the submission. Additionally, the used datasets [40, 41] are cited in the submission.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package dataset.
The answer NA means that the paper does
The authors should state which version of the asset used and, possible, include URL.
21

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi whwoee com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The paper does not release new assets at the time of submission. In case of acceptance, the Code with the corresponding licenses will be released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method developed in this research does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... wwt//o/eeeenne///0000000 for what should or should not be described.
23


Reviews:
[{"review_id": "GoiQxNKW3c", "reviewer": "Reviewer_EBTZ", "summary": "This paper introduces a method for adapting pretrained VFMs to new domains using unlabeled object-centric videos. The approach employs self-distillation with temporally separated frame pairs as positive samples, combined with LoRA and a careful training schedule that includes staged unfreezing and uncertainty-weighted loss. Experiments on MVImageNet and CO3D datasets with DINO, DINOv2, and TIPS models demonstrate consistent improvements over pretrained baselines and existing adaptation methods.", "questions": "1. The experiments are conducted on datasets that feature largely object-centric videos. Could you elaborate on the expected performance and potential challenges of applying VESSA to more general, \"in-the-wild\" videos that may contain significant background clutter, camera motion, or multiple objects?\n\nPlease also check the \"weakness\" section. \nI am willing to change my score if my concerns are addressed.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method is well-motivated, with a clear and intuitive approach to adapting foundation models using video data.\n2. The training is lightweight by utilizing the limited unfrozen network layers and the adoption of LoRA.\n3. The paper is supported by comprehensive experiments and ablation studies that validate the effectiveness of the proposed components and design choices.\n\n\nWeaknesses:\n1. It is unclear whether key implementation details from the original DINO framework, such as temperature setting and teacher centering, were utilized. Clarifying this and the specific hyperparameters would help disambiguate the performance gains attributable to the proposed methods (e.g. UWSD) versus standard training techniques\n2. While the paper acknowledges catastrophic forgetting as a limitation, the extent to which it affects the model's performance on its original pretraining tasks is not explored. A preliminary analysis of this would provide a more complete picture of the trade-offs involved.", "was_revised": true, "final_justification": "I thank the authors for their response to the review comments and the complementary experiments they have conducted. I have also examined the feedback from other reviewers and the authors' corresponding responses. I appreciate the authors for their transparency regarding the quantitative performance drop from catastrophic forgetting, which represents a limitation of the proposed method, as well as their suggestion for processing \"in-the-wild\" video data.\n\nTaking these considerations into account, I raise the score for \"originality\" while maintaining the overall evaluation score unchanged.", "rebuttal": "We sincerely appreciate the reviewer's insightful feedback. Several strengths of our method were highlighted, including its motivation, clear/intuitive approach and its lightweight training. The experiments are regarded as comprehensive,validating the method’s effectiveness. We have carefully considered all the feedback and provided comprehensive responses below. All the suggested points and clarifications will be incorporated into the final version of the manuscript.\n\n> W1: It is unclear whether key implementation details from the original DINO framework, such as temperature setting and teacher centering, were utilized. Clarifying this and the specific hyperparameters would help disambiguate the performance gains attributable to the proposed methods (e.g. UWSD) versus standard training techniques\n\nThe implementation of the VESSA update component was based on DINO for training stability reasons, given the extensive empirical search reported in the DINO work and the similar environment it provides. In particular, the temperature setting and teacher centering were utilized, exactly following the DINO setup. To make it absolutely clear, we list the hyperparameters in detail here: global_crops_scale (0.14, 1.0), local_crops_scale (0.05, 0.25), student_temp (0.1), center_momentum (0.9), number_local_crops (u = 10), warmup_teacher_temp (0.04), teacher_temp (0.07), learning rate (0.001), learning_rate_schedule ('compound'), factors ('constant * cosine_decay * linear_warmup'), warmup_steps (config.steps_per_epoch * 15), steps_per_cycle (total_steps), base_learning_rate (lr * batch_size / 1024.), and alpha (0.01).\nOther key parameters were adapted due to machine memory constraints and hyperparameters specific to VESSA. These are the key adaptations: epochs (20), lora_rank (64), batch_size (128), gamma (UWSD) (1.0), and number_pairs_video (3).\nThese changes were made to ensure compatibility with the available computational resources while maintaining the effectiveness of the method. All the details will be made available on GitHub for transparency and reproducibility.\n\n> W2: While the paper acknowledges catastrophic forgetting as a limitation, the extent to which it affects the model's performance on its original pretraining tasks is not explored.\n\nWe understand the concern regarding the impact of our domain-specific adaptation on more general tasks. We would like to emphasize that our goal is to adapt the model to a specific domain and apply it solely within that target domain, where it performs very effectively for this purpose. However, acknowledging the relevance of the question, we conducted an experiment to evaluate the classification results using KNN on the ImageNet dataset. In this experiment, we compared the performance of the visual foundation models in question (DINO, DINOv2, and TIPS) both pre-trained and after adaptation with VESSA on each of the domains (CO3D, MVImageNet).\n\n| Model               | DINO  | DINOv2 | TIPS  |\n|---------------------|-------|--------|-------|\n| Pretrained          | **76.10** | **82.10**  | **80.0**  |\n| VESSA (CO3D)        | 15.46 | 17.15  | 18.10 |\n| VESSA (MVImgnet)    | 15.68 | 16.78  | 17.10 |\n\nThe first row of the table presents the results for each pre-trained VFM validated on ImageNet. The second row shows the validation results on the same ImageNet dataset after adaptation with VESSA on the CO3D data. It is evident that there is a drastic drop in performance. The same drop is observed in the third row, where the models are adapted on the MVImageNet dataset. This demonstrates the infeasibility of using the model in a general manner. Nonetheless, we reiterate that our goal is to adapt the model to a specific target domain without supervision, achieving excellent results in this setup.\n\n> Q1: The experiments are conducted on datasets that feature largely object-centric videos. Could you elaborate on the expected performance and potential challenges of applying VESSA to more general, \"in-the-wild\" videos that may contain significant background clutter, camera motion, or multiple objects?\n\nThank you very much for the suggestion on ways to expand the application of VESSA. Some of these ideas are being considered for future work that will explore this direction. One potential solution to address the challenges posed by background clutter, dynamic camera motion, and multiple objects is the incorporation of an object tracking method prior to the application of VESSA. This approach, while relying on the quality of the tracking method to ensure effective object localization, would allow VESSA to extend its capabilities to handle scenes with multiple objects and significant camera motion. In this case, this expansion would enable VESSA to perform well even in more chaotic real-world video environments, albeit with some dependence on the precision of the object tracking method employed.\nWe hope these responses provide a clearer understanding of the hyperparameters, the impact of adaptation on the visual foundation models, and a potential expansion in the application of VESSA. We greatly appreciate the feedback and look forward to further enhancing our work."}, {"review_id": "Z8IVT5BnzX", "reviewer": "Reviewer_49Dk", "summary": "This paper proposes to adapt Vision Foundation Models such as DINO or DINOv2 using video from multi-view object centric datasets. For this, they first formulate a frame selection criteria, followed by a multi-crop strategy similar to DINO and an uncertainty-weighted loss. The evaluations are conducted on these same multi-view object-centric datasets, CO3D and MVImageNet, and they show promising results. Also, the components of the method are thoroughly ablated.", "questions": "Overall, I believe the paper requires some revisions. As described in the weaknesses, some tables are a bit overcrowded, and should rather be labeled as ablation tables. The combination of different aspects of the method opens up a discussion if the proposed technical novelties of the paper are the most important for making the method work.\nFurthermore, I wonder how this adaptation of DINO also improves its performance on other tasks/datasets? Does this approach also work for other self-supervised VFMs such as iBOT or I-JEPA? These are just suggestions, other VFMs are fine to try, too.\n\nIf the authors address my concerns in a convincing way, I’m considering to update my score.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The overall idea of the paper is well motivated, sound and interesting\n- The writing of the paper is mostly clear and easy to understand\n- The ablations of the method is thorough and extensive, but presented maybe a bit confusingly\n- The results in Table 3 show that the method can be effective across multiple DINO backbones\n\nWeaknesses:\n- Limited applicability: While the method is being advertised as a video-based adaptation mechanism for VFMs, it’s only designed to work on multi-view object centric datasets, i.e. it is not applicable to any in-the-wild video. With this in mind, I think the framing of the overall paper, including the title, should be narrowed to this use case\n- The main VFM pre-training is not significantly changed, the model is trained with the standard loss with an added Uncertainty Weighting. This makes the technical contribution a bit thin.\n- In Line 253, the authors cite their motivation for using temporal information to improve the VFM. While this is technically not false, I think describing at „using views of an object from a different angle“ would probably refine it. \n- Table 1 is overfilled and a bit confusing. It’s hard to piece together what actually works and how doesn’t different numbers of unfrozen layers and the other components. Also, using local crops seems to bring the biggest increase in performance, a technique already used for the original DINO. This begs the question how relevant the proposed technical contributions are for the performance in relation. \n- Tables 1 and 2 are both ablation tables and probably should be moved to a separate ablation section.\n- It would be interesting to learn if the fine-tuning of DINO with multi-view datasets have improved its performance on benchmarks other than CO3D and MVImageNet\n- Extending the approach to other VFM such as iBOT [1] or I-JEPA [2] would also be interesting\n\n[1] Zhou, Jinghao, et al. \"Image BERT Pre-training with Online Tokenizer.\" International Conference on Learning Representations.\n[2] Assran, Mahmoud, et al. \"Self-supervised learning from images with a joint-embedding predictive architecture.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "was_revised": true, "final_justification": "After the rebuttal, the authors put in significant efforts to address my concerns. After the discussion, and the title/abstract updates, I raise my score to Borderline Accept.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We are glad that the reviewer found the paper well-motivated, sound, and interesting, with clear writing and thorough ablations. We appreciate the suggestions for improving clarity, refining the scope and framing of the paper, and exploring additional evaluations. Below, we address each of the reviewer’s concerns in detail.\n\n> W1: Limited applicability: While the method is being advertised as a video-based adaptation mechanism for VFMs, it’s only designed to work on multi-view object centric datasets, i.e. it is not applicable to any in-the-wild video. With this in mind, I think the framing of the overall paper, including the title, should be narrowed to this use case\n\nWe acknowledge that VESSA was specifically applied to object-centric videos in this work. We agree with the reviewer that clarifying this scope in the framing and title of the paper would improve its accuracy, and we are committed to making these changes. Despite this focus, the method already achieves significant improvements over the baseline on challenging multi-view datasets, demonstrating its practical value.\nWe also recognize the importance of extending VESSA to handle in-the-wild videos with more complex dynamics, which we consider an important direction for future research. One potential avenue for this extension is the incorporation of an object tracking method prior to applying VESSA. Such a pre-processing step could help isolate objects in more cluttered scenes with multiple objects or significant camera motion. While we have not experimentally validated this yet, we believe this approach could allow VESSA to operate effectively in more realistic video scenarios, and we plan to explore it in future work.\n\n> W2: The main VFM pre-training is not significantly changed, the model is trained with the standard loss with an added Uncertainty Weighting. This makes the technical contribution a bit thin.\n\nIt is important to emphasize that our contribution goes far beyond simply adding the Uncertanty-Weighted Self-Distillation (UWSD) loss. VESSA introduces a comprehensive self-supervised adaptation framework for vision foundation models (VFMs), specifically designed to leverage video information efficiently. To ensure stable adaptation, we propose a staged unfreezing approach, where we (i) initially freeze the backbone and train only the projection head, (ii) progressively unfreeze layers, and (iii) apply LoRA-based low-rank adaptation to early transformer layers, while fully fine-tuning later layers. This carefully designed strategy mitigates gradient instabilities and preserves valuable pretrained features. These combined elements form a lightweight yet effective adaptation method that consistently improves classification accuracy on domain-specific datasets (up to +10.54% in Table 3) while requiring minimal computational overhead. We therefore respectfully disagree with the characterization of the contribution as “thin”; rather, we believe the paper offers a significant and practically relevant step toward unsupervised adaptation of VFMs.\n \n> W3: In Line 253, the authors cite their motivation for using temporal information to improve the VFM. While this is technically not false, I think describing at „using views of an object from a different angle“ would probably refine it.\n\nWe appreciate this observation. While we agree that the goal of our approach is to leverage different views of an object, it is important to clarify that the benefit comes specifically from using frames of the same physical object captured over time in a video, rather than simply using different objects of the same category viewed from different angles (e.g., a front view of one wooden chair and a side view of a different plastic chair). Using frames from the same video ensures that the model learns consistent, object-centric representations and focuses less on background information, as also highlighted by our qualitative results. To make this distinction clearer, we will refine the phrasing to “using different views of the object from the video” in the final version of the paper.\n\n> W4: Table 1 is overfilled and a bit confusing. It’s hard to piece together what actually works and how doesn’t different numbers of unfrozen layers and the other components. Also, using local crops seems to bring the biggest increase in performance, a technique already used for the original DINO. This begs the question how relevant the proposed technical contributions are for the performance in relation.\n\nFor the final version, we will expand the results and improve the presentation of the tables to make the contributions of each component clearer. We would like to clarify that, contrary to the impression that local crops bring the largest performance gain, other components — particularly training the classification head and using video as input — have a much stronger impact on the results.\nFor example:\n- Using images as input with local crops (Table 1, top section, last row) achieves 88.54% accuracy, while switching to video input **without** local crops (Table 1, top section, row 2) already improves performance to 90.53%, a much larger gain than local crops alone.\n- When combining video input with local crops and training the head, accuracy reaches 91.87%, whereas using the same video setup with local crops but without training the head (Table 1, top section, penultimate row) drops to 80.87%.\nThis demonstrates that training the head and leveraging temporal information from videos are the dominant factors driving VESSA’s improvements, while local crops provide additional but comparatively smaller gains.\nThe ablation table was designed to explicitly highlight these decisions and show why each proposed step (video-based adaptation, staged unfreezing, head training, local crops, and UWSD loss) is necessary. We will make these findings clearer in the final version to avoid misinterpretation.\n\n> W5: Tables 1 and 2 are both ablation tables and probably should be moved to a separate ablation section.\n\nWe appreciate this suggestion. To improve the organization and clarity of the results presentation, we plan to create a dedicated ablation subsection in the final version of the paper. This subsection will group Tables 1 and 2 together and provide a more structured discussion of the ablation findings, making it easier for readers to follow the contribution of each component.\n\n> W6: It would be interesting to learn if the fine-tuning of DINO with multi-view datasets have improved its performance on benchmarks other than CO3D and MVImageNet\n\nWe would like to emphasize that our focus is not on creating a universal model that performs well across all multi-view datasets. Instead, VESSA is designed for efficient, domain-specific adaptation of visual foundation models. To address the reviewer’s curiosity, we conducted an additional experiment where we adapted the visual foundation models DINO and DINOv2 to MVImageNet and evaluated them on CO3D, aiming to understand the behavior when the model is adapted to one dataset and tested on another.\n\n| Method         | DINO-B     | DINOv2-B   |\n|--------------------|------------|------------|\n| Pretrained         | **78.86**      | **87.86**  |\n| Cross dataset      | 74.40  | 80.36      |\n\nThe results in the above table show a clear degradation in performance for DINO and DINOv2 in the cross-dataset scenario, indicating that adaptation is domain-specific and that improvements on one multi-view dataset do not automatically transfer to another. \nThis behavior could be seen as a limitation if the goal were to adapt the model to multi-view datasets generally, however our motivation is to apply VESSA towards specific domains, as previously mentioned. This is motivated by real-world applications, where practitioners often need to adapt a model to a specific target dataset or domain efficiently, rather than aiming for a single model that generalizes equally well across all possible scenarios.\n\n> W7: Extending the approach to other VFM such as iBOT [1] or I-JEPA [2] would also be interesting\n\n| Dataset   | CO3D  |\n|-----------|-------|\n| Pretrained| 62.79 |\n| VESSA     | **63.70** |\n\nWe appreciate the reviewer’s suggestion to evaluate other VFMs. We conducted additional experiments applying VESSA to iBOT on the CO3D dataset. using the publicly available ViT-base weights from the iBOT student network. (pre-trained on ImageNet-22K), the accuracy improved from 62.79% to 63.70% after adaptation with VESSA. Unfortunately, I-JEPA does not release ViT-Base weights, which prevented us from performing similar experiments for this model. We will include these results in the final version of the paper."}, {"review_id": "UmZ1ictnD9", "reviewer": "Reviewer_jCuG", "summary": "This paper proposes VESSA, an video-based self-supervised fine-tuning method for visual foundation models (VFMs) to achieve domain adaptation without labeled data. The method leverages object-centric temporal consistency in videos by pairing frames of the same object at different timestamps within a self-distillation framework to adapt feature representations. To mitigate forgetting during fine-tuning, the authors adopt a progressive fine-tuning scheme: first training the prediction head, then gradually unfreezing backbone layers, combined with LoRA for parameter-efficient tuning. Additionally, they introduce an Uncertainty-Weighted Self-Distillation (UWSD) loss that emphasizes learning from hard samples by weighting the distillation loss based on teacher model uncertainty. Extensive experiments on three state-of-the-art VFMs (DINO, DINOv2, TIPS) and two large-scale multi-view video datasets (MVImageNet, CO3D) demonstrate consistent improvement over original pre-trained models and competitive baselines.", "questions": "Training Cost Clarification: Please provide detailed information about the training cost, including total training time, computational resource consumption (e.g., GPU/TPU hours), and differences across models or dataset scales, to better understand the method’s practical feasibility.\n\nData Dependency and Generalization: How does the method perform when access to dense multi-view object-centric videos is limited? Could it be adapted or generalized to settings with fewer or no multi-view frames?\n\nForgetting Mitigation: Have you explored additional strategies to mitigate forgetting during fine-tuning beyond partial layer unfreezing and LoRA? Quantitative forgetting metrics or further regularization techniques would strengthen the analysis.", "limitations": "yes, authors addressed the limitations and potential negative societal impact of their work", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Innovative Technical Contributions: The paper effectively utilizes object-level video temporal consistency and uncertainty-weighted self-distillation to improve unsupervised domain adaptation of VFMs.\n- Comprehensive Experiments and detailed analysis: Evaluations on multiple backbone models and datasets corroborate the method’s general applicability and robustness. \n- Potential for Real-world Use: The method requires no manual annotations and can adapt to new data distributions with unlabeled videos.\n\nWeaknesses\n- Training Cost Not Fully Quantified: The paper reports hardware used (TPU v3-8) but lacks detailed analysis of training time, total compute requirements, or energy consumption.\n- Dependence on Multi-view Video Data: Reliance on videos depicting the same object from multiple viewpoints limits applicability where such data is scarce.\n- Forgetting Issue: Although acknowledged, there is limited discussion or experimental analysis on how forgetting is quantitatively assessed or mitigated beyond partial layer unfreezing.\n- Some Implementation Details Could Be Clearer: Additional details on hyper-parameter choices would improve reproducibility.", "was_revised": true, "final_justification": "My concerns have been addressed. I will keep the \"Rating: 5: Accept\".", "rebuttal": "We greatly appreciate the reviewer's thoughtful and detailed feedback. We are glad that our experiments are recognized as comprehensive, validating our method’s general applicability and robustness. Additionally, we appreciate the recognition of the method’s potential for real-world applications. The suggestions will be incorporated into the final version of the manuscript.\nPlease find our detailed responses and clarifications below.\n\n> Q1 (W1): Training Cost Clarification: Please provide detailed information about the training cost, including total training time, computational resource consumption (e.g., GPU/TPU hours), and differences across models or dataset scales, to better understand the method’s practical feasibility.\n\nThank you for raising this question. These details are indeed important and will be included in the final version of the paper. Below, we present an analysis from a single execution of VESSA in an adaptation process for the Co3D dataset (20,412 training pairs and 4,535 test samples) using the ViT-Base architecture.\nThe adaptation required 1.97 hours of training, corresponding to 7.04 core-hours. The total energy consumption was approximately 13.86 kWh, which translates into a carbon footprint of 5.55 kg CO₂, assuming an average emission factor of 0.4 kg CO₂/kWh (typical for data centers). The process ran for 20 epochs, processing a total of 407,040 examples at a throughput of 135 images/second (or 16.88 images/second per core), with an average rate of 1.0485 steps/second. The experiment used 8 devices in parallel, each with a local batch size of 128 and a per-device batch size of 16. The ViT-Base model contained 111,472,896 trainable parameters (including Lora parameters) during adaptation.\nIt is worth noting that, unlike DINO, which trains from scratch, VESSA adapts a pre-trained model. Even considering this distinction, when compared to the reported training cost of the original DINO model on ImageNet, as described in Emerging Properties in Self-Supervised Vision Transformers (Caron et al., 2021), the efficiency gains are clear. DINO’s full pretraining involved approximately 1,024 GPU-hours on V100 (equivalent to hundreds of kWh and a proportionally higher CO₂ footprint), while VESSA completes adaptation in less than two hours, with only a small fraction of the computational and environmental cost. This makes VESSA a computationally efficient and environmentally sustainable option for domain adaptation.\n\n> Q2 (W2): Data Dependency and Generalization: How does the method perform when access to dense multi-view object-centric videos is limited? Could it be adapted or generalized to settings with fewer or no multi-view frames?\n\nWe would like to emphasize that the primary goal of our method is adaptation using object-centric videos, for which it performs well. In line with the idea of expanding the applicability of VESSA, there are several potential directions for future work. One promising avenue would be to incorporate an object tracking stage prior to applying VESSA. This tracking component would be responsible for following and extracting multiple views of the same object, enabling a multi-view VESSA setup even in unconstrained video environments. Such an extension could broaden the applicability of the method to scenarios with fewer or no dense multi-view frames.\n\n> Q3 (W3): Forgetting Mitigation: Have you explored additional strategies to mitigate forgetting during fine-tuning beyond partial layer unfreezing and LoRA? Quantitative forgetting metrics or further regularization techniques would strengthen the analysis.\n\nWe acknowledge the concern regarding the potential impact of domain-specific adaptation on performance in broader, more general tasks. Our approach is deliberately designed to tailor the model to a specific target domain, where it consistently delivers strong results. Nonetheless, recognizing the relevance of this question, we conducted an additional evaluation to measure classification performance on the ImageNet dataset using a KNN classifier. In this assessment, we compared the visual foundation models under consideration (DINO, DINOv2, and TIPS) both in their original pre-trained state and after adaptation with VESSA on each of the target domains (CO3D and MVImageNet).\n\n| Model               | DINO  | DINOv2 | TIPS  |\n|---------------------|-------|--------|-------|\n| Pretrained          | **76.10** | **82.10**  | **80.0**  |\n| VESSA (CO3D)        | 15.46 | 17.15  | 18.10 |\n| VESSA (MVImgnet)    | 15.68 | 16.78  | 17.10 |\n\nThe first row of the table reports the baseline performance of each pre-trained visual foundation model when evaluated on ImageNet. The second row presents the results on the same dataset after adapting the models with VESSA using the CO3D data, where a substantial decrease in accuracy can be observed. A similar reduction appears in the third row, corresponding to models adapted on the MVImageNet dataset. These findings highlight that the adapted models are not suitable for general-purpose use. However, it is important to emphasize that, in practical scenarios, the adaptation is intended for a specific target domain, where performance continues to improve and remains highly effective for that domain.\n\n> W4: Some Implementation Details Could Be Clearer: Additional details on hyper-parameter choices would improve reproducibility.\n\nWe appreciate this observation and will include more comprehensive details on the hyperparameter choices in the final version of the manuscript, noting that all implementation details will also be made available in our GitHub repository. The VESSA learning technique was implemented on top of DINO, primarily to benefit from its stable training setup and the extensive empirical tuning described in the original DINO work, as well as the similar training environment it offers. Consequently, many update-related hyperparameters were inherited directly, including: global_crops_scale (0.14, 1.0), local_crops_scale (0.05, 0.25), student_temp (0.1), center_momentum (0.9), number_local_crops (10), warmup_teacher_temp (0.04), teacher_temp (0.07), learning rate (0.001), learning_rate_schedule (\"compound\"), factors (\"constant * cosine_decay * linear_warmup\"), warmup_steps (config.steps_per_epoch * 15), steps_per_cycle (total_steps), base_learning_rate (lr * batch_size / 1024.), and alpha (0.01).\nSome settings were modified to meet machine memory limitations and to incorporate VESSA-specific adjustments. These include: epochs (20), lora_rank (64), batch_size (128), gamma (UWSD) (1.0), and number_pairs_video (3). These modifications were chosen to align with the available computational resources while preserving the method’s effectiveness.\n\nWe trust that these answers offer a more comprehensive view of VESSA’s training cost, data requirements and generalization behavior, as well as its handling of forgetting and implementation aspects. All points raised will be incorporated into the final version of the paper."}, {"review_id": "PYAeoselLc", "reviewer": "Reviewer_5Abh", "summary": "This paper proposes the VESSA framework, which uses object-centric videos to improve the generalization ability of vision foundation models in an unsupervised manner. By treating the temporally distinct frames in the same video as positive pairs during contrastive training, this method enables learning robust and transferable representations.", "questions": "Please refer to the 'Weaknesses' section above.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n**1. [Good presentation]** The writing is clear and easy to follow.\n\n**2. [Well-illustrated figures]** The overview and pipeline figures are clear enough to show the workflow of the method.\n\n**3. [Carefully-designed modules]** The method proposes several necessary modules and validates through a comprehensive ablation.\n(i) Frame Selection module: In Table 1, using video as input consistently outperforms variants using image as input under the same setting. In Table 3, the authors also validates that using frame distance sampled from a random interval performs best.\n(ii) Preprocessing and Augmentation module: The random augmentaion as well as the _local crops_ are fundamental for VESSA. Using _local crops_ results in a 1.34% improvement in Table 1.\n(iii) The UWSD loss: The model can be more aware of hard samples. Using USWD loss provides a 0.95% improvement on MVImageNet.\n(iv) Fine-tuning strategy: The staged unfreezing optimization strategy detailed in Section 3 provide insights in how to prevent representational drift when fine-tuning a pre-trained foundation model. \n\n**4. [Thorough evaluations]** The experimental results show the model's superioity.\n(i) In Table 3, VESSA outperforms other baselines at most cases.\n(ii) Figure 3 demonstrates the effectiveness of VESSA using complicated samples based on nearest neighbor retrieval.\n\n**Weaknesses**\n\n**1. [Inadequate experiments]** More experimental results are necessary for varifying the efficiency and effectiveness of the method. (i) Ablations on the number of local crops: In Section 3, the paper mentioned that a set of _u_ pairs of local crops are processed by the student model. But it is unclear what this number is set and its influence on the final performance.\n(ii) Training cost comparisons on video-based and image-based method. Although VESSA outperforms its static counterpart, this method requires paired input as well as _local crops_ during training. Comparisons on the training cost would be more intuitive to show the efficient adaptation progress.\n\n**2. [Need further explorations]** Although the method has shown potential in classification task, it would be more convincing to show whether this framework is robust and generalizable to other tasks.\n(i) In Figure 3, based on the retrieval results, VESSA focus more on **object**, which may benefit from the object-centric videos during adaptation. It would be more convincing to show whether such representation improvement can generalize to other **object-centric** tasks and more quantitative results are encouraged. Instance-level recognition experiments as validated in DINOv2[1] are recommended (but not necessary).\n(ii) Since VESSA incorporates temporal dynamic information, comparisons on video classification tasks with foundation models are also encouraged.\n\n[1] DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research Journal, 2024.", "was_revised": true, "final_justification": "This work introduces an effective and efficient unsupervised domain adaptation method. The effectiveness is validated through extensive ablations on key module designs and results on different datasets/models. The efficiency is stated with training cost comparisons. Implementation details are provided for reproducibility. Catastrophic knowledge forgetting during transfer is a major limitation and is quantified. Nonetheless, this method provides valuable insights for adaptation research and practical scenarios.", "rebuttal": "We sincerely thank the reviewer for the constructive and detailed feedback. We are glad that the reviewer found the paper well written, with clear figures, carefully designed modules, and thorough evaluations. Below, we address each of the concerns.\n\n> W1: [Inadequate experiments] More experimental results are necessary for varifying the efficiency and effectiveness of the method. (i) Ablations on the number of local crops: In Section 3, the paper mentioned that a set of u pairs of local crops are processed by the student model. But it is unclear what this number is set and its influence on the final performance. (ii) Training cost comparisons on video-based and image-based methods. Although VESSA outperforms its static counterpart, this method requires paired input as well as local crops during training. Comparisons on the training cost would be more intuitive to show the efficient adaptation progress.\n\n**(i) Ablations on the number of local crops**\n\nWe appreciate the suggestion to further analyze the impact of the number of local crops. In our experiments, we adopted a fixed configuration following standard practices in state-of-the-art self-supervised methods (e.g., DINOv2), which has proven to be effective and stable across different datasets. Due to space constraints and in order to maintain focus on the primary contributions of our work, we did not include a variation of such parameters. However, we would like to remind the reviewer that we already include an experiment without local crops in Table 1 (row 2), which achieves an accuracy of 90.53, while the version using the default DINO number of local crops reaches 91.87. The improvement using local crops is significantly smaller than the improvements using video inputs (compare row 1 to row 8: 91.87 vs 88.54) or the improvements by adapting the training head before updating the backbone (compare row 1 to row 7: 91.87 vs 80.87). Therefore, we believe that variations in the number of local crops would not bring as significant gains as the other aspects we emphasize as main contributions in our work.\n\n**(ii) Training cost comparisons between video-based and image-based methods**\n\nWe agree that reporting training costs is valuable to assess the efficiency of VESSA. Our method introduces a minimal computational overhead compared to its static counterpart. In the image-based method (VESSA variant that receives only images as input, Table 1 row 8), each training step processes two augmented views of the same image. In the video-based method (VESSA variant that receives video frames as input), each training step processes two distinct frames from the same video. In both cases, the model processes exactly two images per step, leading to nearly identical GPU memory usage and computation cost.\n\nThe primary additional cost comes from video decoding, but this step is performed offline as a preprocessing stage, where all videos are decoded and frames are extracted at a fixed FPS before training. During adaptation, the training pipeline only samples pre-extracted frames, resulting in negligible additional cost and no significant impact on the overall throughput of model optimization.\n\nIt is also important to note that VESSA is not trained from scratch. Instead, it builds upon the publicly available pre-trained weights of DINOv2.. For our unsupervised adaptation stage, we measured the actual training cost (details also provided in our response to Reviewer 3, jCuG). Using a ViT-Base backbone on the Co3D dataset (20,412 training pairs, 4,535 test samples), the adaptation stage required only 1.97 hours of training (7.04 core hours) on a TPU v3-8, which consists of 4 TPUs, each with 2 cores (total of 8 cores) , consuming 13.86 kWh of energy and resulting in a carbon footprint of just 5.55 kg CO2 for 20 epochs.\n\nThis is negligible compared to the original DINOv2 training cost: according to their paper, reproducing DINOv2 alone requires 22,016 GPU-hours (9.7 MWh), emitting approximately 3.7 tons of CO2. The entire DINOv2 project is estimated to have a footprint between 0.5k and 1k tons of CO2 (around 200k GPU-days). In comparison, our adaptation stage emits a lot less CO2 than retraining DINOv2 from scratch.\n\nGiven the substantial performance gains demonstrated in Tables 1 and 3, we believe this trade-off is favorable. We will include runtime metrics in the final version of the paper and provide scripts in the released code to facilitate further cost analysis by the community.\n\n> W2: [Need further explorations] Although the method has shown potential in classification task, it would be more convincing to show whether this framework is robust and generalizable to other tasks. (i) In Figure 3, based on the retrieval results, VESSA focus more on object, which may benefit from the object-centric videos during adaptation. It would be more convincing to show whether such representation improvement can generalize to other object-centric tasks and more quantitative results are encouraged. Instance-level recognition experiments as validated in DINOv2[1] are recommended (but not necessary). (ii) Since VESSA incorporates temporal dynamic information, comparisons on video classification tasks with foundation models are also encouraged.\n\nOur primary goal in this work is to develop an effective unsupervised adaptation method tailored to specific downstream tasks, rather than creating a universal model that performs optimally on a wide variety of tasks out of the box. This task-specific adaptation focus is highly relevant in practical applications, where efficient tuning to the target domain is often more valuable than large-scale pretraining alone. While we currently do not include instance-level recognition or video classification experiments, we acknowledge the reviewer’s comment that such experiments are not strictly necessary. We agree that, with appropriate adjustments, VESSA’s use of temporal dynamics and object-centric video information could be extended to these tasks. Extending our evaluations to these settings is part of our planned future work. We view these extensions as promising research directions that naturally follow the contributions of this paper."}]

Author rebuttal:

Meta-review: The paper proposes VESSA, a method for adapting vision foundation models to new domains using unlabeled object-centric videos. Reviewers found the approach well-motivated and clearly presented, with strong experimental validation and ablation studies. Key contributions include leveraging temporal consistency in videos, staged unfreezing with LoRA for efficient fine-tuning, and an uncertainty-weighted self-distillation loss. The method demonstrates consistent improvements across multiple backbones and datasets, while maintaining efficiency compared to training from scratch. Reviewers also appreciated the comprehensive rebuttal, which clarified implementation details, quantified training costs, analyzed forgetting effects, and refined the scope of the paper by explicitly framing it as applicable to multi-view object-centric videos.

Some weaknesses remain, though they are not deal-breaking. Reviewers noted that catastrophic forgetting is significant when models are applied back to general-purpose tasks, and that the method’s reliance on multi-view object-centric video data limits broader applicability. While suggestions for extending to in-the-wild video and additional tasks were raised, reviewers agreed that these fall under future work rather than immediate requirements. Overall, the paper makes a clear and meaningful contribution to efficient adaptation of vision foundation models, and the reviewers converged on acceptance after rebuttal and discussion.
Decision: Accept (poster)