# DIP: Unsupervised Dense In-Context Post-training of Visual Representations

Sophia Sirko-Galouchenko, Spyros Gidaris, Antonin Vobecky, Andrei Bursuc, Nicolas Thome

► To cite this version:

Sophia Sirko-Galouchenko, Spyros Gidaris, Antonin Vobecky, Andrei Bursuc, Nicolas Thome. DIP: Unsupervised Dense In-Context Post-training of Visual Representations. ICCV 2025, 2025. hal-05507571

HAL Id: hal-05507571

https://hal.science/hal-05507571v1

Submitted on 12 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
arXiv:2506.18463v3 [cs.CV] 9 Sep 2025

# DIP: Unsupervised Dense In-Context Post-training of Visual Representations

Sophia Sirko-Galouchenko¹,² Spyros Gidaris¹

Antonin Vobecky¹,³,⁴ Andrei Bursuc¹ Nicolas Thome²,⁵

¹Valeo.ai

²Sorbonne Université, CNRS, ISIR, F-75005 Paris, France

³FEE CTU

⁴CIIRC CTU Prague

⁵Institut universitaire de France (IUF)

<figure><img src="image_2.png" /><figcaption>Figure 1. Dense retrieval-based semantic segmentation in low-shot regimes (40 examples) using patch feature similarities. For each query image (a), we show the ground truth (b), our DIP model's prediction (c), and its retrieved neighbor (d) for the patch marked with a red cross in (a). In (e) and (f), we display DINOv2R's (Dv2R) output and its nearest neighbor, respectively. Our DIP representations retrieve more coherent neighbors, yielding more accurate segmentations than DINOv2R.</figcaption></figure>

## Abstract

We introduce DIP, a novel unsupervised post-training method designed to enhance dense representations in large-scale pretrained vision encoders for in-context scene understanding. Unlike prior approaches using complex self-distillation architectures, our method trains the vision encoder using pseudo-tasks that simulate downstream in-context scenarios, inspired by meta-learning principles. To enable post-training on unlabeled data, we propose an automatic mechanism for generating in-context tasks that combines a pretrained diffusion model and the vision encoder. DIP is simple, unsupervised, and computationally efficient, requiring under 9 hours on a single A100 GPU. By learning dense representations through pseudo in-context tasks, it achieves strong performance across a variety of downstream real-world in-context scene understanding tasks. It outperforms both the initial vision encoder and prior methods, offering a practical and effective solution for improving dense representations. See code here

## 1. Introduction

Our goal is to learn dense image representations from unlabeled data for effective in-context scene understanding. In-context learning, which enables models to adapt to tasks without updating parameters, has seen remarkable success in large language models (LLMs) [7]. By providing a few examples within the input prompt, LLMs generalize effectively to new tasks. Inspired by this, recent efforts aim to bring similar in-context learning capabilities to vision-only models [3, 5]. A notable approach by Balazevic et al. [3] reformulates dense prediction tasks as nearest-neighbor retrieval problems using patch feature similarities, demonstrating strong performance, especially with limited data. However, while recent self-supervised Vision Transformers (ViTs) [3, 56, 75] show promise for in-context scene understanding, they still fall short of matching LLMs' in-context learning success. To bridge this gap, it is crucial to learn dense features that establish strong semantic correspondences between patches in test and training images.

A key insight from prior work [58, 64, 84] is that leverag-
ing pretrained Vision Foundation Models (VFMs), such as DINOv2R [16, 56], and specializing them through unsupervised post-training is more computationally efficient and effective than training from scratch. This aligns with trends in LLMs, where large-scale pretraining is followed by cheaper post-training stages to specialize in specific skills of interest. We refer to the pretrained VFM undergoing post-training as the *base model*.

Most methods, whether trained from scratch or post-trained, rely on self-distillation, where the teacher is an exponential moving average of student weights [26]. These frameworks use architectures and objectives designed to promote dense nearest-neighbor retrieval skills. For instance, state-of-the-art methods enforce object consistency across images and views [46] or between patch rankings from two random views of the same image [58]. However, self-distillation often suffers from poor stability [83] and hyperparameter sensitivity [2]. Moreover, these approaches introduce additional complexity, such as differentiable sorting [58], RoI alignment units [29, 58, 84], Sinkhorn-Knopp optimization [64, 84], or learnable patch pooling [3, 46]. While improving performance, these components make the methods harder to interpret and tune, and less transparent in terms of their underlying mechanisms.

In this work, we adopt a post-training approach due to its computational efficiency and effectiveness. Within this context, we address two key questions: (1) *Are there simpler and more effective alternatives to existing self-distillation frameworks with complex architectures and objectives?*, and (2) *Beyond the base model, can we leverage other vision foundation models to achieve our goal?*

We propose a simpler post-training approach with an unsupervised learning objective explicitly designed for in-context scene understanding. Inspired by meta-learning, which learns general knowledge from diverse tasks to address unseen but related tasks, we automatically construct multiple in-context tasks from unlabeled data and train the model on them. This enables the model to acquire transferable dense representations for efficiently solving new in-context tasks during downstream stages. Unlike prior methods, our approach eliminates complex training objectives and architectures, and avoids self-distillation frameworks.

To achieve unsupervised post-training, we generate in-context pseudo-tasks that mimic the tasks encountered during downstream use. Each pseudo-task consists of support examples with segmentation masks and a query image segmented using the support examples. The support set includes a positive example (sharing objects with the query) and “distractor” examples (likely unrelated). Our pseudo-tasks resemble meta-learning *episodes* [76] with the difference that we do not use any manual annotations.

Instead, we leverage pretrained VFM to construct these pseudo-tasks. Specifically, apart from the base model (DI-

NOv2R) that we post-train, we use an auxiliary pretrained generative model, Stable Diffusion (SD) [27]. SD provides high-fidelity, class-agnostic image segments in an unsupervised manner, while the base model retrieves candidate positives for each query and assigns pseudo-labels to the segments. Although SD is pre-trained on image-caption pairs, we consider our post-training unsupervised as we do not use captions to extract image segments from SD, and neither SD nor the base model rely on dense segmentation labels.

The main contributions of this paper are as follows:

* We introduce DIP, a novel unsupervised post-training method that uses retrieval-based in-context learning to improve dense image representations. Unlike prior work, DIP is based on meta-training principles and eliminates the need for complex self-distillation architectures. Furthermore, DIP is computationally efficient, requiring less than 9 hours on a single A100 GPU for post-training.

* To enable post-training on unlabeled data, we propose an automatic mechanism for generating in-context tasks. In addition to the base model, this approach leverages the Stable Diffusion model (through the unsupervised, training-free DiffCut [15] technique) to produce high-quality pseudo-segmentations from unlabeled images.

* Experiments show that the learned dense representations generalize effectively to a wide variety of downstream tasks. These include six semantic segmentation datasets and one monocular depth prediction dataset, even in low-shot scenarios. Furthermore, we extend our approach to other VFM, such as CLIP [61], and show that it consistently enhances their dense representations.

* Compared to DINOv2R, DIP produces more semantically coherent neighbors, leading to more accurate retrieval-based scene understanding, particularly in low-shot settings (see Fig. 1). Additionally, DIP outperforms recent state-of-the-art post-training methods in most retrieval-based tasks while being simpler.

## 2. Related Work

**Self-supervised learning.** Self-supervised methods learn rich representations from vast amounts of raw data via pre-text tasks providing supervision signals from the data, later enabling fine-tuning on downstream tasks with minimal annotations. The main self-supervised approaches for images include: contrastive objectives that distinguish similar and dissimilar views [11, 30, 54], clustering-based objectives [1, 8, 23], self-distillation objectives that match representations across augmentations [6, 9, 12, 24, 26], and reconstruction objectives that predict pixel colors [31, 80] or masked patch features from a teacher network [2, 25, 39, 55, 75, 83]. Recent approaches use pretrained VFM for distillation in cooperation with self-supervised objectives [4, 17, 36, 59]. These methods focus on learning
global image representations for classification tasks. However, without full-finetuning or task decoders, they are less suitable for dense scene-understanding tasks like segmentation. We build DIP on contrastive objectives [54] equipped with a memory bank [30].

**Dense self-supervision.** Learning localized image representations that can be rapidly adapted to scene understanding requires specialized self-supervised objectives that mimic downstream tasks [3, 10, 33, 34, 46, 68, 71, 78, 79]. One approach contrasts features from dissimilar pixels or pseudo-segments within images, improving object detection [33, 34, 53, 78, 79]. Others employ clustering strategies to produce object-aware supervision signals to improve semantic segmentation [10, 71, 84], while some extend patch-level contrastive learning across images [3] with nearest-neighbor consistency [28, 46, 58]. Although we share similar objectives with [3, 58] for in-context scene understanding, we introduce a novel self-supervised objective inspired by few-shot learning that leverages automatically computed object segment pseudo-labels.

Recent works build upon pretrained VFMs [28, 41, 58, 69, 84] casting their self-supervised training as a post-training stage for preparing the VFM for dense downstream tasks. Similarly, DIP aims to rapidly endow pretrained VFMs with in-context dense reasoning skills.

**Unsupervised semantic segmentation.** These works aim to produce object segments via self-supervised training. Earlier works employed clustering objectives, with segmentations refined via consistency across augmented views [13, 37, 38, 40, 77]. Other works explore pretrained features from self-supervised VFMs [28, 45, 47, 66, 77] or diffusion UNet encoders [15, 73]. The recent DiffCut [15] leverages recursive Normalized Cuts in the final attention layer of a pretrained diffusion UNet encoder to produce object segments without supervision. We use DiffCut to extract pseudo-segmentation maps for our in-context post-training, as these segments - while sometimes noisy - serve as effective proxies for actual object segments in our framework.

**Few-shot learning.** Few-shot learning develops models capable of rapid adaptation from limited labeled examples. These models are typically trained on few-shot tasks (or *episodes*) sampled from the training data, that simulate test-time conditions. Existing approaches include memory-based methods that match queries to labeled examples [50, 65, 76], metric-learning techniques that learn distance functions from few samples [70, 72], optimization-based methods that learn efficient adaptation through few gradient steps [20, 62], and generative approaches that predict classifier weights [21, 22, 60]. Inspired by Vinyals et al. [76], we devise a self-supervised pretext objective consisting of multiple in-context scene understanding tasks, each with a set of support samples stored in a memory and a

query image to be segmented based on the support samples. In contrast to few-shot learning, we use no human annotation during post-training, generating labels automatically.

## 3. Method

Our goal is to fine-tune a pretrained vision encoder, typically a ViT [18], to produce dense features suitable for in-context dense prediction tasks framed as nearest-neighbor retrieval. Given a set of support images (i.e., training images) with semantic segmentation annotations and a query image (i.e., test image), we aim to ensure that patch-wise features extracted from the query image are highly similar (e.g., high cosine similarity) to support patches of the same object category, while being dissimilar to features of different objects.

To achieve this, we propose an unsupervised post-training approach (Fig. 2) that fine-tunes the encoder on dense nearest-neighbor retrieval tasks automatically generated from unlabeled data. Training on pseudo-tasks that simulate real scenarios, the encoder learns robust features transferable to real in-context tasks.

In the following sections, we first detail our dense in-context post-training approach (Sec. 3.1) and then describe the automatic process for constructing dense in-context tasks (Sec. 3.2), including the pseudo-labeling of class-agnostic segments and the selection of support examples.

### 3.1. Dense In-Context Training

During training, we sample an in-context task $\mathcal{T}$ from dataset $\mathcal{D}$. Each task $\mathcal{T}$ consists of $K$ support examples $S = \{(X_{s_i}, Y_{s_i})\}_{i=1}^K$ and one query example $(X_q, Y_q)$, where $X_q$ and $X_{s_i}$ are images, and $Y_q$ and $Y_{s_i}$ are their one-hot semantic segmentation pseudo-labels. Among the $K$ support examples, one is a “positive” example sharing pseudo-classes with the query, while the remaining $K-1$ are “distractor” examples unlikely to share any pseudo-class. Since our method is unsupervised, these tasks and pseudo-labels are automatically constructed (see Sec. 3.2).

**Feature extraction.** Let $f(\cdot)$ be a pretrained encoder that our method fine-tunes. Given an input image $X$ (either query or support), $f(\cdot)$ produces patch-wise features $f(X) \in \mathbb{R}^{L \times D}$, where $L$ is the number of patches and $D$ is the feature dimension. We further apply a patch-wise multi-layer perceptron (MLP) network $h(\cdot)$ to project the features into a $D'$-dimensional space: $F = h(f(X)) \in \mathbb{R}^{L \times D'}$. The MLP includes an $\ell_2$-normalization layer at the end. Let $\theta$ denote the parameters of $(h \circ f)(\cdot)$ model.

**Label pre-processing.** To align label resolution with patch features, we *patchify* the one-hot pseudo-labels $Y_{s_i}$. We divide $Y_{s_i}$ into patches matching the encoder's patch size and average the one-hot labels within each patch, yielding $Y'_{s_i} \in \mathbb{R}^{L \times C}$, where $C$ is the number of pseudo-classes.
<figure><img src="image_3.png" /><figcaption>**Figure 2. Our unsupervised Dense In-context Post-training (DIP) method.** During post-training, the model is given a pseudo in-context task, created automatically without human input. Each task includes a query image and a pseudo-labeled support set with a positive example (sharing objects with the query, as shown in the zebra example) and “distractor” examples that contain different object categories. The model predicts the pseudo-labeled semantic segmentation of the query image using the support set as reference. To do this, it (1) extracts patch-wise features from both the query and support images using the vision encoder $f(·)$ and projection head $h(·)$; (2) computes segmentation predictions for the query image through cross-attention (query patch features as queries, support patch features as keys, and pseudo-labeled support patches as values). A pixel-wise cross-entropy loss is applied to the predictions. By training on these pseudo tasks, DIP enables the encoder to learn transferable dense representations, which are later used to efficiently solve new real in-context tasks.</figcaption></figure>

**In-context dense predictions.** Using patch-wise features and patchified labels from the support set $S$, we define a soft nearest-neighbor classification function $c_S(·)$ to predict dense labels $\hat{Y}_q = c_S(X_q)$ for the query image $X_q$. First, we compute attention scores for each query patch over all $K \times L$ support patches:

$$A = \mathrm{softmax} \left( \frac{F_q \cdot F_S^T}{\tau} \right), \qquad (1)$$

where $F_S \in \mathbb{R}^{(L \cdot K) \times D'}$ concatenates patch-wise features $F_{s_i}$ from all support images, and $\tau$ is the softmax temperature. The softmax normalization is over the support patches. The attention weights $A \in [0, 1]^{L \times (L \cdot K)}$ are used to compute a weighted average of the patchified support labels:

$$\hat{Y}'_q = A \cdot Y'_S, \qquad (2)$$

where $Y'_S \in \mathbb{R}^{(L \cdot K) \times C}$ concatenates patchified labels $Y'_{s_i}$ from all support images, and $\hat{Y}'_q \in \mathbb{R}^{L \times C}$ represents the predicted patchified labels for query. Essentially, this defines a cross-attention layer [74], with $F_q$ as queries, $F_S$ as keys, and $Y'_S$ as values.

Finally, we use nearest-neighbor interpolation to upsample $\hat{Y}'_q$ to the original image size, yielding the final label prediction $\hat{Y}_q = c_S(X_q)$.

**Training objective.** For a single task $T=\{S, (X_q, Y_q)\}$, we minimize the pixel-wise cross-entropy loss $\mathcal{L}_{CE}(Y_q, c_S(X_q), \theta)$ between the predicted labels $c_S(X_q)$ and the pseudo-labels $Y_q$. The model, comprising the

pretrained encoder and randomly initialized MLP $(h \circ f)(\cdot)$ with parameters $\theta$, is trained by optimizing the expected loss over a collection of tasks sampled from $D$:

$$\min_{\theta} \mathbb{E}_{\{S, (X_q, Y_q)\} \sim D} [\mathcal{L}_{CE}(Y_q, c_S(X_q), \theta)]. \quad (3)$$

**Downstream stage.** For downstream in-context scene understanding tasks, we remove the head $h(·)$ and use the fine-tuned encoder $f(·)$. Following [3, 57, 58], we construct a larger memory bank (support set) than used during post-training, with 10,240,000 patch-wise features randomly sampled from the available training images for each dataset. For a query image, we extract its patch-wise features using $f(·)$, retrieve the top-30 nearest neighbors from the memory bank via cosine similarity, and apply cross-attention (as in Eqs. (1) and (2)) with temperature $\tau = 0.07$.

## 3.2. Automatic Dense In-Context Task Construction

To apply our dense in-context post-training approach to unlabeled data, we automatically generate in-context pseudo-tasks with spatially coherent segmentation labels. This involves two steps: (1) generating pseudo-segmentation maps for dataset images and (2) selecting query and support examples to form each in-context task. These steps create meaningful and challenging tasks, enabling the model to learn transferable in-context visual learning skills. Examples of constructed pseudo-tasks are shown in Fig. 3.

**Generating pseudo semantic segmentation labels.** We generate pseudo-segmentation maps by first dividing each image into non-overlapping class-agnostic segments and then assigning a pseudo-label to each segment.
<figure><img src="image_4.png" /><figcaption>**Figure 3.** Examples of automatically constructed in-context scene understanding tasks. Each row shows a query image and its corresponding positive support example. (a) and (b) display the query image and its pseudo segmentation labels, while (c) and (d) show the positive support image and its pseudo segmentation labels. Despite being generated in a fully unsupervised manner, the segmentation masks for salient objects are highly accurate, closely matching the actual objects. In addition, the query and positive image pairs share a common object with the same pseudo-label. During post-training with our DIP method, the model predicts the query image's segmentation using the positive example as a reference, along with distractor support examples (randomly sampled from other images in the mini-batch, not shown here for brevity).</figcaption></figure>

For the first step, we use DiffCut [15], a training-free zero-shot image segmentation method. DiffCut leverages features from a pretrained diffusion model, SSD-1B [27], along with a recursive graph partitioning algorithm to produce fine-grained segmentation maps. These maps, called DiffCut masks, are generated and stored for the entire pre-training dataset. Since DiffCut is unsupervised, our pre-training approach remains fully annotation-free.

For the second step, we assign pseudo-labels to each segment using the self-supervised DINOv2R feature encoder. We compute the mean DINOv2R feature for each DiffCut mask by pooling dense features within the mask region. To group visually similar segments, we apply K-means clustering on these pooled features<fnref n="1" />. The resulting clusters serve as pseudo-classes for annotating the DiffCut masks. To assign a pseudo-class to each DiffCut mask, we first assign each pixel-wise DINOv2R feature the cluster ID of its closest K-means centroid. Then, for each DiffCut mask, we perform majority voting on all pixels within the mask and assign the most frequent cluster ID. This ensures robust and consistent pseudo-labels, even with noisy features.

**Selecting examples for in-context tasks.** To generate in-context tasks, we pair each query image with one “positive”

<figure><img src="image_5.png" /><figcaption>**Figure 4.** In-context scene understanding in low-shot regimes. mIoU results with ViT-B/14 versus training data size.</figcaption></figure>

support image and K-1 “distractor” support images.

To find positive images sharing similar visual content, we use DINOv2R global image representations. For each image, we retrieve five nearest neighbors using DINOv2R global features. This yields five positive pairs per image. We then retain pairs where the query and positive-support images share a common pseudo-class occupying more than 5% of each image's area. This ensures semantically meaningful positive pairs. The final refined list contains all possible query-positive support pairs for training.

For the K-1 “distractor” examples, we randomly sample images from the current mini-batch during training. Including distractors encourages the model to distinguish relevant from irrelevant content, improving feature discriminability.

# 4. Experiments

## 4.1. Experimental Setup

**Compared methods.** We compare our method with state-of-the-art unsupervised learning approaches, including DINO [9], Leopart [84], TimeT [64], iBOT [83], CrOC [71], CrIBO [46], DINOv2R [16, 56], and NeCo [58]. NeCo, a recent method (ICLR'25), is the most related to ours, as it also post-trains DINOv2R for dense representation learning in in-context scene understanding tasks. However, our post-training methodology differs significantly: we explicitly train on in-context pseudo-tasks relevant to our target applications, while NeCo adopts a self-distillation-based approach.

**Training setup.** We apply our unsupervised post-training to DINOv2 [56] with registers [16] (DINOv2R), using ViT-S/14 and ViT-B/14 models. In-context pseudo-tasks are generated using DiffCut and the base model (e.g., DINOv2R ViT-B/14 is used to generate pseudo-tasks for post-training DINOv2R ViT-B/14; see Sec. 3.2). Except otherwise stated, post-training is on COCO for 5 epochs with 1,000 pseudo-classes. We also explore post-training CLIP

<footnote n="1">¹For COCO [48], clustering is performed on the full dataset; for ImageNet [44], clustering is performed on a random subset of 200,000 training images for efficiency.</footnote>
<table><caption>Table 1. **In-context scene understanding with few training examples.** Dense nearest neighbor retrieval performance on ADE20K and PascalVOC datasets, evaluated using the mIoU metric across varying proportions of training data. The fractions $x/y$ below the dataset names indicate the proportion of training data used. Results marked with † are from NeCo [58], while those marked with ‡ are our own.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Backbone</th><th colspan="4">PascalVOC</th><th colspan="4">ADE20K</th></tr><tr><th><sup>1</sup>/<sub>1</sub></th><th><sup>1</sup>/<sub>8</sub></th><th><sup>1</sup>/<sub>64</sub></th><th><sup>1</sup>/<sub>128</sub></th><th><sup>1</sup>/<sub>1</sub></th><th><sup>1</sup>/<sub>8</sub></th><th><sup>1</sup>/<sub>64</sub></th><th><sup>1</sup>/<sub>128</sub></th></tr></thead><tbody><tr><td>DINO<sup>†</sup> [9]</td><td>ViT-S/16</td><td>48.7</td><td>41.3</td><td>30.5</td><td>26.4</td><td>17.9</td><td>15.0</td><td>11.0</td><td>9.5</td></tr><tr><td>SelfPatch<sup>†</sup> [81]</td><td>ViT-S/16</td><td>50.8</td><td>43.2</td><td>32.6</td><td>28.4</td><td>17.7</td><td>14.7</td><td>10.9</td><td>10.0</td></tr><tr><td>CroC<sup>†</sup> [71]</td><td>ViT-S/16</td><td>60.5</td><td>53.8</td><td>41.8</td><td>34.0</td><td>17.3</td><td>15.2</td><td>10.8</td><td>8.7</td></tr><tr><td>TimeT<sup>†</sup> [64]</td><td>ViT-S/16</td><td>62.3</td><td>55.2</td><td>43.8</td><td>38.1</td><td>23.2</td><td>18.9</td><td>14.1</td><td>12.1</td></tr><tr><td>Leopart<sup>†</sup> [84]</td><td>ViT-S/16</td><td>64.5</td><td>58.4</td><td>49.7</td><td>44.6</td><td>23.9</td><td>19.6</td><td>14.8</td><td>12.9</td></tr><tr><td>CriBO<sup>†</sup> [46]</td><td>ViT-S/16</td><td>72.4</td><td>66.9</td><td>59.9</td><td>53.9</td><td>26.6</td><td>22.7</td><td>17.3</td><td>14.6</td></tr><tr><td>DINOv2R<sup>†</sup> [16]</td><td>ViT-S/14</td><td>79.4</td><td>75.2</td><td>67.7</td><td>60.7</td><td>39.3</td><td>33.2</td><td>24.9</td><td>22.6</td></tr><tr><td>NeCo<sup>‡</sup> [58]</td><td>ViT-S/14</td><td><b>81.0</b></td><td>77.3</td><td><b>71.5</b></td><td>65.8</td><td>38.9</td><td>32.5</td><td>24.1</td><td>21.9</td></tr><tr><td>DIP (ours)<sup>‡</sup></td><td>ViT-S/14</td><td><b>81.0</b></td><td><b>77.7</b></td><td>71.4</td><td><b>65.9</b></td><td><b>39.7</b></td><td><b>33.7</b></td><td><b>25.6</b></td><td><b>23.2</b></td></tr><tr><td>DINO<sup>†</sup> [9]</td><td>ViT-B/16</td><td>57.3</td><td>49.8</td><td>37.7</td><td>33.1</td><td>21.5</td><td>18.2</td><td>13.5</td><td>11.5</td></tr><tr><td>Leopart<sup>†</sup> [84]</td><td>ViT-B/16</td><td>69.5</td><td>63.1</td><td>54.7</td><td>50.1</td><td>26.7</td><td>21.8</td><td>16.8</td><td>14.6</td></tr><tr><td>Hummingbird<sup>†</sup> [3]</td><td>ViT-B/16</td><td>71.8</td><td>64.3</td><td>57.2</td><td>50.5</td><td>29.6</td><td>22.3</td><td>15.1</td><td>11.7</td></tr><tr><td>CriBO<sup>†</sup> [46]</td><td>ViT-B/16</td><td>74.2</td><td>69.2</td><td>61.8</td><td>55.9</td><td>28.4</td><td>24.4</td><td>18.4</td><td>15.9</td></tr><tr><td>DINOv2R<sup>‡</sup> [16]</td><td>ViT-B/14</td><td>79.0</td><td>75.3</td><td>67.6</td><td>60.3</td><td>40.8</td><td>35.3</td><td>27.3</td><td>24.9</td></tr><tr><td>NeCo<sup>‡</sup> [58]</td><td>ViT-B/14</td><td><b>82.4</b></td><td>78.7</td><td>71.7</td><td>65.0</td><td>41.2</td><td>35.2</td><td>27.2</td><td>25.1</td></tr><tr><td>DIP (ours)<sup>‡</sup></td><td>ViT-B/14</td><td>82.1</td><td><b>79.6</b></td><td><b>75.1</b></td><td><b>70.1</b></td><td><b>42.6</b></td><td><b>36.8</b></td><td><b>29.4</b></td><td><b>27.0</b></td></tr></tbody></table>

and MAE models with ViT-B/16 (using pseudo-labels from DiffCut and DINOv2R ViT-B/14), as well as post-training on ImageNet. Our post-training is efficient, requiring 17h52min on V100 (ViT-B) and 8h37min on A100 (ViT-S). Pseudo-label generation for COCO, which is done only once and offline, requires 17h on a V100 GPU. More implementation details in Sec. A.

**Evaluation setup.** To assess our representations' generality, we evaluate them on diverse tasks and datasets. These include semantic segmentation on PascalVOC [19], Pascal-Context [51], ADE20K [82], Cityscapes [14], COCO [48] (mIoU↑), and monocular depth prediction on NYUv2 [52] (using RMSE↓). COCO is in-domain, as its training data is used for post-training, while the others are out-of-domain. These datasets test generalization to domain shifts between post-training and downstream tasks. We also evaluate robustness to intra-task domain shifts using the CS→ACDC [63] setting, where Cityscapes (CS) provides the support set and ACDC provides the query images for retrieval-based segmentation. While both datasets contain autonomous driving images, ACDC introduces challenging conditions like snow, night, fog, and rain, unlike Cityscapes' clear-weather daylight images.

All results reported in this paper are produced using our implementation unless marked with †. For DINOv2R and NeCo, we evaluate the publicly available pretrained model checkpoints to ensure a fair comparison.

## 4.2. Retrieval-based Scene Understanding

**Comparison with state-of-the-art in low-shot regimes.** In Tab. 1, we compare our DIP approach with prior state-

of-the-art methods for dense image representation learning on retrieval-based semantic segmentation using the Pascal VOC and ADE20K datasets. We evaluate both full-data ($\frac{1}{1}$) and low-shot regimes ($\frac{1}{8}$, $\frac{1}{64}$, and $\frac{1}{128}$), where only a fraction of the training examples is used in the support set.

Our DIP approach consistently outperforms DINOv2R, with the performance gap widening as the number of training examples decreases, particularly for the PascalVOC dataset (see Fig. 4). Notably, DIP achieves superior results compared to prior work in most settings for both ViT-S and ViT-B, while remaining competitive in others.

**Comparison with DINOv2R and NeCo.** We compare our DIP method with DINOv2R, the base model for our post-training, and NeCo, a recent method that also post-trains DINOv2R. In Tab. 2 we evaluate these methods on in-context semantic segmentation tasks across six datasets.

Our approach shows consistent improvements over DINOv2R on all datasets, both in-domain (COCO) and out-of-domain (ADE20K, PascalVOC, Pascal-Context, and Cityscapes), highlighting the generalization capability of our representations. Unlike NeCo, our DIP consistently improves over DINOv2R, achieving higher mIoU and average performance improvements (see “Avg. Delta” in Tab. 2).

**Intra-task domain shift.** In Tab. 2, the CS→ACDC setting evaluates the robustness of the learned representations to domain shifts between the support set and query images during the downstream stage, as explained in Sec. 4.1. Although our post-training DIP method is not specifically designed to improve this type of robustness, it still achieves a 1.6 mIoU improvement over DINOv2R for ViT-B.
<table><caption>Table 2. **In-context scene understanding benchmark.** Dense nearest neighbor retrieval performance for semantic segmentation on six scene-centric datasets: ADE20K, PascalVOC, Pascal-Context, Cityscapes, CS→ACDC, and COCO. The first five are out-of-domain, while COCO is in-domain (used for post-training). Performance is measured using the mIoU metric. For NeCo and our DIP, which post-train DINOv2R, improvements over DINOv2R are shown in parentheses. The “Avg. Delta” column reports the average improvement.</caption><thead><tr><th>Method</th><th>Backbone</th><th>ADE20K</th><th>PascalVOC</th><th>Pascal-Context</th><th>Cityscapes</th><th>CS→ACDC</th><th>COCO</th><th>Avg. Delta</th></tr></thead><tbody><tr><td>DINOv2R [16]</td><td>ViT-S/14</td><td>39.3</td><td>79.4</td><td>48.0</td><td>55.6</td><td><strong>47.4</strong></td><td>72.6</td><td></td></tr><tr><td>NeCo [58]</td><td>ViT-S/14</td><td>38.9 (-0.4)</td><td><strong>81.0</strong> (+1.6)</td><td>49.4 (+1.4)</td><td>53.9 (-1.7)</td><td>47.2 (-0.2)</td><td><strong>74.3</strong> (+1.7)</td><td>+0.40</td></tr><tr><td><strong>DIP</strong> (ours)</td><td>ViT-S/14</td><td><strong>39.7</strong> (+0.4)</td><td><strong>81.0</strong> (+1.6)</td><td><strong>49.5</strong> (+1.5)</td><td><strong>55.8</strong> (+0.2)</td><td><strong>47.4</strong> (+0.0)</td><td>74.0 (+1.4)</td><td><strong>+0.85</strong></td></tr><tr><td>DINOv2R [16]</td><td>ViT-B/14</td><td>40.8</td><td>79.0</td><td>49.0</td><td>58.4</td><td>50.5</td><td>72.9</td><td></td></tr><tr><td>NeCo [58]</td><td>ViT-B/14</td><td>41.2 (+0.4)</td><td><strong>82.4</strong> (+2.4)</td><td>51.2 (+2.2)</td><td>57.9 (-0.5)</td><td>51.5 (+1.0)</td><td>75.4 (+2.5)</td><td>+1.50</td></tr><tr><td><strong>DIP</strong> (ours)</td><td>ViT-B/14</td><td><strong>42.6</strong> (+1.8)</td><td>82.1 (+2.1)</td><td><strong>51.5</strong> (+2.5)</td><td><strong>59.5</strong> (+1.1)</td><td><strong>52.1</strong> (+1.6)</td><td><strong>76.0</strong> (+3.1)</td><td><strong>+2.00</strong></td></tr></tbody></table>

<table><caption>Table 3. **In-context monocular depth prediction on NYUv2 dataset** [67]. RMSE scores (lower is better) scaled by 10 for readability reasons. All methods use ViT-S/14.</caption><thead><tr><th>Method</th><th>DINOv2R</th><th>NeCo</th><th>DIP (ours)</th></tr></thead><tbody><tr><td>RMSE↓</td><td>.771</td><td>.769</td><td><strong>.756</strong></td></tr></tbody></table>

<table><caption>Table 4. **Post-training with other base models.** Dense nearest neighbor retrieval performance (mIoU) for semantic segmentation on ADE20K and PascalVOC, using DINOv2R, CLIP, and MAE as base models, before and after post-training.</caption><thead><tr><th>Method</th><th>Backbone</th><th>PascalVOC</th><th>ADE20K</th></tr></thead><tbody><tr><td>DINOv2R [16]</td><td>ViT-B/14</td><td>79.0</td><td>40.8</td></tr><tr><td>+DIP</td><td>ViT-B/14</td><td>82.1 (+2.1)</td><td>42.6 (+1.8)</td></tr><tr><td>CLIP [61]</td><td>ViT-B/16</td><td>71.8</td><td>29.0</td></tr><tr><td>+DIP</td><td>ViT-B/16</td><td>73.8 (+2.0)</td><td>30.1 (+1.1)</td></tr><tr><td>MAE [31]</td><td>ViT-B/16</td><td>13.9</td><td>5.1</td></tr><tr><td>+DIP</td><td>ViT-B/16</td><td>47.3 (+33.4)</td><td>11.8 (+6.7)</td></tr></tbody></table>

**Monocular depth prediction.** In Tab. 3, we report in-context depth results on NYUv2 (RMSE). Both DIP and NeCo improve over DINOv2R, with DIP achieving the best RMSE. This is notable, as our in-context pseudo-tasks are designed for semantic segmentation, yet the representations generalize well to depth prediction.

**Post-training other base models.** In Tab. 4, we evaluate in-context segmentation on ADE20K and PascalVOC using CLIP [61] and MAE [31] as base models, before and after DIP post-training. Our method consistently improves performance across all tested models, showing its versatility. MAE improves by +33.4 mIoU on PascalVOC. These gains transform MAE from weak to improved at in-context segmentation, highlighting our approach's effectiveness in enhancing dense features.

**Comparison with supervised baselines** To automatically generate pseudo-segmentations, our post-training approach leverages Stable Diffusion (SD) features [27], trained on weakly annotated (internet-scraped) image-caption pairs. In Tab. 5, we compare DIP to SD and supervised encoder features (SAM [43] and RADIOv2.5 [32]). Crucially, SAM

<table><caption>Table 5. **Comparison with Superised Baselines.** Dense nearest neighbor retrieval performance (mIoU) for semantic segmentation on PascalVOC and ADE20K.</caption><thead><tr><th>Method</th><th>Backbone</th><th>PascalVOC</th><th>ADE20K</th></tr></thead><tbody><tr><td>SD [27]</td><td>SSD-1B</td><td>59.4</td><td>24.4</td></tr><tr><td>SAM [43]</td><td>ViT-B/16</td><td>32.8</td><td>12.9</td></tr><tr><td>DINOv2R [16]</td><td>ViT-B/14</td><td>79.0</td><td>40.8</td></tr><tr><td>RADIOv2.5 [32]</td><td>ViT-B/16</td><td>81.3</td><td>42.1</td></tr><tr><td>DIP (ours)</td><td>ViT-B/14</td><td><strong>82.1</strong></td><td><strong>42.6</strong></td></tr></tbody></table>

requires stronger supervision (manual segmentation masks) than SD, while RADIOv2.5 distills multiple model types (SAM, DINOv2, and CLIP-like features). Thus, both baselines leverage more supervision than our method. Additionally, RADIOv2.5 requires significantly more training compute. Despite these advantages, Tab. 5 shows DIP surpasses RADIOv2.5 in in-context semantic segmentation, confirming our method's effectiveness. SAM underperforms in this setting, consistent with prior findings [42]. Importantly, DIP outperforms the SD features it uses for pseudo-segmentation.

Additionally, we report linear segmentation results with consistent gains over DINOv2R and competitive results compared to NeCo in Tab. 8 in supplementary.

### 4.3. Ablations

In Tab. 6, we evaluate key design choices of DIP approach.

**In-context vs Direct dense prediction (Tab. 6a).** We compare our in-context dense prediction objective with direct dense prediction, where the model predicts pseudo semantic segmentation maps using a classification head. Our in-context approach significantly outperforms the direct method on PascalVOC and reduces RMSE on NYUv2 depth prediction, while both perform similarly on ADE20K.

**Impact of “distractor” examples (Tab. 6b).** Removing the “distractor” support examples significantly reduces performance, as the absence of “distractors” simplifies the post-training task, limiting the model’s ability to learn discriminative dense representations.
<table><thead><tr><th>Objective</th><th>PascalVOC</th><th>ADE20K</th><th>NYUv2↓</th></tr></thead><tbody><tr><td>In-context</td><td><strong>81.0</strong></td><td>39.7</td><td><strong>.756</strong></td></tr><tr><td>Direct.</td><td>79.9</td><td><strong>39.9</strong></td><td>.776</td></tr></tbody></table>

(a) **In-context pretraining** is more effective.

<table><thead><tr><th>Segm. Labels</th><th>PascalVOC</th><th>ADE20K</th></tr></thead><tbody><tr><td>Pseudo-labels w/ DiffCut</td><td>81.0</td><td><strong>39.7</strong></td></tr><tr><td>Pseudo-labels w/o DiffCut</td><td>59.1</td><td>25.5</td></tr><tr><td>Ground truth labels</td><td><strong>81.9</strong></td><td><strong>39.8</strong></td></tr></tbody></table>

(d) **Pseudo-labels generation.**

<table><thead><tr><th>Distractor</th><th>PascalVOC</th><th>ADE20K</th></tr></thead><tbody><tr><td>✓</td><td><strong>81.0</strong></td><td><strong>39.7</strong></td></tr><tr><td>✗</td><td>78.5</td><td>38.4</td></tr></tbody></table>

(b) **Distractor examples** are important.

<table><thead><tr><th>#Pseudo-classes</th><th>PascalVOC</th><th>ADE20K</th></tr></thead><tbody><tr><td>300</td><td>80.8</td><td>39.5</td></tr><tr><td>500</td><td>80.9</td><td>39.7</td></tr><tr><td>1000</td><td><strong>81.0</strong></td><td><strong>39.7</strong></td></tr><tr><td>2000</td><td>80.8</td><td>39.8</td></tr></tbody></table>

(e) **Number of pseudo-classes.**

<table><thead><tr><th>Positive example</th><th>PascalVOC</th><th>ADE20K</th></tr></thead><tbody><tr><td>Nearest neighbor</td><td><strong>81.0</strong></td><td><strong>39.7</strong></td></tr><tr><td>Two random crops</td><td>79.5</td><td>39.4</td></tr></tbody></table>

(c) **Constructing positive examples.**

<table><thead><tr><th>Dataset</th><th>PascalVOC</th><th>ADE20K</th></tr></thead><tbody><tr><td>COCO</td><td><strong>81.0</strong></td><td><strong>39.7</strong></td></tr><tr><td>ImageNet</td><td>80.8</td><td>39.6</td></tr></tbody></table>

(f) **Post-training dataset.**

<caption>Table 6. **Ablation Study of DIP.** Default settings in light blue. “Two random crops” in (c) refers to the case where the query and positive image are constructed as two random crops of the same original image.</caption>

**Construction of positive examples (Tab. 6c).** We compare two strategies for creating positive examples: (1) retrieving nearest neighbors using DINOv2R image-wise features and (2) using two random crops from the same image. The nearest neighbor strategy outperforms random crops, validating our design choice and showing that our method avoids reliance on complex augmentations common in self-distillation and contrastive approaches.

**Impact of DiffCut (Tab. 6d).** We assess the role of DiffCut [15], which generates class-agnostic segments for pseudo segmentation maps before K-means clustering. Compared to direct K-means on DINOv2R features, Diff-Cut significantly improves performance, demonstrating its effectiveness as a training-free, unsupervised method leveraging Stable Diffusion. Furthermore, our pseudo labels achieve results nearly matching ground-truth semantic segmentation, underscoring their quality.

**Impact of number of pseudo classes (Tab. 6e).** We study the effect of pseudo-class count (K-means clusters) for pseudo-label generation. Our approach is robust to this hyperparameter, with stable performance across values.

**Post-training data: scene-centric vs object-centric (Tab. 6f).** We compare DIP post-trained on scene-centric COCO vs. object-centric ImageNet. Results are comparable, COCO slightly outperforms ImageNet. This indicates that our method does not depend on human-curated, object-centric data and performs better on scene-centric data, easier to collect at scale.

### 4.4. Qualitative results

In Fig. 5, we present correlation maps between a query image patch and a reference image, comparing dense representations from DINOv2R and our DIP. While DINOv2R produces localized, part-based correlations, DIP captures semantic-level correspondences, more accurately delineating entire objects of the same semantic type as the query.

This shows that DIP provides better semantic correspondences. As visualized in Fig. 1, this enables DIP to retrieve more semantically coherent nearest neighbors, particularly

<figure><img src="image_6.png" /><figcaption>Figure 5. Correlation maps between a query image patch (highlighted with a red cross) and a reference image, comparing DINOv2R and DIP. DIP generates coherent, object-level correlations, while DINOv2R produces localized, part-level responses.</figcaption></figure>

in low-shot regimes (as in Fig. 1), resulting in more accurate segmentation outputs compared to DINOv2R.

### 5. Conclusion

We introduced DIP, a novel unsupervised post-training method that enhances dense image representations for in-context scene understanding. Leveraging meta-learning principles and automatically generating in-context tasks with pseudo-segmentations using Stable Diffusion, our approach avoids complex self-distillation architectures. DIP is computationally efficient (<9h on an A100) and generalizes well across downstream dense retrieval tasks, including semantic segmentation and depth prediction. It outperforms both the initial pretrained vision encoder and prior state-of-the-art post-training methods, providing a simpler and more effective solution for improving dense representations in vision models.
**Acknowledgements** This work was performed using HPC resources from GENCI-IDRIS (Grants 2025-AD011015037R1, 2024-AD011012884R3, and 2025-AD011016523). We thank PEPR Sharp (ANR-23-PEIA-0008, ANR, FRANCE 2030).

## References

[1] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In *ICLR*, 2020. 2

[2] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In *ECCV*, 2022. 2

[3] Ivana Balažević, David Steiner, Nikhil Parthasarathy, Relja Arandjelović, and Olivier J. Hénaff. Towards in-context scene understanding. In *NeurIPS*, 2023. 1, 2, 3, 4, 6, 13

[4] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In *ICLR*, 2022. 2

[5] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A. Efros. Visual prompting via image inpainting. In *NeurIPS*, 2022. 1

[6] Adrien Bardes, Jean Ponce, and Yann LeCun. Video: Variance-invariance-covariance regularization for self-supervised learning. In *ICLR*, 2022. 2

[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In *NeurIPS*, 2020. 1

[8] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In *NeurIPS*, 2020. 2

[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In *ICCV*, 2021. 2, 5, 6

[10] Mathilde Caron, Neil Houlsby, and Cordelia Schmid. Location-aware self-supervised transformers for semantic segmentation. In *WACV*, 2024. 3

[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In *ICML*, 2020. 2

[12] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In *CVPR*, 2021. 2

[13] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In *CVPR*, 2021. 3

[14] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In *CVPR*, 2016. 6

[15] Paul Couairon, Mustafa Shukor, Jean-Emmanuel Haugeard, Matthieu Cord, and Nicolas Thome. Diffcut: Catalyzing

zero-shot semantic segmentation with diffusion features and recursive normalized cut. In *NeurIPS*, 2024. 2, 3, 5, 8

[16] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In *ICLR*, 2024. 2, 5, 6, 7

[17] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu, and Baining Guo. Peco: Perceptual codebook for bert pre-training of vision transformers. In *AAAI*, 2023. 2

[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In *ICLR*, 2021. 3

[19] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. *IJCV*, 2010. 6

[20] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In *ICML*, 2017. 3

[21] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In *CVPR*, 2018. 3

[22] Spyros Gidaris and Nikos Komodakis. Generating classification weights with gnn denoising autoencoders for few-shot learning. In *CVPR*, 2019. 3

[23] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In *CVPR*, 2020. 2

[24] Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Obow: Online bag-of-visual-words generation for self-supervised learning. In *CVPR*, 2021. 2

[25] Spyros Gidaris, Andrei Bursuc, Oriane Simeoni, Antonín Vobecký, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Moca: Self-supervised representation learning by predicting masked online codebook assignments. *TMLR*, 2024. 2

[26] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In *NeurIPS*, 2020. 2

[27] Yatharth Gupta, Vishnu V Jaddipal, Harish Prabhala, Sayak Paul, and Patrick Von Platen. Progressive knowledge distillation of stable diffusion x1 using layer level loss. *arXiv preprint arXiv:2401.02677*, 2024. 2, 5, 7

[28] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In *ICLR*, 2022. 3

[29] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In *ICCV*, 2017. 2

[30] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In *CVPR*, 2020. 2, 3

[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In *CVPR*, 2022. 2, 7
[32] Greg Heinrich, Mike Ranzinger, Yin Hongxu, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavlo Molchanov. RADIOv2. 5: Improved baselines for agglomerative vision foundation models. In CVPR, 2025. 7

[33] Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In ICCV, 2021. 3

[34] Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. In ECCV, 2022. 3

[35] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). *arXiv preprint arXiv:1606.08415*, 2016. 12

[36] Zejiang Hou, Fei Sun, Yen-Kuang Chen, Yuan Xie, and Sun-Yuan Kung. Milan: Masked image pretraining on language assisted representation. *arXiv preprint arXiv:2208.06049*, 2022. 2

[37] Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chieh Chen. Segsort: Segmentation by discriminative sorting of segments. In *ICCV*, 2019. 3

[38] Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In *ICCV*, 2019. 3

[39] Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked image modeling. In *ECCV*, 2022. 2

[40] Asako Kanezaki. Unsupervised image segmentation by backpropagation. In *ICASSP*, 2018. 3

[41] Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Dino-foresight: Looking into the future with dino. *arXiv preprint arXiv:2412.11673*, 2024. 3

[42] Tommie Kerssies, Daan de Geus, and Gijs Dubbelman. How to benchmark vision foundation models for semantic segmentation? In CVPRW, 2024. 7

[43] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In *ICCV*, 2023. 7

[44] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012. 5

[45] Mengcheng Lan, Xinjiang Wang, Yiping Ke, Jiaxing Xu, Litong Feng, and Wayne Zhang. Smooseg: smoothness prior for unsupervised semantic segmentation. NeurIPS, 2024. 3

[46] Tim Lebailly, Thomas Stegmüller, Behzad Bozorgtabar, Jean-Philippe Thiran, and Tinne Tuytelaars. Cribo: Self-supervised learning via cross-image object-level bootstrapping. In ICLR, 2024. 2, 3, 5, 6

[47] Kehan Li, Zhennan Wang, Zesen Cheng, Runyi Yu, Yian Zhao, Guoli Song, Chang Liu, Li Yuan, and Jie Chen. Acseg: Adaptive conceptualization for unsupervised semantic segmentation. In CVPR, 2023. 3

[48] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context. In ECCV, 2014. 5, 6

[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 12

[50] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR, 2018. 3

[51] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014. 6

[52] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 6

[53] Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. In NeurIPS, 2020. 3

[54] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. *arXiv preprint arXiv:1807.03748*, 2018. 2, 3

[55] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. In TMLR, 2023. 2

[56] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024. 1, 2, 5

[57] Valentinos Pariza, Mohammadreza Salehi, and Yuki Asano. Hummingbird evaluation for vision encoders, 2024. 4, 13

[58] Valentinos Pariza, Mohammadreza Salehi, Gertjan J Burghouts, Francesco Locatello, and Yuki M Asano. Near, far: Patch-ordering enhances vision foundation models' scene understanding. In ICLR, 2025. 1, 2, 3, 4, 5, 6, 7

[59] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. *arXiv preprint arXiv:2208.06366*, 2022. 2

[60] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In CVPR, 2018. 3

[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 2, 7

[62] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017. 3

[63] Christos Sakaridis, Haoran Wang, Ke Li, René Zurbrügg, Arpit Jadon, Wim Abbeloos, Daniel Olmeda Reino, Luc Van Gool, and Dengxin Dai. ACDC: The adverse conditions dataset with correspondences for robust semantic driving scene perception. In ICCV, 2021. 6
[64] Mohammadreza Salehi, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. Time does tell: Self-supervised time-tuning of dense image representations. In *ICCV*, 2023. 1, 2, 5, 6

[65] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In *ICML*, 2016. 3

[66] Hyun Seok Seong, WonJun Moon, SuBeen Lee, and Jae-Pil Heo. Leveraging hidden positives for unsupervised semantic segmentation. In *CVPR*, 2023. 3

[67] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In *ECCV*. Springer, 2012. 7

[68] Walter Simoncini, Andrei Bursuc, Spyridon Gidaris, and Yuki Asano. No train, all gain: Self-supervised gradients improve deep frozen representations. In *NeurIPS*, 2024. 3

[69] Sophia Sirko-Galouchenko, Alexandre Boulch, Spyros Gidaris, Andrei Bursuc, Antonin Vobecky, Patrick Pérez, and Renaud Marlet. Occfeat: Self-supervised occupancy feature prediction for pretraining bev segmentation networks. In *CVPRW*, 2024. 3

[70] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In *NeurIPS*, 2017. 3

[71] Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars, and Jean-Philippe Thiran. Croc: Cross-view online clustering for dense visual representation learning. In *CVPR*, 2023. 3, 5, 6

[72] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In *CVPR*, 2018. 3

[73] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-Franco. Diffuse attend and segment: Un-supervised zero-shot segmentation using stable diffusion. In *CVPR*, 2024. 3

[74] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In *NeurIPS*, 2017. 4

[75] Shashanka Venkataramanan, Valentinos Pariza, Mohammadreza Salehi, Lukas Knobel, Spyros Gidaris, Elias Ramzi, Andrei Bursuc, and Yuki M Asano. Franca: Nested matrixoshka clustering for scalable visual representation learning. *arXiv preprint arXiv:2507.14137*, 2025. 1, 2

[76] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In *NeurIPS*, 2016. 2, 3

[77] Antonin Vobecky, David Hurych, Oriane Siméoni, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, and Josef Sivic. Un-supervised semantic segmentation of urban scenes via cross-modal distillation. *IJCV*, 2025. 3

[78] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In *CVPR*, 2021. 3

[79] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In *CVPR*, 2021. 3

[80] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In *CVPR*, 2022. 2

[81] Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In *CVPR*, 2022. 6

[82] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In *CVPR*, 2017. 6

[83] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In *ICLR*, 2022. 2, 5

[84] Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In *CVPR*, 2022. 1, 2, 3, 5, 6
# DIP: Unsupervised Dense In-Context Post-training of Visual Representations

## Supplementary Material

### A. Implementation details

During post-training, we fine-tune only the last three transformer blocks of the pretrained ViT encoder $f(\cdot)$ while keeping the remaining layers frozen. Our MLP projector $h(\cdot)$ consists of two linear layers with a non-linear activation function GELU [35]. Hidden feature dimension is set to 7D and fixed output dimension 6144. $\ell_2$-normalization is applied at the output of the MLP. We set the temperature parameter $\tau$ of the softmax operator to 0.07 (see Tab. 7). Our ablation study demonstrates that the method is robust to the choice of temperature, with values ranging from 0.03 to 0.09 yielding similar performance on the PascalVOC dataset. During post-training our support set consists of 1 positive and 7 “distractor” examples (8 total examples). During training, we use the AdamW [49] optimizer with a learning rate of $2.25 \times 10^{-7}$ and a weight decay of 0.05. We train for 5 epochs. We employ cosine learning rate schedule.

<table><caption>Table 7. Ablation of temperature $\tau$ during post-training.</caption><tr><td>Temperature</td><td>0.09</td><td>0.07</td><td>0.03</td></tr><tr><td>PascalVOC</td><td>80.8</td><td>81.0</td><td>81.0</td></tr></table>

**Generation time of pseudo semantic segmentation labels.** COCO pseudo-label generation required 17 hours on 1 V100 GPU, with DiffCut inference as the bottleneck due to its current lack of batch processing optimization and suboptimal GPU utilization. However, this one-time offline process can be used to post-train multiple encoders.

### B. Additional quantitative results

**In-context scene understanding: impact of memory size.** In Fig. 6, we analyze the effect of memory size on in-context semantic segmentation using the ADE20K dataset (full) for DINOv2R, NeCo, and our DIP. Results show that DIP outperforms both DINOv2R and NeCo across all memory sizes.

**Linear segmentation.** Tab. 8 presents linear segmentation results on COCO and ADE20K benchmarks. For fair comparison, we re-evaluated both DINOv2R and NeCo using our implementation, ensuring consistent evaluation protocols across all methods. Our approach consistently improves over the strong DINOv2R baseline and shows improvements over NeCo. Notably, with the ViT-B/14 backbone on COCO, our method achieves 86.7 mIoU, surpassing DINOv2R by 2.0 points.

<table><caption>Table 8. Linear segmentation results on COCO and ADE20K datasets. All methods (DINOv2R, NeCo, and DIP) are evaluated using our implementation. DIP consistently improves over our base model DINOv2R and outperforms NeCo across both datasets.</caption><thead><tr><th>Method</th><th>Backbone</th><th>COCO</th><th>ADE20K</th></tr></thead><tbody><tr><td>DINOv2R</td><td>ViT-S/14</td><td>82.1</td><td>33.5</td></tr><tr><td>NeCo</td><td>ViT-S/14</td><td>81.1 (-1.0)</td><td>33.1 (-0.4)</td></tr><tr><td>DIP (ours)</td><td>ViT-S/14</td><td><strong>82.6 (+0.5)</strong></td><td><strong>33.7 (+0.2)</strong></td></tr><tr><td>DINOv2R</td><td>ViT-B/14</td><td>85.5</td><td>38.6</td></tr><tr><td>NeCo</td><td>ViT-B/14</td><td>85.2 (-0.3)</td><td><strong>39.5 (+0.9)</strong></td></tr><tr><td>DIP (ours)</td><td>ViT-B/14</td><td><strong>86.7 (+2.0)</strong></td><td><strong>39.5 (+0.9)</strong></td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Figure 6. In-context scene understanding: impact of memory size. Semantic segmentation performance on ADE20K (full dataset) using dense nearest neighbor retrieval, evaluated across varying memory sizes.</figcaption></figure>

<table><caption>Table 9. Larger backbones evaluation We show performance of DIP and DINOv2R on a larger backbone ViT-L. Dense nearest neighbor retrieval performance (mIoU) for semantic segmentation on PascalVOC across varying proportions of training data.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Backbone</th><th colspan="4">PascalVOC</th></tr><tr><th>1</th><th><sup>1</sup>&frasl;<sub>8</sub></th><th><sup>1</sup>&frasl;<sub>64</sub></th><th><sup>1</sup>&frasl;<sub>128</sub></th></tr></thead><tbody><tr><td>DINOv2R</td><td>ViT-B</td><td>79.0</td><td>75.3</td><td>67.6</td><td>60.3</td></tr><tr><td>DIP (ours)</td><td>ViT-B</td><td>82.1</td><td><strong>79.6</strong></td><td><strong>75.1</strong></td><td><strong>70.1</strong></td></tr><tr><td>DINOv2R</td><td>ViT-L</td><td>76.9</td><td>72.8</td><td>61.4</td><td>54.4</td></tr><tr><td>DIP (ours)</td><td>ViT-L</td><td><strong>81.1</strong></td><td><strong>78.7</strong></td><td><strong>70.0</strong></td><td><strong>64.6</strong></td></tr></tbody></table>
**Larger backbones.** While ViT-L (DINOv2) underper-
forms ViT-B in in-context segmentation [3, 57], our method
still improves results with ViT-L, as shown in Tab. 9. This
demonstrates DIP's scalability across backbone sizes.

<table><caption>Table 10. Additional ablation on the construction of positive examples. Dense nearest neighbor retrieval performance (mIoU) for semantic segmentation on PascalVOC and ADE20K across varying proportions of training data.</caption>
  <thead>
    <tr>
      <th></th>
      <th colspan="4">PascalVOC</th>
      <th colspan="4">ADE20K</th>
    </tr>
    <tr>
      <th></th>
      <th>1</th>
      <th><sup>1</sup>&frasl;<sub>8</sub></th>
      <th><sup>1</sup>&frasl;<sub>64</sub></th>
      <th><sup>1</sup>&frasl;<sub>128</sub></th>
      <th>1</th>
      <th><sup>1</sup>&frasl;<sub>8</sub></th>
      <th><sup>1</sup>&frasl;<sub>64</sub></th>
      <th><sup>1</sup>&frasl;<sub>128</sub></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>Two Crops</th>
      <td>79.5</td>
      <td>75.1</td>
      <td>67.7</td>
      <td>61.2</td>
      <td>39.4</td>
      <td>33.2</td>
      <td>24.7</td>
      <td>22.4</td>
    </tr>
    <tr>
      <th>NN</th>
      <td><strong>81.0</strong></td>
      <td><strong>77.7</strong></td>
      <td><strong>71.4</strong></td>
      <td><strong>65.9</strong></td>
      <td><strong>39.7</strong></td>
      <td><strong>33.7</strong></td>
      <td><strong>25.6</strong></td>
      <td><strong>23.2</strong></td>
    </tr>
  </tbody>
</table>

**Nearest Neighbors (NN) vs. Two Crops** We compare two strategies for creating positive examples: (1) retrieving nearest neighbors using DINOv2R image-wise features (NN) and (2) using two random crops from the same image. NN consistently outperforms Two Crops, with the performance gap widening when fewer training examples are available (see Tab. 10). This scalability advantage justifies our design choice.

C. Additional qualitative results

We present additional examples of automatically con-
structed in-context tasks in Fig. 7, showing the quality of
our pseudo-labeling approach. We display query images
paired with their corresponding positive support examples,
along with both pseudo-labels and ground truth labels, in-
cluded only for comparison.
<figure><img src="image_8.png" /><figcaption>Figure 7. **Examples of automatically constructed in-context scene understanding tasks.** Each row shows a query image and its corresponding positive support example. (a) and (b) display the query image and its pseudo segmentation labels, while (d) and (e) show the positive support image and its pseudo segmentation labels. (c) and (f) present the ground truth segmentation labels for the query and positive images, respectively, included only for comparison with the pseudo labels.</figcaption></figure>