Near, far: Patch-ordering enhances vision foundation models' scene understanding

## Abstract

Abstract We propose sorting patch representations across views as a novel self-supervised learning signal to improve pretrained representations.
To this end, we introduce NeCo : Patch Ne ighbor Co nsistency, a novel training loss that enforces patch-level nearest neighbor consistency across a student and teacher model, relative to reference batches. Our method leverages a differentiable sorting method applied on top of pretrained representations, such as DINOv2-registers to bootstrap the learning signal and further improve upon them. This dense post-pretraining leads to superior performance across various models and datasets, despite requiring only 19 hours on a single GPU. We demonstrate that this method generates high-quality dense feature encoders and establish several new state-of-the-art results: +5.5 % and + 6% for non-parametric in-context semantic segmentation on ADE20k and Pascal VOC, and +7.2% and +5.7% for linear segmentation evaluations on COCO-Things and -Stuff.

## 1 Introduction

Dense self-supervised learning trains feature extractors to produce representations for every pixel or patch of an image without supervision. This field has seen substantial advancements in recent years, notably improving unsupervised semantic segmentation [, , , , ], object-centric representation learning [], and other dense downstream tasks like object tracking and object detection [, , , ].

One particularly interesting use-case of densely pretrained encoders was developed by Balazevic et. al. []. They propose to solve semantic segmentation by posing it as a nearest-neighbor retrieval problem utilizing the features of the spatial patches.
This non-parametric method not only mirrors in-context learning in large language models (LLMs) [] but also delivers rapid and robust performance, especially with limited data.

Inspired by this idea, we turn this evaluation approach on its head and propose to use nearest-neighbor retrieval as a training mechanism for encoders.
By integrating nearest-neighbor relationships into the training phase, we aim to train models that can naturally cluster and distinguish intricate visual similarities. This method promises to yield models with deeply semantic spatial features specifically tailored for in-context tasks, enhancing their adaptability and robustness. This approach, while promising, presents two main challenges.

The first is the source of supervision.
In the case of evaluation, ground-truth labels are used, yet we are interested in obtaining better self-supervised representations.
While previous works [, ] address this by essentially converting dense learning to image-level learning via learnable pooling of patches, we offer a more practical and versatile solution.
We simply start from already image-level pretrained models and adapt them further.
We term this stage dense post-pretraining and demonstrate that it is an effective and fast solution to this problem, taking only 19 hours on a single GPU for tuning for a ViT-S/16 model.

The second challenge is the discrete nature of nearest-neighbor retrieval, which does not yield gradients.
To overcome this, we apply a differentiable sorting method [], originally proposed for ranking supervision, that we can use to backpropagate gradients.
As we demonstrate empirically, this results in a more efficient and effective algorithm.

Our method enforces Patch Neighbor Consistency, so we term it NeCo.
We show that it can be applied on top of image-level pretrained models such as DINO [], and densely trained ones like iBOT [], Leopart [], CrIBO [] and DINOv2 [, ] to obtain superior features for in-context scene understanding. Despite not being as close to our NeCo  training task, our method also consistently improves on downstream benchmarks such as unsupervised semantic and linear segmentation.

Overall, our contributions can be summarized as follows:

- •
We propose a new post-pretraining adaptation that applies a dense, patch-sorting-based self-supervised objective, NeCo, applicable to any pretrained Vision Transformer
- •
We demonstrate NeCo’s utility by applying it to six different backbones and evaluating it on four datasets and four evaluation protocols, achieving performance gains from 6% to 16%.
- •
We set several new state-of-the-art performances, for example on the in-context segmentation benchmark of [], we outperform the previous methods such as CrIbo and DINOv2 on Pascal VOC and ADE20k by 4% to 13% across different metrics.

## 2 Related Works

#### Dense Self-supervised Learning.

Dense self-supervised learning methods aim to generate categorizable representations at the pixel or patch level, rather than the image level. This field has gained significant attention [, , , , , , , , , , , ] due to the observation that image-level self-supervised methods [, , , , , , , , , , , , , , , , ] do not necessarily produce expressive dense representations [, , , ].

CroC [] is a recent method that has proposed a dense self-supervised loss to address the issue. It applies joint clustering between different views, ensuring that cluster centers capturing the same object are similar. Leopart [] improves dense representations by applying a dense clustering loss to pretrained models. Extending this concept, Timetuning [] has demonstrated that finetuning pretrained backbones over the temporal dimension of unlabeled videos enhances dense reasoning capabilities. Recently, Hummingbird [] has proposed a dense loss that leverages attention within and across images during training, showing strong in-context scene understanding during evaluation. CrIBo [] takes this further by explicitly enforcing cross-image nearest neighbor consistency between image objects, achieving state-of-the-art results.

We similarly adopt nearest neighbor consistency due to its promising results in in-context scene understanding but with two major differences: (1) instead of using pooled versions of patches at either the image or object level, we directly apply it to patch features; (2) in addition to nearest neighbor consistency, we ensure that the order of neighbors for the same patches from different views is similar. These changes result in more semantic patch-level features, directly enhancing in-context scene understanding and stabilizing training, as there is no need to infer object-level features through clustering methods, which can be unstable during training.

#### Unsupervised Object Segmentation.

Several works specifically target unsupervised object segmentation [, , , , , , , , ]. The goal of these works is not to learn semantic patch-level representations; instead, they often utilize the existing information in frozen pretrained backbones and train another model to explicitly solve semantic segmentation. For instance, Seitzer et al. [] train a slot-attention encoder and decoder module [] to reconstruct DINO [] pretrained features with a few slots for each input image. This process enables the creation of per-image object cluster maps, where each slot represents a distinct object or part of an object within the image, based on the features it predicts.

In contrast, our approach learns distinct features for various objects and employs dense representations as intermediaries for dense tasks like semantic segmentation. These features can then be used to develop per-dataset cluster maps for semantic segmentation use cases.

## 3 Patch Neighbor Consistency

Figure: Figure 1: NeCo overview. Given an input image $I$, two augmentations $\tau_{1}$ and $\tau_{2}$ are applied to create two different views, which are processed by the teacher and student encoders, $\phi_{t}$ and $\phi_{s}$ respectively. The teacher encoder is updated using Exponential Moving Average (EMA). The encoded features are then aligned using ROI Align and compared with reference features $F_{r}$ obtained by applying $\phi_{s}$ to other batch images. Next, pairwise distances $D_{ij}$ between $F_{s}$ and $F_{r}$, as well as between $F_{t}$ and $F_{r}$, are computed using cosine similarity. These distances are then sorted using differentiable sorting and utilized to force nearest order consistency across the views through the NeCo loss.
Refer to caption: x1.png

The goal is to develop a feature space in which, for a given input, patches representing the same object exhibit similar features, whereas patches representing different objects show distinct features. A key challenge for a self-supervised method in this process is defining the similarity between image patches. Although patches from the same object (e.g. a cat) are expected to be more similar to each other than to those from different objects (e.g. a dog), they may still depict different parts of the object (e.g. a cat’s tail and legs).
Both part- and whole-object representations have important applications. Therefore, to generate a feature space that can reliably represent both, merely establishing a single clustering per object is not enough.
Instead, this requires accurately ordering similarities, such that patches representing the same object and its parts are more similar to one another and less to those of other objects. To this end, our method works by extracting dense features of the inputs, finding their pair-wise distances, and forcing a consistency between the order of nearest neighbors within a batch across two views. Figure 1 provides an overview of the method, which we describe in detail below.

#### Feature Extraction and Alignment.

Given an input image $I$, two augmentations specified by parameters $\tau_{1}$ and $\tau_{2}$ are applied to create two different views $V_{1}$ and $V_{2}$. These views are then divided into $N=\lfloor\frac{H}{P}\rfloor\times\lfloor\frac{W}{P}\rfloor$ separate patches, where $H$ and $W$ represent the height and width of the input image, and $P$ represents the patch size. The patches are represented as $V_{1}^{p}=\bigl{[}v^{1}_{1},\ldots,v^{1}_{N}\bigr{]}$ and $V_{2}^{p}=\bigl{[}v^{2}_{1},\ldots,v^{2}_{N}\bigr{]}$, which are fed to the feature extractor.

We utilize the Vision Transformer (ViT) architecture [] as the backbone and employ a teacher-student framework, where the student and teacher models are denoted by $\phi_{s}$ and $\phi_{t}$, respectively. The teacher’s weights are updated using the exponential moving average of the student’s weights.

As the generated views cover different parts of the input, the extracted features do not necessarily correspond to the same objects. To address this, we align the features by applying ROI-Align [], adjusted according to the crop augmentation parameters. This process creates spatially aligned dense features for the teacher and student networks, represented by $F_{s}\in\mathbb{R}^{N^{\prime}\times d}$ and $F_{t}\in\mathbb{R}^{N^{\prime}\times d}$. These features are then forced to maintain a consistent order of nearest neighbors, ensuring more robust and meaningful feature representations, as explained next.

#### Pairwise Distance Computation.

To identify the nearest neighbors of the patches, it is necessary to extract features from other images in the batch and compute their distances with respect to $F_{s}$ and $F_{t}$. To achieve this, all batch images are fed through the student network, $\phi_{s}$, to obtain the batch features $F_{B}\in\mathbb{R}^{BN\times d}$. We sample a random fraction $f\ll 1$ of these patches to obtain the $R=fBN$ reference patches $F_{r}\in\mathbb{R}^{r\times d}$ which we use to compare the nearest neighbors of our $F_{s}$ and $F_{t}$ features.
To this end, we compute distances based on cosine similarities,

$$ $\displaystyle D_{s}(i,j)=1-\frac{\langle F_{s}^{i},F_{r}^{j}\rangle}{\|F_{s}^{ i}\|\|F_{r}^{j}\|},$ (1) $\displaystyle D_{t}(i,j)=1-\frac{\langle F_{t}^{i},F_{r}^{j}\rangle}{\|F_{t}^{ i}\|\|F_{r}^{j}\|},$ (2) $\displaystyle i\in(1,\ldots,N^{\prime}),\,\,\,j\in(1,\ldots,R),$ (3) $$

Next, these distance matrices are sorted in a differentiable manner to produce a loss that enforces a similar sorting across the two views.

#### Differentiable Sorting of Distances.

To determine the order of nearest neighbors from distance matrices, sorting is necessary. However, traditional sorting algorithms cannot propagate gradients because they use non-differentiable operations such as $d^{\prime}_{i}\leftarrow\min(d_{a},d_{b})$ and $d^{\prime}_{a}\leftarrow\max(d_{a},d_{b})$ to facilitate element swapping in the sequence for an ordering $a<b$. Given a sequence $s=\bigl{(}d_{1},\ldots,d_{R}\bigr{)}$, where $R$ is the length of the sequence, We use relaxed, differentiable versions of these operations by defining their soft versions following recent work [], as follows:

$$ $\displaystyle d_{a}^{\prime}=\text{softmin}(d_{a},d_{b})$ $\displaystyle:=d_{a}f(d_{b}-d_{a})+d_{b}f(d_{a}-d_{b}),$ (4) $\displaystyle d_{b}^{\prime}=\text{softmax}(d_{a},d_{b})$ $\displaystyle:=d_{a}f(d_{a}-d_{b})+d_{b}f(d_{b}-d_{a}),$ (5) $$

where the function $f(x)=\frac{1}{2}(\arctan(\beta x)+0.5)$, and $\beta>0$ is an inverse temperature parameter, specifying the steepness of the operator. This function is sigmoid-shaped and centered around $x=0$. As $\beta$ approaches infinity, the relaxation converges to the discrete swap operation.
This operation can be defined in an approximate permutation matrix $P_{\text{swap}}(d_{i},d_{j})\in\mathbb{R}^{L\times L}$, which is essentially an identity matrix except for the entries $P_{ii}$, $P_{ij}$, $P_{ji}$, and $P_{jj}$ defined as

$$ $\displaystyle P_{ii}=P_{jj}=f(d_{j}-d_{i}),\qquad P_{ij}=P_{ji}=f(d_{i}-d_{j}),$ (7) $$

such that one step of swapping the pair $(d_{i},d_{j})$ in the sequence is equivalent to multiplying $P$ with that sequence. The final permutation matrix for the entire sequence is determined by the sorting algorithm employed. For example, in the odd-even sorting algorithm, the permutation matrix $P_{t}$ for a step $t$ is defined as:

$$ $\displaystyle P_{t}$ $\displaystyle=\prod_{i\in M}P_{\text{swap}}(d_{i},d_{i+1}),$ (8) $$

where $M$ is the set of odd indices if $t$ is odd and the set of even indices if $t$ is even. The overall permutation matrix $Q$ is obtained by multiplying the permutation matrices from all steps of the sorting algorithm,
$Q=\prod_{t=1}^{L}P_{t}.$
As shown by [], $L=R$ of such steps are sufficient for efficient sorting. In the discrete case, for each column $i$, the permutation matrix has exactly one entry of 1, indicating the sequence element that should be placed in the $i$-th column. In the relaxed version, column values represent a distribution over possible sequence elements. In our case, a row $i$ of the distance matrix $D_{s}$ shows the distance of the $i$-th student feature to all the reference features.

With its sorting matrix $Q_{i}$, the $(r,k)$ element of this matrix can be viewed as the probability of a reference feature $r$ being the $k$-th nearest neighbor for the $i$-th feature.
Hence, to maintain the order of nearest neighbors for every ROI-aligned patch feature, we compute $Q_{i}$ for every row of $D_{s}$ and $D_{t}$ and force these to be similar.
This results in final matrices $Q^{s}=[Q^{s}_{1},\ldots,Q^{s}_{N^{\prime}}]$ and $Q^{t}=[Q^{t}_{1},\ldots,Q^{t}_{N^{\prime}}]$, which are used in the training loss.

#### Training Loss.

After computing permutation matrices, we enforce similarity on the order of nearest neighbors for each of the aligned patch features using the cross-entropy loss. The loss for the permutation matrix of patch $i$ is defined as:

$$ $\mathcal{L}_{\text{CE}}(Q^{t}_{i},Q^{s}_{i})=-\sum_{j,k}{Q^{t}_{i}}{(j,k)}\log ({Q^{s}_{i}}{(j,k)}),$ $$

To ensure robust nearest neighbor consistency, we compute the cross-entropy loss in both directions and sum the losses.
The final training loss, incorporating both directions, for all the patches is:

$$ $\mathcal{L}_{\texttt{NeCo}}=\sum_{i=1}^{N^{\prime}}\mathcal{L}_{\text{CE}}(Q^{ t}_{i},Q^{s}_{i})+\mathcal{L}_{\text{CE}}(Q^{s}_{i},Q^{t}_{i})$ $$

This ensures, in a differentiable manner, that the order of nearest neighbors is consistent between the student and teacher features in both directions.

## 4 Experiments

### 4.1 Setup

#### Benchmarked Methods.

We compare our method against state-of-the-art dense self-supervised learning methods, including CrIBo [], Hummingbird [], TimeT [], and Leopart []. To provide a more comprehensive evaluation, We also include the performance of DINOv2 enhanced with registers, referred to as DINOv2R [, ], as it has demonstrated strong dense capabilities. Additionally, we benchmark our method against leading unsupervised semantic segmentation approaches such as COMUS [].

#### Training.

We run our experiments on ViT-Small and ViT-Base with a patch size of 14.
We start from various pretrained backbones, and use DINOv2 with registers’ unless otherwise noted. We post-pretrain these models for 25 COCO epochs on a single NVIDIA RTX A6000-46GB GPU, taking around 19 hours. For other training details, we refer readers to the Appendix A.

#### Evaluation.

In all our evaluations, we discard the projection head, following previous works [, , ], and directly use the spatial tokens from the Vision Transformer backbone. Scores in all experiments are reported as mean intersection over union (mIoU). We conduct four types of evaluations: linear segmentation fine-tuning with a $1\times 1$ convolution, end-to-end segmentation with the Segmenter head [], clustering and overclustering semantic segmentation [, ], and dense nearest neighbor retrieval [].
For clustering and overclustering, we apply K-Means to spatial tokens, setting $K$ to the number of ground truth objects and to high values like 300 and 500, as previously used [, ]. We then extract object cluster maps and match them using Hungarian matching []. For dense nearest neighbor retrieval, we follow the protocol from [], implemented in [].

#### Datasets.

We train our model on ImageNet-100 [], Pascal VOC12 [], and COCO [] for ablations and use COCO as our primary training dataset for all state-of-the-art comparisons. For evaluations, we use the validation sets of Pascal VOC12 [], COCO [], ADE20k [], and Pascal Context []. For finetuning and feature transferability evaluations on COCO [], we train using a 10% split of the training set, while we use the full training splits of the other datasets.

### 4.2 Comparison to State-of-the-Art

Figure: Figure 2: In-context scene understanding benchmark. Dense nearest neighbor retrieval performance is reported across various training data proportions on two scene-centric datasets, Pascal VOC and ADE20k. The retrieved cluster maps are compared with the ground truth using Hungarian matching [], and their mIoU score is reported. For all models, ViT-S16 is used except for DINOv2R and NeCo, where it is ViT-S14. For full tables, refer to Appendix B.3
Refer to caption: x2.png

In this section, we first compare the quality of frozen features learned through NeCo  with state-of-the-art methods in in-context learning via nearest neighbor retrieval and unsupervised semantic segmentation tasks. Next, we demonstrate that NeCo’s versatility by applying it to five different pretraining models and show it improves their dense features consistently. Finally, we evaluate the transferability of our learned dense representations to other datasets by using linear head semantic segmentation and end-to-end fine-tuning with Segmentor [].

#### Visual In-Context Learning Evaluation.

We compare our approach to a recently proposed benchmark [] that evaluates in-context reasoning in vision models. Unlike traditional linear segmentation methods, this evaluation does not require fine-tuning or end-to-end training. Instead, it creates validation segmentation maps by matching patch-level, feature-wise nearest neighbor similarities between validation images (queries) and training samples (keys). This method, inspired by NLP strategies, tests how well models learn tasks from a few examples. The results are presented in Figure 2. As shown, NeCo outperforms prior state-of-the-art methods such as CrIBo and DINOv2R by 4% to 13% on Pascal and ADE20k across different fractions. The performance gap between NeCo and others increases, particularly in the data-efficiency regime.
This improvement is due to our method’s explicit enforcement of patch-level nearest neighbor consistency, resulting in higher-quality patch-level representations that remain effective even with fewer images. In contrast, other methods that promote image-level [] or object-level [] consistency force consistency between a pooled vector of patches, potentially leading to inadequate semantic patch-level representations, particularly in smaller datasets.
Our method’s ability to perform well with few images brings vision models one step closer to in-context learning style generalist reasoning. For complete tables and details on scaling the results to larger models, please refer to Appendix B.3. For visualizations please refer to Appendix C.

**Table 1: Frozen clustering-based evaluations. (a) We evaluate the models by running $K$-means with various clustering granularities $K$ on the spatial features on two datasets. The resulting cluster maps are matched to the ground-truth by Hungarian matching, and the intersection is reported in mIoU. (b) Following previous works [, ], we post-process the resulting maps and report unsupervised semantic segmentation on Pascal VOC. Both tables use ViT-S with the patch size of 16, except for DINOv2 and NeCo, where it is 14.**
|  | Pascal VOC | COCO-Things |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | K=GT | K=300 | K=500 | K=GT | K=300 | K=500 |
| DINO [] | 4.3 | 13.9 | 17.3 | 5.4 | 18.8 | 19.2 |
| iBOT [] | 4.4 | 23.8 | 31.1 | 7.6 | 26.6 | 28.0 |
| CrOC [] | 3.4 | 16.4 | 20.0 | 4.9 | 14.7 | 18.1 |
| TimeT [] | 12.2 | 43.6 | 46.2 | 17.5 | 42.7 | 44.6 |
| DINOv2R [] | 12.2 | 46.7 | 49.5 | 12.3 | 38.9 | 41.2 |
| CrIBo [] | 18.3 | 51.3 | 54.5 | 14.5 | 46.0 | 48.3 |
| NeCo | 17.8 | 69.4 | 72.6 | 18.2 | 61.2 | 64.5 |

**Table 2: Linear segmentation performance. A linear segmentation head is trained on top of the frozen spatial features obtained from different feature extractors. We report the mIoU scores achieved on the validation sets of 4 different datasets.**
| Method | Backbone | Params | COCO-Things | COCO-Stuff | Pascal VOC | ADE20K |
| --- | --- | --- | --- | --- | --- | --- |
| DINO | ViT-S/16 | 21M | 43.9 | 45.9 | 50.2 | 17.5 |
| TimeT | ViT-S/16 | 21M | 58.2 | 48.7 | 66.3 | 20.7 |
| iBOT | ViT-S/16 | 21M | 58.9 | 51.5 | 66.1 | 21.8 |
| CrOC | ViT-S/16 | 21M | 64.3 | 51.2 | 67.4 | 23.1 |
| CrlBo | ViT-S/16 | 21M | 64.3 | 49.1 | 71.6 | 22.7 |
| DINOv2R | ViT-S/14 | 21M | 75.3 | 56.0 | 74.2 | 35.0 |
| NeCo | ViT-S/14 | 21M | 82.3 | 62.0 | 81.3 | 40.1 |
| DINO | ViT-B/16 | 85M | 55.8 | 51.2 | 62.7 | 23.6 |
| MAE | ViT-B/16 | 85M | 38.0 | 38.6 | 32.9 | 5.8 |
| iBOT | ViT-B/16 | 85M | 69.4 | 55.9 | 73.1 | 30.1 |
| CrIBo | ViT-B/16 | 85M | 69.6 | 53.0 | 73.9 | 25.7 |
| DINOv2R | ViT-B/14 | 85M | 78.3 | 57.6 | 79.8 | 40.3 |
| NeCo | ViT-B/14 | 85M | 85.5 | 63.3 | 83.3 | 44.9 |

**Table 3: NeCo starting from different pretrainings. We report frozen clustering and linear segmentation on Pascal VOC and COCO-Things. NeCo can considerably boost (↑) the performance of models with different initialization, showing our approach’s generality. The backbone is ViT-S16.**
|  | Pascal VOC | COCO-Things |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | At Init | +NeCo | At Init | +NeCo |  |  |  |  |  |  |  |  |
| Pretrain | K=GT | K=500 | Lin. | K=GT | K=500 | Lin. | K=21 | K=500 | Lin. | K=21 | K=500 | Lin. |
| iBOT [] | 4.4 | 31.1 | 66.1 | 15.4^↑11.0 | 51.2^↑20.1 | 68.6^↑2.5 | 7.6 | 28.0 | 58.9 | 20.4^↑12.8 | 52.8^↑24.8 | 67.7^↑8.8 |
| DINO [] | 4.3 | 17.3 | 50.2 | 14.5^↑10.2 | 47.9^↑30.6 | 61.3^↑11.1 | 5.4 | 19.2 | 43.9 | 16.9^↑11.5 | 50.0^↑30.8 | 62.4^↑18.5 |
| TimeT [] | 12.2 | 46.2 | 66.3 | 17.9^↑5.7 | 52.1^↑5.9 | 68.5^↑2.2 | 18.4 | 44.6 | 58.2 | 20.6^↑2.2 | 54.3^↑9.7 | 64.8^↑6.6 |
| Leopart [] | 15.4 | 51.2 | 66.5 | 21.0^↑5.6 | 55.3^↑4.1 | 68.3^↑1.8 | 14.8 | 53.2 | 63.0 | 18.8^↑4.0 | 53.9^↑0.7 | 65.4^↑2.4 |
| CrIBo [] | 18.3 | 54.5 | 71.6 | 21.7^↑3.4 | 59.6^↑5.1 | 72.1^↑0.5 | 14.5 | 48.3 | 64.3 | 21.1^↑6.6 | 54.0^↑5.7 | 68.0^↑3.7 |

#### Frozen Clustering-based Evaluations.

Next, we evaluate the representation quality of our learned dense features across different objects in each dataset. Ideally, we expect that all patch features belonging to the same object, when clustered, will be assigned to the same cluster. If the learned representations are more fine-grained, such as learning object parts instead of whole objects (e.g., hands or faces instead of a person), they should consistently cover the same part across the entire dataset. To measure this, we extract dense features from all images and apply $K$-means clustering with various $K$ values to create cluster maps for each image. These cluster maps are then matched with the ground truth using Hungarian matching [], and their mIoU is reported. For the first scenario, $K$ matches the number of ground truth objects. Additionally, to account for the second scenario, we also report performance in overclustering setups.

The results in 1(a) show that NeCo passes state-of-the-art by CrIBo by 14.5% on average across various datasets and metrics. Note that this gain is not due to the DINOv2R initialization, as it performs 4% lower than CrIBo on average. In 1(b), we report clustering performance when $K$ matches the number of ground truth objects, with clustering applied solely to foreground patches extracted by methods used in [, ]. We outperform other methods by at least 5.1% without relying on self-training, which requires training a separate segmentation head, as used in COMUS.

#### Linear Semantic Segmentation Evaluation.

In this experiment, we keep the pretrained backbone frozen and train a linear layer on top of the spatial features to solve a supervised semantic segmentation task. Bilinear interpolation is used to match the spatial feature resolution to the image size, enabling the application of pixel-wise cross-entropy loss. This setup provides a better evaluation of the pretrained models compared to end-to-end finetuning, where all learned parameters are overwritten.
The results, reported in Table 2, show that NeCo surpasses CrIBo on all datasets by at least 10% and outperforms DINOv2R by 5% to 7%.
These significant improvements demonstrate that patches representing the same object or object part have higher similarities in feature space compared to other methods, as a simple linear layer can utilize these features for strong semantic segmentation.

#### Compatibility with Differently Pretrained Backbones.

As shown in Table 3, our method is generalizable across various self-supervised learning initialization, improving them by roughly 4% to 30% across different metrics and datasets. Surprisingly, NeCo even enhances the performance of methods specifically designed for dense tasks, such as CrIBo, TimeT, and Leopart. For instance, CrIBo demonstrates a performance increase of approximately 5% in overclustering evaluations, which measures how fine-grained and semantic the representations learned during pretraining are. This indicates that NeCo applied to CrIBo can extract more discriminative features, leading to improved transfer performance, shown by 0.5% and 3.7% better linear classification performance on Pascal VOC and COCO-Things.

#### End-to-End Full-Finetuning Evaluation.

One advantage of self-supervised pretraining is the ability to transfer learned general semantic features to specialized downstream tasks, improving performance in an end-to-end finetuning setup. We evaluate this capability of NeCo  by adding a transformer-based decoder from Segmenter [] on top of the feature extractor and finetuning the entire network for semantic segmentation. The backbone’s spatial features are fed into a transformer decoder along with $K$ learnable class tokens. These class and spatial tokens are projected onto each other to obtain patch-level predictions, which are then upsampled to match the input image size, enforcing pixel-wise cross-entropy loss. We report the mIoU scores achieved on Pascal VOC, Pascal Context, COCO-Stuff, and ADE20k in Table 4. Despite all parameters being adapted, the results show that NeCo  learns superior features, leading to better performance in downstream tasks, outperforming CrIBo by approximately 4%.
Notably, while DINOv2R has demonstrated strong transfer results in various tasks, including semantic segmentation, due to being trained on a massive dataset of 142M images and using a combination of dense and classification losses, NeCo surpasses even this.
By training only 19 GPU-hours on COCO, which is a fraction of the original compute, we obtain consistent gains, setting a new state-of-the-art.

**Table 4: Evaluation of Full Fine-Tuning with Segmenter. Various backbones pre-trained with different self-supervised learning methods are fine-tuned using Segmenter []. The table shows the mIoU scores obtained on validation sets across 4 different datasets.**
| Method | Backbone | Params | Pascal Context | Pascal VOC | COCO-Stuff | ADE20K |
| --- | --- | --- | --- | --- | --- | --- |
| DINO | ViT-S/16 | 21M | 46.0 | 80.3 | 43.2 | 43.3 |
| CrOC | ViT-S/16 | 21M | 46.0 | 80.9 | 42.9 | 42.8 |
| TimeT | ViT-S/16 | 21M | 47.4 | 80.4 | 43.1 | 43.5 |
| CrIBo | ViT-S/16 | 21M | 49.3 | 82.3 | 43.9 | 45.2 |
| DINOv2R | ViT-S/14 | 21M | 55.0 | 85.9 | 46.8 | 49.3 |
| NeCo | ViT-S/14 | 21M | 55.2 | 86.3 | 46.5 | 49.5 |
| DINO | ViT-B/16 | 85M | 45.8 | 82.2 | 44.4 | 45.0 |
| MAE | ViT-B/16 | 85M | 47.9 | 82.7 | 45.5 | 46.4 |
| CrIBo | ViT-B/16 | 85M | 49.2 | 83.4 | 44.6 | 46.0 |
| DINOv2R | ViT-B/14 | 85M | 57.8 | 89.3 | 49.7 | 55.3 |
| NeCo | ViT-B/14 | 85M | 58.0 | 89.5 | 49.8 | 55.1 |

**Table 5: Ablations of the key parameters of our method. We evaluate the models by training a linear layer on top of the frozen representations (Lin.) or using the in-context (IC) evaluation of [] using the validation images for PascalVOC12 and ADE20k.**
|  | Pascal | ADE20K |  |  |
| --- | --- | --- | --- | --- |
| Location | Lin. | IC | Lin. | IC |
| backg. | 78.4 | 60.5 | 35.8 | 20.6 |
| foreg. | 78.4 | 61.6 | 36.8 | 21.5 |
| both | 78.9 | 62.0 | 37.3 | 21.7 |

### 4.3 Ablation Studies

Here, we examine the essential parameters of our method by training NeCo on Pascal VOC12 and ADE20k. We assess its ability to perform linear segmentation and in-context scene understanding using the frozen representations learned with each set of parameters. For in-context scene understanding evaluations, we use $\frac{1}{128}$ fraction of the training data and reduce the spatial dimension to $448^{2}$. The number of training epochs for linear segmentation evaluations is set to 20 epochs. For more ablations refer to Appendix B.

#### Patch Selection Approach.

We demonstrate the effect of selecting patches from the foreground, background, or both in 5(a). Foreground patches are selected using the attention map averaged across heads. Our results indicate that selecting patches from the foreground gives 1% better results compared to the background selection in 3 out of 4 metrics. However, the performance peaks when we select patches from both locations. This improvement can be attributed to the use of scene-centric images for training, where the background often contains meaningful objects that contribute to enhanced performance.

#### Utilizing a Teacher.

We ablate the role of teacher-student architecture in 5(b). As shown, employing a teacher network updated by exponential moving average can significantly improve the performance across all the metrics by 8% to 20%. This is consistent with the previous works [, ], which reported a more stable training process when the teacher-student architecture is employed.

#### Nearest Neighbor Selection Approach.

We evaluate the influence of picking nearest neighbors from the same image (intra) or different batch images (inter) in 5(c). The results indicate that selecting patches across images consistently boosts performance by roughly 0.4% to 1% across different metrics. The higher diversity of patches involved in the latter approach likely accounts for this improvement.

#### Training Dataset.

5(d) presents the impact of the training dataset based on the ImageNet-100, Pascal, and COCO datasets. ImageNet-100 comprises relatively simple images with few objects, whereas Pascal and COCO feature more complex scenes with multiple objects. Our method shows consistent improvements when trained on multi-object datasets, achieving a performance increase of 2% to 7% on COCO compared to ImageNet-100. This improvement is due to the greater quantity and diversity of objects per batch in multi-object scenes, which provide stronger learning signals by requiring discrimination against a higher number of objects. Notably, the additional performance boost we observe from finetuning DINOv2R on Pascal—despite it already being trained on this dataset []—further underscores the efficacy of our proposed loss function.

#### Sorting Algorithm.

We ablate the effect of changing the sorting algorithm in 5(e). Our method demonstrates robust performance across different sorting approaches, achieving the best results with Bitonic sorting, which yields slightly better performance on average. We present the ablation study on the sorting steepness parameter in the Appendix B.4, as our method is robust to variations in this parameter.

## 5 Discussion

#### Broader impact.

By leveraging pretrained models, our method has implications for reducing the carbon footprint of model training. Similar to how once an LLMs is trained, it can be re-used and tuned with very little compute, we demonstrate that this is possible for visual foundation models. Our finetuning requires only 19 GPU-h, yet yields vastly better models for dense image understanding. We believe this work spurs further research into post-pretraining methods.

#### Conclusion.

In this work, we propose Patch Nearest Neighbor Consistency as a new method for dense post-pretraining of self-supervised
backbones. By applying our method to the many backbones including the DINOv2-registers model, we improve upon these models by a large margin for frozen clustering, semantic segmentation and full finetuning, setting several new state-of-the-art performances.

## Appendix A Experimental Setup

### A.1 Dense Post-Pretraining

#### Implementation Framework

Our model is implemented in Python, using Torch [] and PyTorch Lightning [].

#### Datasets

Our pretraining datasets consist of COCO [] and ImageNet-100 subset of the original Imagenet []. COCO contains approximately 118,000 scene-centric images, whereas ImageNet-100k includes around 100k object-centric images.

#### Data Augmentations

Our Data augmentations are the same as in []. More specifically, we use: random color-jitter, Gaussian blur, grayscale and multi-crop augmentations. Similarly, the global crop’s resolution is 224x224 and the local crop’s resolution is 96x96, for the all the experiments except when working with Dinov2 where we use 98x98 for local crops. Furthermore, our generated global and local crops have the constraint that they intersect at least by $1\%$ of the original image size.

#### Network Architecture

For our backbone, we employ vision transformers. More specifically, we train on ViT-Small and ViT-Base []. Moreover, following [, ], we use a student-teacher setup where the teacher weights are updated by the exponential moving average of the student weights.

#### Projection Head

Following [], the projection head consists of three linear layers with hidden dimensionality of $2048$, a Gaussian error linear units as activation function [], and an output dimensionality of $256$.

#### Dense Image Representation Alignment of Crops

Following [], due to the distinction between global and local crops, after projecting the dense spatial output to a lower space, the alignment step is applied on the dense image representations to bring them to a fixed spatial resolution of size of 7x7 during training. This ensures that the local and global crop feature maps have the same size and that they correspond to each other. The alignment is done using region of interest alignment (roi align) [].

#### Optimization

We train both network sizes with a cosine learning rate schedule going down to 0 0 over $25$ training epochs, except for the ablation studies where we use 10 epochs. The initial projection head learning rate is $1\mathrm{e}{-4}$ for all the experiments, whereas the backbone’s learning rate is $1\mathrm{e}{-5}$, with the exception of being $1\mathrm{e}{-6}$ when applying our method on Dinov2. The exponential moving average for updating the teacher’s weights is adapted with a cosine schedule starting at $0.9995$ and going up to $1$. We use Adam Optimizer [] with a cosine weight decay schedule.

#### Differentiable Sorting Networks

By default we use the Bitonic Differentiable Sorting Networks [] and the steepnesses (i.e., inverse temperatures) used for the network are $100$ for the Student and $100$ for the teacher. All the other parameters remain as the default ones; i.e., we use the $logistic_{\phi}$ function with a $\lambda=0.25$ for the interpolation of numbers in the differentiable sorting algorithms.

### A.2 Evaluation Setup

#### Visual In-Context Learning

The Dense Nearest Neighbor Retrieval Evaluation is a retrieval-based scene understanding evaluation introduced by []. Its goal is to assess the scene understanding capabilities of a dense image encoder. It works as follows:

- 1.
Memory Bank Construction: Using a dataset of images and their dense annotations, two memory banks are created. One memory bank stores image patch features extracted from the spatial output of a dense encoder applied to the training images. The other memory bank stores the corresponding patch labels from the dataset annotations.
- 2.
Query Processing: For an image from the validation split, the spatial output of the dense image encoder is processed. For each patch representation in this output, the $k$ nearest neighbors are identified from the memory bank of features. The labels of these nearest neighbors are then combined to construct the query’s label.
- 3.
Comparison: After constructing the annotation for the entire image, it is compared with the ground truth annotation.

Due to the unavailability of the original implementation by [], we use the open implementation from []. This implementation aligns with the original authors’ description and details, including the use of the ScaNN Library [] for efficient nearest neighbor retrieval. We adhere to the setup from the Hummingbird Model authors [] for our experiments. We use a memory size of 10,240,000 and configure ScaNN with 30 nearest neighbors, consistent with the evaluation of the Hummingbird model on this memory size.

The final results are reported as mean Intersection over Union (mIoU) on four different fractions of two datasets: Pascal VOC 2012 [] and ADE20K []. The sub-sampling factors are 1, 8, 64, or 128. For factors greater than 1, results are averaged over five different seeds. These dataset subsets are created by uniformly and randomly selecting a unique set of images from the training split, ensuring an approximately equal number of distinct images for each annotation label. For example, for the 1/128 fraction of the Pascal VOC 2012 dataset, we would collect around 83 images, ensuring each of the 20 labels (excluding the background) appears in at least 4 different images in the subset.

#### Overclustering

For the Overclustering experiment, following [], we run $K$-Means (using faiss []) on all spatial tokens from our backbone (i.e., with the projection head discarded) for a given dataset. We then group the clusters to the ground-truth classes of the dataset by applying greedy matching to the pixel-level precision and then run Hungarian matching [] on the combined cluster maps, which makes the evaluation metric permutation-invariant []. We use a crop size of 448x448 for the input images, and overclustering is applied on downsampled 100x100 masks in order to speed up the Hungarian matching. The final results are reported as an average of mean Intersection over Union (mIoU) over five different seeds on four different datasets: COCO-Thing and COCO-Stuff [], Pascal VOC 2012 [], and ADE20K [].

#### Linear segmentation

For linear segmentation, we closely follow the setup from Leopart []. Concisely, we take 448x448 images, encode them with our backbone to get the spatial outputs, apply bilinear interpolation to match the mask resolution, and finally apply a linear head to obtain the segmentation predictions. These predictions are then compared with the ground truth segmentation masks and trained via cross-entropy loss.

For training the linear head, we downsample the segmentation masks to 100x100 to increase training speed. We use Stochastic Gradient Descent with a weight decay of $0.0001$, a momentum of $0.9$, and a step learning rate scheduler. We found that a learning rate of $0.01$ works quite well for the backbone models we evaluated and our setup. We fine-tune the linear heads for 20 epochs.

Moreover, we train and evaluate linear heads on four versions of datasets: Pascal VOC 2012 [], subsets of COCO-Thing and COCO-Stuff (explained in Appendix A), and ADE20K [].

#### Segmenter Finetuning

Following the evaluation setup from [], we finetune our backbones and the transformer-based decoder from Segmenter [] in an end-to-end manner. We use the Segmenter implementation available within the MMSegmentation Library [].

The performance metric used here is the mIoU score, reported on four different datasets: Pascal Context [], Pascal VOC 2012 [], COCO-Stuff 164K [], and ADE20K []. The crop size used is 512×512. For the DINOv2 model and our method on it, we apply zero padding around the image of 512×512 to bring it to the size of 518×518.

The remaining configurations follow []. For the ADE20K and COCO-Stuff 164K datasets, we use 160k iterations, and for Pascal VOC 2012 and Pascal Context, we use 80k iterations, all with an eta_min of $0.1\cdot lr$. We use the Adam optimizer [] and for each pretraining method and dataset, we experiment with four different learning rates ($8\times 10^{-5}$, $3\times 10^{-5}$, $1\times 10^{-5}$, $8\times 10^{-6}$) before reporting the highest mIoU score.

#### Fully unsupervised semantic segmentation

To better evaluate the scene understanding abilities of our method, we also evaluate it using the Fully Unsupervised Semantic Segmentation Evaluation method []. This evaluation consists of two parts: Cluster-based Foreground Extraction (CBFE) and Overclustering with Community Detection (CD).

The CBFE clusters the spatial outputs of a model over a dataset and assigns each cluster as background (bg) or foreground (fg). The separation of foreground and background clusters is facilitated by attention maps from a Vision Transformer, which provide cues for the fg/bg distinction. We construct the final hard fg-bg assignment by averaging the attention heads, applying Gaussian filtering with a 7x7 kernel size, and retaining $70\%$ of the attention mass to obtain the final binary mask. The rest of the configurations remain the same as the original setup [].

The CD metric [] exploits local co-occurrence statistics among clusters to identify and categorize objects. This approach uses no labels for categorizing semantic parts; it simply finds local co-occurrence of clusters in an image by utilizing an information-theoretic definition of network communities. Our configurations for the CD evaluation remain the same as in Leopart [].

We use the implementation from Leopart [] and apply CBFE and CD on the non-augmented (train) split of Pascal VOC 2012 [], and evaluate on its full validation set. For CD, we report the best results over 10 seeds obtained from a hyper-parameter search, leading to our best parameters for CD+CBFE: weight_threshold = 0.07, markov_time = 1.2, and k_community = 189.

## Appendix B Additional Experiments

### B.1 Unsupervised Semantic Segmentation

In 1(b), we show the contribution of each component to the final clustering evaluation gains on Pascal VOC 2012 with 21 clusters for DINOv2R ViT-small. Starting with our method NeCo, we then apply Clustering Based Foreground Extraction (CBFE), and finally community detection (CD). We observe that CBFE provides the largest boost (23.3%) due to the high quality of our overclustering maps, as also shown in 1(a). While CD contributes a more modest increase (13.8%), which is about half as much as CBFE, combining both CBFE and CD results in a significant improvement, bringing the overall gain to 55.1%, compared to the initial 17.8%.

**Table 6: Component contributions. We show the gains that each individual component brings for PVOC segmentation and K=21.**
|  | mIoU |
| --- | --- |
| DINOv2R | 12.2 |
| + NeCo | 17.8 (+5.6%) |
| + CBFE | 41.3 (+23.3%) |
| + CD | 55.1 (+13.8%) |

### B.2 Full Clustering Tables

In Table 7, we show full clustering results shown by 1(b).

**Table 7: Clustering evaluation performance. K-means with various clustering granularity $K$ is applied to the spatial features obtained from different feature extractors on two datasets. The resulting cluster maps are matched to the ground truth by Hungarian matching [], and the intersection is reported in mIoU.**
|  |  |  | Pascal VOC | COCO-Things |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Backbone | Params | K=GT | K=300 | K=500 | K=GT | K=300 | K=500 |
| DINO [] | ViT-S/16 | 21M | 4.3 | 13.9 | 17.3 | 5.4 | 18.8 | 19.2 |
| iBOT [] | ViT-S/16 | 21M | 4.4 | 23.8 | 31.1 | 7.6 | 26.6 | 28.0 |
| CrOC [] | ViT-S/16 | 21M | 3.4 | 16.4 | 20.0 | 4.9 | 14.7 | 18.1 |
| TimeT [] | ViT-S/16 | 21M | 12.2 | 43.6 | 46.2 | 17.5 | 42.7 | 44.6 |
| DINOv2R [] | ViT-S/14 | 21M | 12.2 | 46.7 | 49.5 | 12.3 | 38.9 | 41.2 |
| CrIBo [] | ViT-S/16 | 21M | 18.3 | 51.3 | 54.5 | 14.5 | 46.0 | 48.3 |
| NeCo | ViT-S/14 | 21M | 17.8 | 69.4 | 72.6 | 18.2 | 61.2 | 64.5 |
| MAE | ViT-B/16 | 85M | 3.5 | 6.0 | 7.4 | 6.9 | 9.2 | 10.1 |
| DINO | ViT-B/16 | 85M | 5.3 | 19.6 | 23.9 | 6.4 | 19.1 | 21.2 |
| iBOT | ViT-B/16 | 85M | 6.5 | 29.0 | 34.0 | 7.2 | 26.4 | 30.5 |
| DINOv2R | ViT-B/14 | 85M | 14.4 | 47.7 | 50.5 | 12.4 | 30.9 | 33.5 |
| CrIBo | ViT-B/16 | 85M | 18.9 | 56.9 | 56.8 | 16.2 | 43.1 | 44.5 |
| NeCo | ViT-B/14 | 85M | 18.6 | 64.2 | 71.8 | 13.3 | 61.3 | 65.5 |

### B.3 Full Visual In-Context Learning Tables

We show the results shown by Figure 2 in Table 8. To provide a more comprehensive comparison, we also evaluate our method against SelfPatch [], a finetuning approach with a similar aim of enhancing dense representations.

**Table 8: In-context scene understanding benchmark. Dense nearest neighbor retrieval performance is reported across various training data proportions on two scene-centric datasets, ADE20k and Pascal VOC. The retrieved cluster maps are compared with the ground truth using Hungarian matching [], and their mIoU score is reported.**
|  |  |  | ADE20K | Pascal VOC |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Backbone | Params | 1/128 | 1/64 | 1/8 | 1/1 | 1/128 | 1/64 | 1/8 | 1/1 |
| DINO | ViT-S/16 | 21M | 9.5 | 11.0 | 15.0 | 17.9 | 26.4 | 30.5 | 41.3 | 47.9 |
| SelfPatch | ViT-S/16 | 21M | 10.0 | 10.9 | 14.7 | 17.7 | 28.4 | 32.6 | 43.2 | 50.1 |
| CrOC | ViT-S/16 | 21M | 8.7 | 10.8 | 15.2 | 17.3 | 34.0 | 41.8 | 53.8 | 59.5 |
| TimeT | ViT-S/16 | 21M | 12.1 | 14.1 | 18.9 | 23.2 | 38.1 | 43.8 | 55.2 | 61.3 |
| Leopart | ViT-S/16 | 21M | 12.9 | 14.8 | 19.6 | 23.9 | 44.6 | 49.7 | 58.4 | 63.4 |
| CrlBo | ViT-S/16 | 21M | 14.6 | 17.3 | 22.7 | 26.6 | 53.9 | 59.9 | 66.9 | 71.2 |
| DINOv2R | ViT-S/14 | 21M | 19.6 | 22.8 | 30.1 | 35.9 | 53.0 | 57.9 | 68.2 | 73.8 |
| NeCo | ViT-S/14 | 21M | 23.7 | 27.2 | 34.0 | 39.8 | 66.5 | 70.3 | 76.3 | 78.9 |
| MAE | ViT-B/16 | 85M | 10.0 | 11.3 | 15.4 | 18.6 | 3.5 | 4.1 | 5.6 | 6.9 |
| DINO | ViT-B/16 | 85M | 11.5 | 13.5 | 18.2 | 21.5 | 33.1 | 37.7 | 49.8 | 56.5 |
| Leopart | ViT-B/16 | 85M | 14.6 | 16.8 | 21.8 | 26.7 | 50.1 | 54.7 | 63.1 | 68.2 |
| Hummingbird | ViT-B/16 | 85M | 11.7 | 15.1 | 22.3 | 29.6 | 50.5 | 57.2 | 64.3 | 71.8 |
| CrlBo | ViT-B/16 | 85M | 15.9 | 18.4 | 24.4 | 28.4 | 55.9 | 61.8 | 69.2 | 73.1 |
| DINOv2R | ViT-B/14 | 85M | 22.1 | 25.8 | 33.2 | 38.7 | 51.8 | 58.9 | 70.6 | 76.2 |
| NeCo | ViT-B/14 | 85M | 29.1 | 33.7 | 39.2 | 44.2 | 67.0 | 71.4 | 77.5 | 82.2 |

### B.4 Extra Ablations

#### Sorting Steepness.

In 9(c), we vary the sorting steepness, denoted by $\beta$, for both teacher and student networks to evaluate the influence of hard or soft nearest neighbor assignments. The performance improves when the teacher’s steepness is higher or equal to the student’s, consistent with previous findings []. Our best results are achieved when both networks have equal steepness. However, extreme steepness values (e.g., 1000) harm performance. This is because sorting patch similarities lacks clear boundaries, and formulating it as a hard assignment can force incorrect orderings, negatively impacting performance.

#### Batch Size.

9(b) examines the impact of batch size on performance. We maintain the same experimental setup as reported in the paper for all ablations. As anticipated, smaller batch sizes result in marginal improvements due to the weaker supervisory signal. However, a moderately large batch size, such as the 64 used in the paper, shows significant improvements over the baseline. Furthermore, the trend suggests that increasing batch sizes could lead to even greater performance gains.

#### Number of Neighbors.

As mentioned in the method section, compute and sort the distances between each patch and all other patches using a differentiable sorting algorithm. Ultimately, The sorted distances for the same patches in different views should be similar. Here, we conduct an ablation study, selecting only the top $K$ distances for each patch instead of all distances. The results are reported in 9(a). As shown, incorporating more neighbours in NeCo  loss enhances performance. However, after reaching a threshold (e.g., 32 neighbors), adding more becomes less effective. This indicates our method’s robustness against this hyperparameter, not relying on computing distances for all batch patches to improve the baseline.

#### Training Epochs.

We show the performance across different training epochs in 9(d). As the table shows, even after just one epoch of training, DINOv2R improves by 1% to 3% across various metrics. The performance continues to increase with more training epochs, but the improvements become smaller after 25 epochs, which is the number used in the paper.

**Table 9: Ablations of the key parameters of our method. We evaluate the models by training a linear layer on top of the frozen representations (Lin.) or using the in-context (IC) evaluation of [] using the validation images for PascalVOC12 and ADE20k.**
|  | Pascal | ADE20K |  |  |
| --- | --- | --- | --- | --- |
| Num | Lin. | IC | Lin. | IC |
| 4 | 74.4 | 54.8 | 35.2 | 19.7 |
| 8 | 76.8 | 61.1 | 36.3 | 20.2 |
| 16 | 77.7 | 60.9 | 36.7 | 20.9 |
| 32 | 78.1 | 61.3 | 37.1 | 21.4 |
| All | 78.9 | 62.0 | 37.3 | 21.7 |

### B.5 Computational Analysis

We provide a detailed runtime analysis for DINO, CrIBo, TimeT, and NeCoin Table 10. All experiments are conducted on 8 NVIDIA RTX A6000-46GB GPUs. As shown in Table 3, starting from already existing pretrained networks, NeCo  only requires 2.5 GPU hours to enhance the performance of the state-of-the-art method, CrIBo, by 3.7% in linear segmentation. Furthermore, when NeCo  is applied to TimeT, starting from DINO initialization, it can surpass CrIBO with nearly 30% less total training time (20h + 1.6h + 2.5h vs. 70h). This trend is also observed in clustering and overclustering numbers, showing up to a 6% improvement over CrIBO when starting from TimeT, as detailed in Table 3. These results validate the computational efficiency of the proposed method.

**Table 10: Computational analysis and segmentation performance. NeCo demonstrates superior computational efficiency, requiring only 2.5 GPU hours to enhance CrIBo’s performance by 3.7% in linear segmentation.**
| Method | Epoch time | Init | Num epoch | Batch size | Total training time | Linear segmentation |
| --- | --- | --- | --- | --- | --- | --- |
|  | (Min:Sec) |  |  |  |  |  |
| DINO | 1:33 | random | 800 | 64 | 20h | 43.9 |
| TimeT | 3:12 | DINO | 30 | 64 | 2h | 58.2 |
| CrIBo | 5:16 | random | 800 | 64 | 70h | 64.3 |
| NeCo | 4:48 | TimeT | 25 | 64 | 1.6h | 64.8 |
| NeCo | 4:48 | CrIBo | 25 | 64 | 2.5h | 68.0 |

## Appendix C Additional Visualizations

#### Visualization of nearest patch retrieval.

In Figure 4, we take one patch from an image in Pascal VOC as the query and retrieve its seven nearest patches across the dataset. We compare NeCo  against DINOv2R. As illustrated, the nearest neighbors retrieved by NeCo  are not only more relevant compared to DINOv2R but also more precise, successfully finding nearest patches not only within the same object but also within object parts.

In Figure 5, we show some borderline cases where DINOv2R retrieves more relevant patches than our method. NeCo  occasionally retrieves patches of similar parts from different objects. For example, a patch from a bicycle wheel might be matched with a motorcycle wheel. Additionally, since we rely on cropping to induce nearest neighbor similarity, small objects in the input, which may not significantly affect the overall semantics, can alter the semantics at the patch level, leading to unexpected nearest neighbors, as seen in the case of the sheep photo.

#### Visualization of clustering and Overclustering.

We display the visualizations for both the clustering and overclustering approaches in Figure 7 and Figure 6, respectively. For the clustering approach, detailed in 1(b), we apply cluster-based foreground extraction combined with community detection to identify the foreground regions from features extracted across the entire dataset. The extracted features are then masked with the extracted masks and clustered according to the number of objects in the dataset, which is 21 for Pascal. As shown in Figure 7, this process successfully assigns unique cluster IDs to the detected objects and accurately sketches their boundaries.

For overclustering, we don’t extract foreground regions and instead cluster all the features into a significantly higher number of clusters compared to the ground truth. For Pascal, this number ($K$) is set to 100. Figure 6 illustrates the results. We observe a similar effect as with clustering, except that some objects, such as humans or certain animals, are partitioned into their constituent parts, which remain relatively consistent across different samples.

Figure: Figure 3: Pascal VOC visualizations. We overlay the ground truth on top of a subset of images in Pascal VOC. These images and their ground truth segmentation maps are used for our tasks, such as visual in-context learning and linear segmentation.
Refer to caption: x3.png

Figure: (a) DINOv2R
Refer to caption: x4.png

Figure: (a) DINOv2R
Refer to caption: x6.png

Figure: (a) Input
Refer to caption: x8.png

Figure: Figure 7: Fully unsupervised segmentation on Pascal for K=21. We extract foreground masks using the CBFE+CD method, following the approach outlined in []. These masks are then clustered into the number of objects present in the Pascal dataset, with $K=21$. As demonstrated, NeCo yields distinct and accurate segmentation maps for each object.
Refer to caption: x11.png

## Appendix D Dataset Details

Pascal VOC 2012 []
This dataset, the latest split version of trainaug, features 10,582 images and their annotations distributed across 21 classes, with one referring to the background class. The validation set consists of 1,449 images. Following [] we ignore unlabelled objects as well as the boundary class. Moreover, for hyper-parameter tuning of the fully unsupervised segmentation method [] that we apply on our method, we use the PVOC12 train split with 1464 images. Figure 3 shows the dataset images overlaid by the annotations.

Pascal Context [] This scene-centric dataset includes 4,998 training images covering 60 semantic classes, including the background. The validation set consists of 5,105 images. We use this dataset for the Linear Segmentation and Segmenter experiments, via the MMSegmentation Library [].

COCO-Stuff 164K [] This scene-understanding dataset includes labels across 91 "stuff" categories and 80 "things" categories. The training set comprises 118,000 images, and the validation set contains 5,000 images. We follow the same setup as [] and thus we use the COCO benchmark in two ways to isolate further the given object definitions.

Concisely, we begin by extracting stuff annotations, which refer to objects without clear boundaries and often found in the background, using the COCO-Stuff annotations []. Then, we consolidate the 91 detailed labels into 15 broader labels, as described in [] and we assign the general label “other” to non-stuff objects, as this label lacks specific semantic meaning. Non-Stuff objects are ignored during training and evaluation. We indicate this version of the dataset within our work as COCO-Stuff used in Overclusterring and Linear Segmentation in Appendix A.2.

Next, we extract foreground annotations utilizing the panoptic labels from []. We combine the instance-level annotations into object categories using a script provided by the authors. Additionally, we consolidate the 80 detailed categories into 12 broad object classes.The background class is ignored during training and evaluation. This leads as to the COCO-Thing version of the dataset which we use for the Overclusterring and our Linear Segmentation in Appendix A.2.

ADE20K [] The dataset is a collection of images used for semantic segmentation tasks, featuring finely detailed labels across 150 unique semantic categories. Some of the categories include stuffs like sky and grass, as well as distinguishable objects like person, and a car. Overall, it includes a wide variety of scenes, with 20,210 images in the training set and 2,000 images in the validation set, making it one of the most challenging and diverse datasets for scene understanding. We use the full dataset in our experiments. In our experiments, we ignore the others label of the dataset.

Imagenet [] The dataset, is a large-scale visual database designed for use in visual object recognition research. It contains over 1.3 million images categorized into 1,000 object classes. Each image is labeled with detailed annotations, making it a critical resource for training and evaluating machine learning models, particularly in the field of computer vision. In our work, we also explore training on part of the Imagenet, the Imagenet100k that consists 100K images across 100 classes, from the original dataset.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Could the authors provide additional results of semantic segmentation on urban-scene datasets like Cityscapes and remote sensing datasets?
- Could the authors explain why the fully-fine-tuning results are not better than the original pre-trained encoders? Indeed, it can be expected that the post-trained encoder is good than pre-trained encoder when evaluated under a frozen state, as the feature relationships are explicitly strengthened by the proposed strategy.
soundness: 3
strengths: 1. The paper is clearly written and easy to follow.
2. The motivation of further improving advanced vision foundation models is promising, especially by designing a lightweight post-training technique.
3. The results under frozen encoders are impressive, compared with the original encoders.
4. The method of applying sorting-based consistency regularization is new to the post-training field.
summary: This paper proposes to enforce consistency of the sorting relationship between the student model and the EMA teacher model as a post-training strategy for advanced vision foundations models, such as DINOv2. The model is post-trained on the COCO or Pascal dataset. Improvements are observed in downstream semantic segmentation tasks on Pascal and COCO, especially under the evaluation protocol of linear probing.
weaknesses: 1. My main concern lies in the difference between sorting-based regularization and a much simpler soft cross-entropy loss. It means, if we directly compute the similarity between $F_s \in R^{N \times d}$ and $F_r  \in R^{N_r \times d}$ as well as the similarity between $F_t  \in R^{N \times d}$ and $F_r  \in R^{N_r \times d}$ (suppose the obtained two similarity maps are $S_s, S_t \in R^{N\times N_r}$), can we directly apply a soft cross-entropy loss between $S_s$ and $S_t$ along the $N_r$ dimension? Will this be a stricter optimization target than the plain sorting-based strategy? Because it not only requires the sorted order to be the same, but also their pair-wise similarities with the referenced patches to be the same.

2. It is good to achieve promising results on downstream Pascal, COCO, and ADE20K datasets, when applying the post-training strategy. But I am wondering whether the post-training stage, especially only conducted on a small-scale COCO dataset, will potentially ruin the original DINOv2's good representation in broader scenarios, such as sketch images or urban-scene images (Cityscapes). It will be good to know how the post-trained model trades off between generalization and scene specification.

3. The proposed method fails to achieve obvious improvements over the original pre-trained encoders when evaluated under the fully-fine-tuning protocols (Table 4). For example, although the model is explicitly fine-tuned on COCO, it is still inferior to the original DINOv2 on COCO-Stuff dataset (46.8 *vs.* 46.5). Are there any insights into these results?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: It would be good to clarify the main weakness of proposed baselines. An understanding of how the method performs against baseline approaches, when their training flops/epochs are matched can bolster the claims made in the paper.
soundness: 2
strengths: 1. The paper is well written, with a wide range of ablations substantiating most of the design choices introduced in the method.
2. The improvements obtained by the proposed method in linear probing and multiview feature consistency evaluations indicate the superior quality of the learned features. Overall the method shows good improvement in a wide range of downstream evaluations.
summary: The paper introduces a new self-supervised training loss called Patch Neighbor Consistency (NeCo). The key benefit of the proposed method is that the loss yields richer/fine-grained supervision over contrastive approaches. The proposed training scheme is applied over DINOv2 pretrained backbones and shows further improvements on a wide range of downstream tasks.
weaknesses: 1. The main weakness of the proposed method is the baselines. Since the proposed method is introduced as post-training adaptation scheme. It would be fair to continually pretrain DINOv2 and other pretraining methods also by the same duration as NeCo. An ablation on matching training flops/epochs between all the pretraining methods and NeCo would be good to have in the paper.
2. The method has carefully crafted hyperparameters to ensure some overlap between teacher and student patches. Is it possible that the method can do away with ROIAlign by not performing any geometric transformations in data augmentation. At the moment it seems like ROIAlign is used to undo the geometric transformation applied to the image presented to the teacher model.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1.	Please give more details on sampling and collecting patch features with ROIAlign. How the box coordinates of patches are sampled? Also, how the different views of one single image are sampled? 
2.	How the sorting algorithm is implemented? Are O(nlogn) algorithms such as quick sorting (other than odd-even sorting mentioned in the manuscript) compatible with differentiable swapping?
soundness: 3
strengths: 1.	The paper is well-written. 
2.	Experiments are extensive and convincing, providing rich results on various settings.
3.	The proposed method is simple, interesting, and effective, surpassing previous works by a large margin.
summary: The paper introduces NeCo, a self-supervised training algorithm that enhances patch-level nearest neighbor consistency across student and teacher models, aiming at improving scene understanding in vision foundation models. NeCo leverages differentiable sorting on pretrained representations, such as DINOv2, to boost learning signals and performance across various models and datasets. The method requires minimal training time and establishes new state-of-the-art results in non-parametric in-context semantic segmentation and 3D understanding.
weaknesses: 1.	The major contribution of NeCo seems to be introducing sorting consistency into visual representation pretraining. In pages 4-5, the description of differentiable swapping and sorting takes majority of space in Section Method, but these contents are proposed by previous works and simply pasted here. 
2.	The method requires a pretrained backbone (e.g. DINO) for visual representation learning, and outperforms previous methods. However, some previous learning methods (e.g. MAE) does not require a pretrained backbone, so the improvement may partially come from the additional information brought by the pretrained backbone DINO. Is it possible to use the sorting consistency between different views to learning representation without the pretrained backbone teacher?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses. I also have an additional empirical question: Would incorporating a 'repel' signal for distant patches (similar to contrastive loss) alongside the computation of near patch consistency enhance the performance of the proposed method?
soundness: 3
strengths: 1. It makes sense to perform self-supervised learning by bringing Neighbor Patch Consistency closer.

2. Although the overall method design is based on existing simple strategies, it shows significant performance improvements. The proposed method achieves SOTA on several downstream tasks.

3. Further self-supervised training based on existing pretrained models (e.g., DINOv2) can significantly reduce computational cost and GPU training time.
summary: The paper introduces NeCo, a self-supervised training loss that enforces patch-level nearest neighbor consistency between a student and teacher model. Unlike binary contrastive methods, NeCo leverages fine-grained learning signals by sorting spatially dense features relative to reference patches. By applying differentiable sorting on top of pretrained representations like DINOv2, NeCo achieves superior results across various models and datasets with minimal GPU time.
weaknesses: 1.The comparative experiments in the paper are not comprehensive. If DINOv2 is used as a baseline, it is necessary to align with the downstream tasks validated in the DINOv2 paper as much as possible. Additionally, if the goal is to claim dense prediction capability, it is also important to validate the model’s representation abilities on more instance and dense recognition tasks, such as CityScapes, KITTI, and NYUd.

2.Regarding pairwise distance computation, if this method only calculates distances within a single batch, it results in a limited computational space and may not capture all potential patches that are near the selected patches. 

3.Obtaining patch features prior to distance computation heavily relies on the feature extraction capability of the pretrained foundation model. If the pretrained foundation model introduces noise in feature extraction (a common phenomenon), how does your method prevent the accumulation of such errors? 

4.The use of cosine similarity for distance calculation may not be the most accurate choice, as it lacks an ablation study.

5.Does the proposed Patch Neighbor Consistency self-supervised strategy have generalizability? For instance, can it be applied to more foundation models like CLIP, SIGLIP, SAM, etc.?  In other words, can the proposed method adapt to more pretrained models as in the experiment in Table 3?

6.Additionally, is this method suitable for transformer models trained from scratch, assuming it can use another foundation model to extract features for distance computation?

Meta-review: This paper introduces NeCo, a self-supervised training loss intended to enforce patch-level nearest neighbor consistency between student and teacher models. NeCo is a lightweight post-training technique that leverages differentiable sorting applied to pretrained representations to enhance the learning signal and boost overall performance. The results show significant improvements across various downstream tasks, achieving new state-of-the-art results in non-parametric in-context semantic segmentation and 3D understanding, all while requiring minimal training time.

After the rebuttal phase, the paper received consistently positive scores: 6, 6, 6, and 8, leading to an average score of 6.5. All reviewers praised the paper for being well-written, sound, and easy to understand, highlighting that the proposed NeCo is both simple and effective. Based on this feedback, the area chair recommends accepting the paper.

Decision: Accept (Poster)