EffoVPR: Effective Foundation Model Utilization for Visual Place Recognition

## Abstract

Abstract The task of Visual Place Recognition (VPR) is to predict the location of a query image from a database of geo-tagged images. Recent studies in VPR have highlighted the significant advantage of employing pre-trained foundation models like DINOv2 for the VPR task. However, these models are often deemed inadequate for VPR without further fine-tuning on task-specific data.
In this paper, we propose a simple yet powerful approach to better exploit the potential of a foundation model for VPR. We first demonstrate that features extracted from self-attention layers can serve as a powerful re-ranker for VPR. Utilizing these features in a zero-shot manner, our method surpasses previous zero-shot methods and achieves competitive results compared to supervised methods across multiple datasets.
Subsequently, we demonstrate that a single-stage method leveraging internal ViT layers for pooling can generate global features that achieve state-of-the-art results, even when reduced to a dimensionality as low as 128D. Nevertheless, incorporating our local foundation features for re-ranking, expands this gap. Our approach further demonstrates remarkable robustness and generalization, achieving state-of-the-art results, with a significant gap, in challenging scenarios, involving occlusion, day-night variations, and seasonal changes.

## 1 Introduction

The task of Visual Place Recognition (VPR), also known as Geo-Localization, aims to predict the place where a photo was taken relying solely on the visual information in the image. This is typically done by an image retrieval approach [, , , , , ] where a database of geo-tagged images is used, often referred as gallery. Real world data, including tagged VPR datasets, rely on two major sources for images: 1) car street-view and 2) people personal cameras (commonly mobile phones) [, , , , ]. As a result, images contain natural objects that are irrelevant and sometimes misleading for VPR task. Figure 1 - top row, shows an example, where people, vehicles, daylight, or camera angles might differ between images.

Modern models commonly use a deep neural network to extract a so-called global feature (a.k.a descriptor) for the query and gallery images [, , , , ]. This single-stage approach is then followed by a nearest neighbor search in the feature space, to retrieve the matching candidates from the gallery. In practice, global features of the entire gallery need to be uploaded to RAM to enable fast retrieval. In large-scale and real-world scenarios, reducing the memory footprint for each image is crucial to ensure real-time applicability. Therefore, several works often promote compact features to achieve both accuracy and applicability [, , ]. A popular strategy to improve the accuracy is to conduct a two-stage search, with subsequent similarity search on the top-k ranked results, that re-ranks the first-stage retrieved results. This process is commonly performed by matching local key-points and their corresponding descriptors.

Following best practices in computer vision, VPR methods are often initialized with ImageNet pre-trained weights (*e.g*. [, ]), followed by finetuning on VPR datasets, *e.g*. MSLS [], or trained from scratch [, , ]. Recent advancements in VPR exploit the capabilities of foundation models [, , , ] such as DINOv2 [], a transformer [] based model trained by self-supervised learning on a vast amount of data.
Recent approaches [, , ] argue that using vanilla DINOv2 (without fine-tuning) is ineffective. They criticize this approach for failing by capturing dynamic and irrelevant elements (pedestrians or vehicles), thereby diverting attention from crucial VPR features (buildings or scene layout). Some methods propose to address this issue by integrating learned adaptors into the foundation model architecture [, ] or changing the standard fine-tuning scheme with a unique pooling mechanism [], in order to facilitate the foundation model adaptation to the VPR task. In contrast, Anyloc [] suggested to use DINOv2 as a general-purpose feature representation at zero-shot for various localization tasks. However, despite achieving its best performance with the extremely high dimensional features (of size 49K), faces scalability challenges, AnyLoc only demonstrates a modest performance, and struggles in cases where queries exhibit large appearance changes.
In this paper, we utilize the internal ViT features to firstly surpasses previous zero-shot approaches and achieve competitive results compared to several VPR trained methods.

Typically, VPR methods [, , , , , ] adhere to a conventional approach of initially extracting local features from a pretrained model, then using pooling methods such as GeM [] or NetVLAD [], to obtain a global feature, for each image.
Methods using VLAD[], such as AnyLoc[], or NetVLAD[] for aggregating local features necessitate learning a dictionary for each specific gallery. These models tend to "overfit" to the particular gallery distribution, which hampers their generalization. Additionally, they often require fine-grained clustering, leading to high-dimensional feature vectors.
In addition, a recent study[], highlighted the GeM hyper-parameter sensitivity, spatial information loss, and convergence complications.
Alternatively, we suggest to utilize the internal ViT self-attention mechanism, and training with the class $[CLS]$ token, for classification loss. This approach allows implicit aggregation from local features to a global representation and
eliminates the need for previously used external special components and learned features, that often yield excessively large feature representation [, , ] (see Figure 1).

Figure: (a)
Refer to caption: extracted/5625734/imgs/rerank_pair_1-sf-occ-1.jpg

We hereby present an Effective foundation based VPR method called EffoVPR which achieves a superior single-stage performance without the need for additional components. Unlike more intricate DINOv2-based methods with built-in adapter components [, ] or special aggregation techniques [], EffoVPR delivers State-of-The-Art (SoTA) performance across multiple datasets while maintaining compactness, as shown in Figure 1. Our two-stage approach, builds upon local feature matching originated from the internal ViT attention maps, thus further expanding our performance gap. Additionally, EffoVPR demonstrates strong generalization and robustness across various cities and landscapes, effectively handling challenges such as occlusions, time differences and seasonal changes as demonstrated in Figure 1, which illustrates significant occlusion.

To summarize, our contributions in this work include:

- 1.
We present a method that effectively leverages a foundation model for the VPR task, exploiting its intermediate attention layers. Leveraging this finding, we propose a zero-shot method that outperforms previous zero-shot approaches while showing comparable results, even with trained VPR methods, on few datasets.
- 2.
Our work introduces a novel foundation model based VPR approach that eliminates the need for external aggregation methods or specialized pooling layers (such as NetVLAD or GeM).
- 3.
We suggest an effective yet simple re-ranking process based on ViT internal attention layers, that significantly boosts performance.
- 4.
Our method achieves state-of-the-art performance across various VPR benchmarks and generalizes to challenging scenarios, as occlusions, day-night variations or seasonal changes.

## 2 Related work

Traditional approaches utilized a single-stage approach for Visual Place Recognition (VPR), using SIFT [] SURF [], or RootSIFT [], focused on matching queries to gallery images through the use of image local feature matching.
Two-stage methods[, , , ] entail an initial ranking based on global representation similarity, succeeded by re-ranking the top-K retrieved candidates in the second stage, utilizing local features.

First deep learning approaches for the VPR task used CNN [, , ] that was later replaced with Vision Transformers (ViT) [, , , , , ].
TransVPR [] and R2Former [] suggested the application of transformers for VPR and adopted a two-stage approach that included re-ranking. However, training from scratch or initialized on ImageNet, and lack of effective view-variability in training (in contrast to [, ]) has restricted their performance.

A handful of visual foundation models (VFMs) have recently emerged as the backbones for numerous tasks. VFMs like CLIP [], DINOv2 [], SAM [] use ViT that are trained with distinct objectives, exhibiting unique characteristics for various downstream applications often even at zero-shot (without requiring any fine-tuning).
Among zero-shot applications, AnyLoc [] proposed a single-stage solution for VPR, that leveraged a DINOv2 pre-trained model alongside dense local (patch-level) features. By employing VLAD for feature pooling across multiple layers, AnyLoc approach produces an extremely large global representation of $\sim$49K dimensions, subsequently reduced to 512D through PCA whitening, but in expense of lower performance. However, AnyLoc’s VLAD aggregation tends to fail in generalizing to queries with large time-gaps, day vs. night or of different season. In this paper, we suggest a two-stage zero-shot method based on first global ranking, then re-ranking, by matching local descriptors, extracted from the ViT self-attention layers.

On the contrary, SALAD [] proposed fine-tuning the pre-trained DINOv2 model to improve its performance for VPR task. Their single-stage approach involves pooling local features from the output layer of DINOv2 replacing NetVLAD’s soft-assignment to clusters by an optimal transport methodology. Notably, their most significant results are achieved with large features exceeding 8K dimension, which adversely impacts memory consumption.

Using DINOv2 as their backbone, SelaVPR [] and CricaVPR [] took a different path and avoided fine-tuning of the model, claiming for a side-effect of a catastrophic forgetting and performance deterioration. They propose the incorporation of a trainable adapters integrated into DINOv2 architecture. SelaVPR suggested a two-stage methodology that trains global and local features on two separate branches, built above DINOv2’s output tokens, while discarding the standard [CLS] token. Their re-ranking strategy relies on mutual-Nearest Neighbor (mutual-NN) computed over a newly learned patch level (local) features. CricaVPR instead, suggests learning a features pyramid above DINOv2’s output tokens, to learn a special viewpoint robust encoder. Both, SelaVPR and CricaVPR incorporate adapters in the DINOv2 architecture, using GeM pooling for feature aggregation and are trained with a contrastive loss.

Training a VPR model often sourced from Street-View images, introduces a significant challenge in selecting appropriate positive and hard negative images[, , , ].
EigenPlaces [] introduced a novel training paradigm that organizes the training data into classes, with each class containing multiple viewpoints of the same scene.
By employing a classification loss rather than the conventional contrastive loss (used in [, , ]), the resulting model demonstrated high resilience to varying viewpoints in testing. Hence, we adopt a similar strategy for our training process.

Current VPR methods predominantly, derive a global representation by aggregating local features obtained from the output layer of a trained backbone. This aggregation is often conducted using external learnable pooling techniques, such as VLAD (Vector of Locally Aggregated Descriptors) [], SPoC (Sum-Pooled Convolutional features) [], RMAC (Regional Maximum Activation of Convolutions) [], the known NetVLAD [] or the nowadays widely-used GeM (Generalized Mean) pooling layer[, ], as utilized in various studies [, , , , , , , ].
However, NetVLAD is encumbered by high computational costs and dimensional complexities (with $\sim$ 32K dimensional feature)[], while GeM encounter convergence issues and require hyperparameter tuning []. In contrast, we suggest to forgo the explicit and external aggregation approach, replacing it with training of the single [CLS] token in ViT, with classification loss.

In summary, this paper revisits the potential of foundation DINOv2 for VPR task, considering two-stage zero-shot and single as well as two-stage finetuned models. Unlike previous approaches, we propose leveraging the existing internal aggregation layers in DINOv2 without any additional components or adaptors. Notably, our approach achieves SoTA results with compact features, a crucial consideration given realistic memory constraints.

## 3 Method

For the VPR task, we explore ViT [] feature maps as local patch descriptors. In a ViT architecture, an
image is split into $p$ non-overlapping patches which are processed into tokens by linearly projecting each patch to a $d$-dimensional space, and adding learned positional embeddings. An additional [CLS] token is inserted to capture global image properties. The set of tokens are then passed through $L$ transformer encoder layers, each consists of normalization layers, Multihead Self-Attention modules, and MLP blocks. In ViT, each patch is directly associated with a set of features, a Key, Query, Value that can be used as patch descriptors. We utilize the self-attention matrices at layer $l$, $\{K_{l},Q_{l},V_{l}\}\in\mathbb{R}^{p\times d}$, with $p$ indicating the number of patches resulting $p+1$ tokens (number of patches in the image plus one added global token) and $d$ standing for the embedding dimension. The self-attention function at layer $l$ is then given by:

$$ $\text{Attention}(Q_{l},K_{l},V_{l})=\text{Softmax}(\frac{K_{l}^{T}Q_{l}}{\sqrt {d}})V_{l}$ (1) $$

We indicate the key feature of the $[CLS]$ token at layer $l$ by $k_{cls}\in K_{l}$.
Each image patch therefore is directly associated with the set of features $\{q_{i}^{l},k_{i}^{l},v_{i},^{l}\}_{i\in{[p+1]}}$ including its query, key and value at each layer $l\in[1,L]$, respectively.

In the global search stage, the aim is to perform an efficient search across a vast corpus of images in a gallery. To this end, a shared embedding space for query and gallery images is typically utilized to identify the most similar images and rank them accordingly. In the re-ranking stage, the top-K nearest neighbors (candidates) are further refined by evaluating the mutual similarity between their local features and those of the query image. Throughout both stages, we use a ViT [] encoder as our backbone model. Our EffoVPR retrieval method is presented in Figure 2.

Figure: Figure 2: An overview of EffoVPR. Left: During inference, we identify the nearest neighbors of the query by using the [CLS] token as the global representation for each image ($v_{g}$). For the second re-ranking stage, we extract (dashed-line) intermediate value features $V$ and filter them (with a predefined threshold $T_{1}$) using a score derived from the partial attention maps $S$. Right: Lastly, we re-rank the top-K candidates from the first stage based on the count of strongly connected mutual nearest neighbors (MNN) with a score exceeding a predefined threshold (denoted as $T_{2}$).
Refer to caption: x3.png

### 3.1 Training strategy

Our training strategy employs a classification loss applied on the output $[CLS]$ token. Following EigenPlaces [], we partition the map into cells measuring $15\times 15$ meters each, where cells define classes. Each class contains images capturing the same location from various viewpoints, enhancing the model’s ability to accurately recognize locations despite substantial changes in the captured view angle. To enforce class discrimination while ensuring viewpoint invariance, we utilize the Large Margin Cosine Loss (CosFace)[, ].

We initialize our ViT[] backbone with pre-trained DINOv2 weights [, ]. To retain the rich visual representations learned during pre-training, while adapting the model for the VPR task, we fine-tune only the final layers of our backbone. Note that due to the inherent design of the ViT architecture, which incorporates a self-attention mechanism (Eq. 1), the global feature is implicitly trained to aggregate local features without the need of additional specialized components. For learning more compact global feature we simply add a linear layer on top of the output [CLS] token.

### 3.2 Inference strategy

Global ranking stage:
Post-training, we extract the global feature of an image from the $[CLS]$ output token of the penultimate classification layer. We then retrieve the K nearest neighbors of the query’s global feature.

Re-ranking stage:
We start by extracting patch local features from each candidate among the top-k retrieved images in the previous global stage, and re-rank the candidates based on these features. Local features are derived from the intermediate layer $l$ of ViT, utilizing the self-attention matrices, thus their computation is integrated into the process of computing global features and does not necessitate recalculation.
Aligned with the findings in [], where $V_{l}$ showed to have a higher instance-level characteristics, we find $V_{l}$ matrix mentioned in Eq. 1 as the most propitious facet for the task at hand (see Tab. S3 in Appx.). Then we leverage the model’s internal prioritization to identify discriminative features and extract the attention map: $S:=\text{Softmax}(Q_{l}\cdot k_{cls})\in\mathbb{R}^{p}$, which represents the attention of each Value feature with the global $[CLS]$ image representation key-token $k_{cls}$.
We therefore select a subset $\mathcal{V}$ of the values $\mathcal{V}\subseteq\{v_{1},...,v_{p}\}:=V_{l}$ as the image’s local features, based on the score $S$: $\mathcal{V}:=\{v_{i}~{}|~{}S_{i}>T_{1}\}$ for a predefined threshold $T_{1}$.
We use a rather low threshold that filters out patches that are “less significant” as reflected from the layer’s attention.
Note that the number of selected local features might differ between images.

Next, given the query’s local features $\mathcal{V}$ and a candidate’s $\mathcal{V}^{\prime}$, we calculate the pairs of mutual nearest neighbors (MNN) between $\mathcal{V}$ and $\mathcal{V}^{\prime}$ by the number of feature pairs that are the nearest neighbor of each other. Our observation revealed that applying a threshold to the MNN scores enhances the model’s resilience to clutter and directs its attention to the pertinent key-points for accurate matching. We therefore count only the pairs with cos-similarity higher than a predefined threshold $T_{2}$ (for ablation on thresholds see Tab. S3 in Appx.). We formulate that process in Equation 2:

$$ $\text{MNN}(\mathcal{V},\mathcal{V}^{\prime}):=\{(v_{i},v^{\prime}_{j})\in \mathcal{V}\times\mathcal{V}^{\prime}~{}|~{}v_{i}:=NN(V,v^{\prime}_{j})~{} \text{and}~{}v^{\prime}_{j}:=NN(v_{i},V^{\prime}),~{}v_{i}^{T}v^{\prime}_{j}>T _{2}\}$ (2) $$

Finally, the re-ranking stage concludes by sorting the top-K candidates based on their MNN counts. Note that our re-ranking strategy, which matches local features, does not require any additional learning, optimization, or spatial verification. The thresholds are established once and remain fixed across all test sets (20 different scenarios).

Zero-shot: For the first stage ranking we use the [CLS] token from vanilla-DINOv2. The results are then refined by re-ranking, while employing our suggested features $\mathcal{V}$ with MNN in Eq. (2).

## 4 Evaluation

In this section, we compare our approach with several SoTA VPR methods following the common VPR Benchmarks[].
We propose, a single-stage (EffoVPR-G) and two-stage, that includes a re-ranker (EffoVPR-R) approaches with backbone trained on the publicly available SF-XL [] dataset containing streetview of San Francisco. For more implementation details see Appx.
We then test EffoVPR on a large number of diverse datasets (20), including *e.g*. Pitts30k [], Tokyo24/7[], MSLS-val/challenge[] Nordland[] and more, exhibiting a wide variety of conditions, including different cities, day/night images, and seasonal changes. Note that, the MSLS Challenge[] is a hold-out set whose labels are not released, but researchers submit the predictions to the challenge server to get the performance. More details on the benchmarks can be found in the Appendix.

Datasets with gallery made from street-view images and with largest viewpoint variance, include Tokyo 24/7[] and SF-XL[, ], where the query images are collected from a smartphone, usually from sidewalks. Most datasets are from urban footage, with the main exception being Nordland[], which is a collection of photos taken across different seasons with a camera mounted on a train. Some datasets present various degrees of day-to-night changes, namely MSLS [], Tokyo 24/7 [], SF-XL []
SVOX-Night []. AmsterTime [] contains grayscale historical queries and modern-time RGB gallery images, making it the only dataset with large-scale time variations of multiple decades.

We follow common evaluation metric used in previous works *e.g*. [, , , , , ] and use 25 meters radius as the threshold for correct localization and report Recall@K metrics for K=1,5,10. For Nordland we evaluate $\pm 10$ frames as the common evaluation protocol used in [, ]. For a more comprehensive description of all 20 datasets and implementation details see Appx.

### 4.1 Zero-shot performance

**Table 1: Comparison on Zero-Shot with R@1.**
|  | Pitts30k | Tokyo24/7 | MSLS-Val | Nordland |
| --- | --- | --- | --- | --- |
| DINOv2[] | 78.1 | 62.2 | 47.7 | 33.0 |
| Anyloc[] | 87.7 | 60.6 | 68.7 | 16.1 |
| EffoVPR-ZS | 89.4 | 90.8 | 70.3 | 57.9 |

We present the performance of our zero-shot method (EffoVPR-ZS) in Table 1 compared to two zero-shot alternatives, the recently published AnyLoc, and DINOv2 global feature (using the output [CLS] token), without finetuning, where EffoVPR-ZS re-ranks its top-100 global retrieved candidates. The results show that our method significantly improves over the baseline and is superior to AnyLoc. Note the significant gap for more challenging scenarios of Tokyo24/7 and Nordland exhibiting day vs. night and seasonal variations.
AnyLoc tends to fail in these challegning scenarios as its VLAD aggregation learned (in unsupervised manner) on the gallery can not generalize well to challenging out-of-distribution queries.
In Figure 3(a) we compare our zero-shot approach with several methods that have used VPR datasets for training. Although EffoVPR-ZS was not trained on VPR task, it still achieves comparable results to the trained methods on three popular datasets. This success can be attributed to the robust features in DINOv2, specifically those selected from the $\mathcal{V}$ facet, combined with our mutual-NN matching and scoring. Figure 3(b) demonstrates this by showcasing a scenario where the original attention, mistakenly focusing on an advertisement placed in front of a building. However, our method successfully identifies relevant key-points on the building itself, enabling correct image matching (even though there is a different ad on the gallery image).

Figure: (a)
Refer to caption: x4.png

### 4.2 Comparison with State-of-The-Art

In this section, we compare our single stage (EffoVPR-G) and two-stage methods (EffoVPR-R) with previous state-of-the-art including the recent works of [, , ].
SelaVPR and R2Former were trained on a combination of Pitts30k and MSLS while CricaVPR and SALAD were trained on GSV-Cities, and Cosplace and EigenPlaces on SF-XL (similar to ours). We show in Table 2 the global retrieval results (without re-ranking) with Recall@1 on five different benchmarks. EffoVPR-G achieves SoTA performance on three out of five datasets, while being ranked second on the other two. This highlights the effectiveness of the single global representation learned by our method. Notably, it achieves $+2.9\%$ on Tokyo24/7, $+2.8\%$ on the challenging Nordland dataset that exhibits extreme seasonal changes, and $+3.2\%$ on the hold-out MSLS-challenge dataset. We further demonstrate the performance of our global feature learning with reduced dimensions of 256D and even 128D, significantly decreasing the memory footprint and enabling efficient searches within a considerably larger gallery. The findings indicate only a marginal degradation in performance with lower-dimensional features, while achieving parity with the SALAD on Tokyo24/7 using 128D, compared to 8,448D, feature size (a 66-fold reduction in dimensionality). Figure 1 illustrates this quality on Tokyo24/7 and the hold-out MSLS-challenge, showing our top performing results even with 128D feature size. Results on more datasets can be found in the Appendix.
The strong impact of our finetuning process including the last five layers, is visualized in Figure 4.

**Table 2: Comparison with our single stage method - Recall@1 performance. Two-stage methods are marked with †, and present 1st-stage performance (for fair comparison). The best results are highlighted in bold and the second is underlined. We present results from EffoVPR with three different feature dimensions.**
| Method | Dim | Pitts30k | Tokyo24/7 | MSLS-val | MSLS-chall. | Nordland |
| --- | --- | --- | --- | --- | --- | --- |
| CosPlace [] | 512 | 90.5 | 81.9 | 82.8 | 61.4 | 66.5 |
| MixVPR [] | 4096 | 91.5 | 86.7 | 88.2 | 64.0 | 58.4 |
| R2Former []^† | 256 | 76.3 | 45.7 | 79.3 | 56.2 | 50.9 |
| EigenPlaces [] | 2048 | 92.5 | 93.0 | 89.1 | 67.4 | 71.2 |
| SelaVPR^† [] | 1024 | 90.2 | 81.9 | 87.7 | 69.6 | 72.3 |
| CricaVPR [] | 4096 | 94.9^* | 93.0 | 90.0 | 69.0 | 90.7 |
| SALAD [] | 8448 | 92.4 | 94.6 | 92.2 | 75.0 | 76.0 |
| EffoVPR-G^† | 1024 | 94.8 | 97.5 | 90.9 | 78.2 | 93.5 |
| EffoVPR-G^† | 256 | 93.8 | 95.9 | 90.4 | 75.6 | 79.7 |
| EffoVPR-G^† | 128 | 92.6 | 94.6 | 88.2 | 73.8 | 70.4 |

Figure: (a)
Refer to caption: extracted/5625734/imgs/img-sf-occ-5.jpg

Next, we showcase the comprehensive performance of our two-stage approach (EffoVPR-R) in Table 3. EffoVPR-R achieves top performance across all datasets, taking the second place only on Pitts30K-R@1, with very close result. Note that CricaVPR reports using Pitts30k as a validation set, which may have contributed to the improved results on this dataset. However, EffoVPR-R demonstrates notable improvements, particularly evident in Tokyo24/7, where it achieves a remarkable increase in R@1 from 94.6% to 98.7% and in MSLS-challenge (from to 75.0% to 79.0%). These results underscore the generalization capability of our approach, demonstrating its resilience in handling significant variations between query and gallery images, such as viewpoint discrepancies (as seen in Pitts30k) and changes in illumination (in Tokyo24/7), across a diverse range of locations.
Following the common practice, we report EffoVPR-R re-ranking performance over the top-100 candidates retrieved in the first-stage ($K=100$), however we achieve SoTA R@1 results even with a low number of candidates (from $K=5$ onwards, see Tab. S4 in Appx).
Note that EffoVPR matching method is highly expedient, with an average processing time of just 1 millisecond per match.

**Table 3: Comparison to state-of-the-art methods on four benchmarks. The bests results are highlighted in bold and the second is underlined. Two-stage methods are marked with †.**
| Method | Pitts30k | Tokyo24/7 | MSLS-val | MSLS-challenge |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |  |
| NetVLAD [] | 81.9 | 91.2 | 93.7 | 60.6 | 68.9 | 74.6 | 53.1 | 66.5 | 71.1 | 35.1 | 47.4 | 51.7 |
| SFRS [] | 89.4 | 94.7 | 95.9 | 81.0 | 88.3 | 92.4 | 69.2 | 80.3 | 83.1 | 41.6 | 52.0 | 56.3 |
| Patch-NetVLAD^† [] | 88.7 | 94.5 | 95.9 | 86.0 | 88.6 | 90.5 | 79.5 | 86.2 | 87.7 | 48.1 | 57.6 | 60.5 |
| CosPlace [] | 88.4 | 94.5 | 95.7 | 81.9 | 90.2 | 92.7 | 82.8 | 89.7 | 92.0 | 61.4 | 72.0 | 76.6 |
| MixVPR [] | 91.5 | 95.5 | 96.4 | 86.7 | 92.1 | 94.0 | 88.2 | 93.1 | 94.3 | 64.0 | 75.9 | 80.6 |
| R2Former^† [] | 91.1 | 95.2 | 96.3 | 88.6 | 91.4 | 91.7 | 89.7 | 95.0 | 96.2 | 73.0 | 85.9 | 88.8 |
| EigenPlaces [] | 92.5 | 96.8 | 97.6 | 93.0 | 96.2 | 97.5 | 89.1 | 93.8 | 95.0 | 67.4 | 77.1 | 81.7 |
| SelaVPR^† [] | 92.8 | 96.8 | 97.7 | 94.0 | 96.8 | 97.5 | 90.8 | 96.4 | 97.2 | 73.5 | 87.5 | 90.6 |
| CricaVPR [] | 94.9 | 97.3 | 98.2 | 93.0 | 97.5 | 98.1 | 90.0 | 95.4 | 96.4 | 69.0 | 82.1 | 85.7 |
| SALAD [] | 92.4 | 96.3 | 97.4 | 94.6 | 97.5 | 97.8 | 92.2 | 96.2 | 97.0 | 75.0 | 88.8 | 91.3 |
| EffoVPR-R^† | 93.9 | 97.4 | 98.5 | 98.7 | 98.7 | 98.7 | 92.8 | 97.2 | 97.4 | 79.0 | 89.0 | 91.6 |

In Figure 5, we present a failure case of our zero-shot approach, which is resolved after fine-tuning. In this instance, both the query and gallery contain a visually identical vehicle (an SF cable car), which leads to incorrect matching. Although such instances are rare in general case in context of pedestrians or vehicles in the images, where the objects are commonly not identical, this example highlights a limitation of our zero-shot approach.

Figure: (a)
Refer to caption: extracted/5625734/imgs/rerank-zs-pair-1-sf-occ-38.jpg

Finally, we demonstrate the efficacy of our approach in the most challenging VPR benchmark scenarios by conducting experiments on six demanding datasets: Nordland [], which includes extensive seasonal changes; AmsterTime [], spanning over an extended time period; SF-Occlusion[], that features queries with significant field-of-view obstructions; SF-Night[], with severe illumination changes; and SVOX [], with extreme weather and illumination variations. The results, detailed in Table 4, underscore the significant superiority of our method over previous approaches across these datasets. EffoVPR-R shows improvements of +4.3%, +0.8%, +7.9%, and +15%, +2% on Nordland, AmsterTime, SF-Occlusion, SF-Night, and SVOX-Night respectively and comparable results on SVOX-Rain. This demonstrates the high versatility of our model, which can handle extreme variations even when trained without seasonal or day-to-night changes. Figure 1 shows an examples of this case. We attribute this robustness primarily to the combination of our training method and specific re-ranking strategy over the DINOv2 model. We conduct an extensive ablation study on various hyperparameters and aspects of our approach in the Appendix.

**Table 4: Comparison (R@1) to SoTA methods on more challenging datasets.**
| Method | Nordland | Amster<br>Time | SF-XL<br>Occlusion | SF-XL<br>Night | SVOX<br>Night | SVOX<br>Rain |
| --- | --- | --- | --- | --- | --- | --- |
| EigenPlaces [] | 71.2 | 48.9 | 32.9 | 23.6 | 58.9 | 90.0 |
| SelaVPR [] | 85.2 | 55.2 | 35.5 | 38.4 | 89.4 | 94.7 |
| CircaVPR [] | 90.7 | 64.7 | 42.1 | 35.4 | 85.1 | 95.0 |
| SALAD [] | 76.0 | 58.8 | 51.3 | 46.6 | 95.4 | 98.5 |
| EffoVPR-R | 95.0 | 65.5 | 59.2 | 61.6 | 97.4 | 98.3 |

Figure 6 qualitatively highlights the superior performance of our method.
While other methods fail in challenging scenarios, such as viewpoint changes, seasonal variations, illumination differences, and severe occlusions, EffoVPR demonstrates high robustness against these challenges.

Figure: Figure 6: Qualitative comparison to SoTA Methods with challenging examples.
Refer to caption: extracted/5625734/imgs/comparison.jpg

## 5 Summary and Limitations

In this paper, we introduced a single and two-stage approach for VPR, that effectively leverages a foundation model. Our method utilizes existing internal self-attention and pooling mechanisms to propose a new approach that achieves high performance even in a zero-shot setting.

We observed that despite the success of our zero-shot approach, it does not grasps the relevance of certain objects in the scene, for localization. This limitation is highlighted in Figure 5, where the iconic and visually identical cable car in SF causes distraction.
Fine-tuning the model resolves this problem by shifting the attention from transient foreground objects to static VPR-relevant cues, as seen in Figure 4. Nevertheless, while our second-stage matching approach proves highly effective, we forgo geometric verification for sake of speed. Integrating such approach, in the future may offer further refinement and performance enhancement.

The experimental results demonstrated that our trained model outperforms previous SoTA often by a large margin, particularly in demanding scenarios that exhibit strong appearance change. Having compact features, our method provides a promising way to address the VPR task in real-world, large-scale applications.

## Appendix A Datasets

We evaluate EffoVPR performance across a large number of datasets, to underscore its top-performance in variable scenarios and cities.
Following prior work, we have used [] open-source code for downloading and organizing datasets, to ensure maximum reproducibility. In the following we shortly describe each of the datasets.

### A.1 Datasets Summary

AmsterTime[] consists of $1,231$ image pairs from Amsterdam, Holland, exhibiting long-term changes.
The queries are historical grayscale images, where for each query there is a reference of a modern-day photo which represents the same place.
The pairs curated by human experts, and provide multiple challenges over different viewpoints and cameras, color vs grayscale and long-term changes.

Eynsham []
is a collection of a car street-view camera, capturing photos around the same route of Oxford countryside twice. The grayscale images are divided to $23,935$ queries and $23,935$ gallery.

Mapillary Street-Level Sequences (MSLS) [] is image and sequence-based VPR dataset. The dataset consists of more than 1.6M geo-tagged images collected during over seven years from 30 cities, in urban, suburban, and natural environments. There are 3 non-overlap subsets - a training set, validation (MSLS-val),
and withheld test (MSLS-challenge).
MSLS-val and MSLS-challenge provide various challenges, including viewpoint variations, long-term changes, and illumination and seasonal changes.

Nordland [] was collected by a mounted camera on the top of a riding train in the Norwegian countryside, presenting rural and natural scenes. The data collected over the same route across four seasons, providing seasonal and illumination variability.
Following [, ] we use the post-processed versions of winter as queries and summer as database, determining correct localization by retrieval of an image that is in less than 10 frames away. This dataset consists of $27,592$ query images and $27,592$ gallery images

Pittsburgh30k [] is collected from Google Street View 360° panoramas of downtown Pittsburgh, split into multiple images. Ensuring queries and gallery were taken in different years, it provides 3 splits - a training set, validation and test. Pitts30k-test consists of 10k gallery images and 6816 queries. Pitts250k consists of $8280$ queries including these of Pitts30k, and its gallery size is $83,952$.

San Francisco Landmark (SF-R) [] is a dataset from downtown San Francisco, which provides viewpoint variations. It presents a collection of $598$ of smartphone camera queries and gallery of $1,046,587$ images.

San Francisco eXtra Large (SF-XL) [, ] is an enormous dataset covering the whole city of San Francisco. it consists of a training set, which includes also raw 360° panoramas, a small validation set of $7,983$ queries and $8,015$ gallery images, and a test gallery of $2,805,840$ images.
There are four sets of queries:
SF-XL-v1[] consists of $1,000$ queries curated from Flickr, and provides viewpoint and camera variations, illumination changes and even some occlusions.
SF-XL-v2[] is the queries of San Francisco Landmark (SF-R).
SF-XL-Night[] is a collection of $466$ Flickr images of night scenes from San-Francisco. It provides viewpoint variations and very-challenging illumination changes.
SF-XL-Occlusion[] is a collection of $76$ Flick images from the city of San Francisco, which suffers from severe occlusions, mostly by vehicles and crowd.

SPED[] is a collection of surveillance cameras images consists of $607$ pairs of queries and gallery, captured accros time. It provides challenging viewpoint with seasonal and illumination changes.

St Lucia[]
is a collection of a nine videos of car-mounted camera from the St Lucia suburb of Brisbane. Following [] open-source code, we select the first and last videos as queries and database, and sample one frame every 5 meters of driving. The gallery consists of $1,549$ images and there are $1464$ query images.

SVOX[]
is a dataset which presents multiple weather conditions VPR challenge. It consists of $17,166$ gallery images, of the city of Oxford. The queries were collected from the Oxford RobotCar dataset [], providing multiple weather conditions queries sets, such as night (823 queries), overcast (872 queries), rainy (937 queries), snowy (870 queries) and sunny (854 queries).

Tokyo 24/7 [] is a dataset from downtown Tokyo, which provides viewpoint changes and challenging illumination variations.
It consists of a gallery of $75,984$ images, and a collection of $315$ smartphone camera queries from $185$ places. Each place is portrayed by three photos - one taken during the day, one at sunset and one at night.

### A.2 Train Dataset

Following VPR classification methods, Eigneplaces and CosPlace, we train on SF-XL[] while other studies[, , , ] train on GSV-Cities[] or combinations of Pittsburgh30k[] and MSLS[, , , ], including a large mixture of different cities around the world (introducing higher variability).
Note that similar to EigenPlace, our approach is designed for training on panoramas with heading information, and requires slicing them for lateral and frontal views, which cannot be applied other training datasets.

## Appendix B Ablation Study

Here we conduct extensive experiments on two different datasets to ablate over several key-components of our EffoVPR method.

Re-ranking features: We explore various configurations for selecting features in the re-ranking stage. Our initial focus is on the choice of the layer from which features are extracted. Table S1 demonstrates that extracting features from the $n-1$ layer yields the most significant enhancement in overall performance. Generally, employing re-ranking with any layer, except of the last layer, improves results compared to omitting re-ranking entirely (*i.e*., relying solely on the global feature from the first stage). Subsequently, upon extracting the Q, K, V components from the chosen layer, we find that the Value set ($\mathcal{V}$) represents the most effective local features for re-ranking, as detailed in Table S3. We ablate in Table S3 the impact of our two thresholds, the Attention Map threshold $T_{1}$ and the threshold on the countable local feature matching score threshold $T_{2}$, showing their necessity.

**Table S 1: Ablation study on the choice of the layer for the re-ranking stage. We find the $n-1$ layer to be the optimal for re-ranking feature extraction. Notably, the last layer $n$ is ineffective and downgrades global performance. Results (in %) are the R@1.**
| Dataset | Global | n-5 | n-4 | n-3 | n-2 | n-1 | n |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MSLS-val | 90.9 | 90.3 | 90.9 | 92.0 | 92.3 | 92.8 | 88.2 |
| Tokyo-24/7 | 97.5 | 98.1 | 98.1 | 98.1 | 98.7 | 98.7 | 97.1 |

**Table S 2: Ablation study on the impact of the thresholds. Results (in %) are the R@1. $T_{1}$ is the Attention Map threshold and $T_{2}$ is the threshold on the countable local features matching score**
|  | Tokyo24/7 | MSLS-val |
| --- | --- | --- |
| no thr. | 95.9 | 86.4 |
| $T_{1}$ | 97.1 | 91.5 |
| $+T_{2}$ | 98.7 | 92.8 |

Number of Candidates to Re-rank: The second re-ranking stage is applied to the top-K candidates retrieved during the global stage. Although common choice in literature is $K=100$ (*e.g*. [, , ]), we explore different choices of K, as detailed in Table S4. We achieve SoTA results even with $K=5$. It is important to note that in some cases, an increase in K can introduce a greater number of “distractor” candidates, potentially leading to a decrease in performance. However, EffoVPR SoTA performance is consistent for all tested K’s.

**Table S 4: Re-ranking ablation. $K$ indicates re-ranking over top-$K$ results. We achieve SoTA results even with $K=5$. Bold values indicate SoTA results.**
| Top-K | Pitts30k | Tokyo24/7 | MSLS-val | Nordland | SF-XL-Occ. | SF-XL-Night | SPED |
| --- | --- | --- | --- | --- | --- | --- | --- |
| K=5 | 94.2 | 97.8 | 92.4 | 95.3 | 59.2 | 61.6 | 93.4 |
| K=10 | 94.2 | 98.1 | 92.2 | 95.3 | 59.2 | 61.2 | 92.9 |
| K=15 | 94.1 | 98.4 | 92.3 | 95.3 | 60.5 | 60.3 | 93.1 |
| K=20 | 94.0 | 98.4 | 92.4 | 95.3 | 59.2 | 60.3 | 93.1 |
| K=50 | 93.9 | 98.7 | 92.7 | 95.2 | 57.9 | 60.9 | 93.2 |
| K=100 | 93.9 | 98.7 | 92.8 | 95.0 | 59.2 | 61.2 | 93.2 |

Choice of Trainable Layers: Table S5 presents a few different sets of trainable layers in our backbone model. We find the vanilla fine-tuning of the entire model, end-to-end, that includes all layers, drastically harms the performance of EffoVPR. We attribute this decline to the fact that the DINOv2 backbone was trained on significantly larger datasets compared to those typically used in VPR. Subsequently, we establish that training only the last five layers represents a “sweet-spot”, yielding peak performance. Both increasing or decreasing the number of trainable layers from this configuration leads to lower results.

**Table S 5: Ablation study on the choice of trainable last layers. Results (in %) are the R@1 of the 1st stage. Note that the 0 column represent a zero-shot performance of DinoV2.**
| Dataset | 0 | 1 | 2 | 3 | 4 | 5 | 6 | all layers |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSLS-val | 47.7 | 89.5 | 88.2 | 89.7 | 89.5 | 90.9 | 89.7 | 86.1 |
| Tokyo-24/7 | 62.2 | 96.8 | 96.5 | 95.9 | 96.2 | 97.5 | 96.8 | 94.0 |

## Appendix C Performance vs. Feature Compactness - Additional Results

In Figure S1 we present performance comparison of our global feature (EffoVPR-G) versus feature dimensionality for more datasets. While the current leading methods achieve their performance using large features, EffoVPR demonstrates high performance even with an extremely compact feature size.

Figure: Fig. S 1: Recall@1 performance of EffoVPR-G global feature versus feature dimensionality for more datasets.
Refer to caption: x5.png

## Appendix D Visualizations

### D.1 Additional Zero-Shot Visualizations

In Figure S2 we show more visualizations of EffoVPR-ZS method. While the attention map of pre-trained DINOv2 doesn’t focus on
discriminative VPR elements, EffoVPR is able to fill the gap in zero-shot with local features matching.
In the first row The pre-trained attention-map is mainly focused on temporal traffic signs and a far ad
and almost not attend the building, and in the second row it is mainly focused on an insignificant back of a traffic sign. However EffoVPR method finds multiple local matches to the right geo-tagged image in the gallery.

Figure: (a)
Refer to caption: extracted/5625734/imgs/attn_zs-tokyo-31.jpg

### D.2 Additional Re-ranking Visualizations

Figure S3 exhibits EffoVPR-R local features matching invariability to highly challenging scenes with top-1 results. From the top left to the bottom right - to camera rotation, a nature scene, color variance across time (building renovation), tree matching, challenging day-time change with hardly noticed electric cables matching, night to day significant change.

Figure: Fig. S 3: EffoVPR-R top-1 matching visualizations. For each pair matching, the left image is the query and the right is the top-1 result.
Refer to caption: extracted/5625734/imgs/rerank-trained-pair-1-msls-5.jpg

## Appendix E Implementation details

We use ViT-L/14 as the backbone, initialized with pre-trained weights of DINOv2 with registers[, ]. We only train the last five layers of the backbone, which appeared to be most beneficial. We employ EigenPlaces’s[] group and class partitioning with its default hyper-parameters, and both lateral and frontal views, on the publicly available SF-XL street-view panoramas dataset []. We set an AdamW optimizer to the backbone, and an Adam optimizer to the classification heads, both with a constant learning rate of $1\times 10^{-5}$. We train EffoVPR with a batch size of 16, for 25 epochs, on a single NVIDIA-A100 node. We otherwise follow EigenPlaces training recipe. We choose the best epoch by SF-XL validation set, measuring Recall@1 global ranking performance. Given that ViT is independent of the image input size (provided it can be segmented into $14\times 14$ patches), we evaluated using images sized $504\times 504$, but trained on 224 × 224 images to expedite training. For benchmarking EffoVPR-G of the global feature, we report nearest-neighbors performance on normalized output class token. In the re-ranking stage we extract the $V$ self-attention facet from layer $n-1$ (with $n$ being the output layer), measure cosine similarity, we filter the features by class attention map with a threshold $\mathcal{T}_{1}=0.05$, and count only mutual nearest-neighbors with a score above the threshold $\mathcal{T}_{2}=0.65$.

### E.1 Additional information

#### E.1.1 Zero-shot

In evaluating AnyLoc[], we tackle the significant memory requirements of its VLAD pooling algorithm by implementing an online clustering scheme. We observed that their recommendation for layer 31 outperformed layer 23. In our zero-shot evaluation, we assess EffoVPR-ZS method by extracting the $V$ features from layer $n-2$ to re-rank the top-100 candidates retrieved from the first-stage global [CLS] feature. In this framework, our performance is constrained by the first-stage Recall@100, achieving rates of $99.2\%$, $96.8\%$, $81.5\%$, and $78.1\%$ on Pitts30k, Tokyo24/7, MSLS-Val, and Nordland, respectively.

#### E.1.2 Benchmarking

Generally, for consistent benchmarking, we adhere to []. In addition, we report the results of other methods in accordance with the evaluation choices of SelaVPR[] and CricaVPR[], including the specific versions of trained models utilized. For the recent state-of-the-art methods SelaVPR, CricaVPR, and SALAD, we provide results from the original publications whenever available. When such results are not directly available, we utilize their code and published weights. Specifically for SelaVPR, which has two sets of weights (trained on Pitts30k and MSLS), we report the best-performing for each dataset.

### E.2 Other

We evaluate EffoVPR matching runtime by averaging matching function runtime on Tokyo 24/7.

## Appendix F Additional Quantitative Results

To ensure comprehensiveness, the following Table S6 presents the complete results for datasets that were only partially presented in the main paper, as well as for some datasets that were previously omitted. Our method, EffoVPR, demonstrates SoTA performance on the majority of these datasets, and remains competitive with the SoTA on others.

**Table S 6: Comparison to SoTA on more datasets**
|  | SPED | SF-R | SF-XL-v1 | SF-XL-v2 | SF-XL-Occ. |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |
| EigenPlaces | 70.2 | 83.5 | 87.5 | 89.6 | 94.3 | 95.3 | 84.1 | 89.1 | 90.7 | 90.8 | 95.7 | 96.7 | 32.9 | 48.7 | 52.6 |
| SelaVPR | 88.6 | 95.1 | 97.2 | 88.5 | 92.0 | 93.0 | 74.9 | 80.7 | 82.1 | 89.3 | 95.7 | 96.3 | 35.5 | 47.4 | 55.3 |
| CricaVPR | 91.3 | 95.2 | 96.2 | 88.6 | 94.0 | 95.7 | 80.6 | 87.6 | 89.8 | 90.6 | 96.3 | 97.7 | 42.1 | 52.6 | 57.9 |
| SALAD | 92.1 | 96.2 | 96.5 | 92.3 | 95.7 | 96.8 | 88.6 | 93.5 | 94.4 | 94.8 | 97.3 | 98.3 | 51.3 | 65.8 | 68.4 |
| EffoVPR | 93.1 | 97.9 | 98.4 | 93.0 | 96.0 | 96.3 | 95.5 | 98.1 | 98.3 | 94.5 | 97.8 | 98.2 | 59.2 | 68.4 | 73.7 |

|  | SF-XL-Night | Amster Time | SVOX | SVOX Night | SVOX Overcast |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |
| EigenPlaces | 23.6 | 30.7 | 34.5 | 48.8 | 69.5 | 76.0 | 98.0 | 99.0 | 99.2 | 58.9 | 76.9 | 82.6 | 93.1 | 97.8 | 98.3 |
| SelaVPR | 38.4 | 50.9 | 55.4 | 55.2 | 72.6 | 78.0 | 97.2 | 98.7 | 99.0 | 89.4 | 95.5 | 96.6 | 97.0 | 99.1 | 99.3 |
| CricaVPR | 35.4 | 48.3 | 53.4 | 64.7 | 82.5 | 87.9 | 97.8 | 99.2 | 99.3 | 86.3 | 95.3 | 96.6 | 96.7 | 99.0 | 99.0 |
| SALAD | 46.6 | 59.0 | 62.2 | 58.8 | 78.9 | 84.2 | 98.2 | 99.3 | 99.4 | 95.4 | 99.3 | 99.4 | 98.3 | 99.3 | 99.3 |
| EffoVPR | 61.6 | 73.4 | 77.0 | 65.5 | 87.2 | 90.7 | 98.7 | 99.5 | 99.6 | 97.4 | 99.5 | 99.5 | 98.4 | 99.3 | 99.7 |

|  | SVOX Rain | SVOX Snow | SVOX Sun | Sr. Lucia | Eynsham |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |
| EigenPlaces | 90.0 | 96.4 | 98.0 | 93.1 | 97.6 | 98.2 | 86.4 | 95.0 | 96.4 | 99.6 | 99.9 | 100.0 | 90.7 | 94.4 | 95.4 |
| SelaVPR | 94.7 | 98.5 | 99.1 | 97.0 | 99.5 | 99.5 | 90.2 | 96.6 | 97.4 | 99.8 | 100.0 | 100.0 | 90.6 | 95.3 | 96.2 |
| CricaVPR | 94.8 | 98.5 | 98.7 | 96.0 | 99.2 | 99.2 | 93.8 | 98.1 | 98.8 | 99.9 | 99.9 | 99.9 | 91.6 | 95.0 | 95.8 |
| SALAD | 98.5 | 99.7 | 99.9 | 98.9 | 99.7 | 99.8 | 97.2 | 99.4 | 99.7 | 100.0 | 100.0 | 100.0 | 91.6 | 95.1 | 95.9 |
| EffoVPR | 98.3 | 99.6 | 99.6 | 98.7 | 99.7 | 99.7 | 97.7 | 99.3 | 99.4 | 100.0 | 100.0 | 100.0 | 91.0 | 95.2 | 96.3 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weakness section.
soundness: 3
strengths: 1. The proposed method is easy to follow. It leverages the foundation model’s feature representation capability to improve the effectiveness of image retrieval and re-ranking. 

2. The proposed method outperforms some existing approaches on several VPR benchmarks.
summary: This paper presents an effective approach for visual place recognition (VPR) using a foundation model (DINOv2). The features extracted from the self-attention layers are used as re-ranker for VPR. Experiments on several datasets show the proposed method is robust and effective.
weaknesses: 1. Since the proposed method mainly leverages the strong representation power of a foundation model (DINOv2), the technical novelty of the proposed method in VPR is somewhat incremental. 
2. Certain design choices, such as using the V_l matrix for keypoint descriptors, lack clear motivation and justification, particularly in explaining why such choices are effective.
3. Although the proposed method outperforms several existing methods, the improvement could come solely from using a stronger foundation model feature representation. Therefore, the comparison may not be fair. 
4. The computational complexity and memory footprint of the proposed method should be discussed and compared with other existing methods. 
5. It would be good to also show some failure cases and provide analysis.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: see weaknesses.
soundness: 3
strengths: 1.	The methods proposed in the paper are generally novel.
2.	The experiments provided in the paper are sufficient. The author provides the results of other SOTA methods on datasets such as SF-XL. As far as I know, completing such evaluation experiments requires a lot of time and effort.
3.	The performance of the proposed method is really good. It achieves excellent results even with compact global features.
summary: This paper proposes a new method, called EffoVPR, to effectively use vision foundation models for visual place recognition. This method leverages the output class token of the DINOv2 model to generate global features to retrieve candidates. It also uses ViT internal attention matrices to obtain local features for re-ranking these candidates. EffoVPR achieves good zero-shot performance. After training on the SF-XL dataset with the EigenPlaces framework, it outperforms previous SOTA methods on several VPR datasets.
weaknesses: 1.	The methodological contribution of this paper is a little weak. Its training strategy follows the EigenPlaces work. The global feature is the class token (following a linear layer) of the ViT model. The local matching in the re-ranking process is based on the mutual nearest neighbors searching, and it also already exists. I think the authors should state that the previous SelaVPR work also does not require spatial verification in re-ranking. The biggest contribution of this work seems to be the use of the internal self-attention matrices to yield selected local features, which is meaningful but maybe not enough.
2.	The proposed method is trained on SF-XL, which is a very large dataset and time-consuming to train. If training on other datasets, such as GSV-Cities or even MSLS/Pitts30k. Can it still achieve such good results? If the proposed method cannot extend to other training sets, it is better to have an appropriate explanation in the paper.
3.	The proposed method uses 224×224-pixel images for training, while 504×504-pixel images for evaluation. Using the DINOv2-large model to process such high-resolution images is time-consuming and memory-consuming. Although the paper gives results using low-resolution images on MSLS and Tokyo247, I think it is also necessary to give a comparison of GPU memory usage, and better to provide results on more datasets (e.g., Nordland and SF-XL-Night).

Overall, the strengths of this paper outweigh its weaknesses.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - Q1: The paper shows that EffoVPR-R achieves SOTA results even with K=5. I am curious, how much does this change for the zero-shot setting. I.e., when the last layers of DINOv2 are not finetuned, is it also sufficient to use K=5, or would K need to be increased?

- Q2: It should be clearly stated that the ablations in the appendix we done using the EffoVPR-R. Would it be possible to confirm this?

- Q3: The re-ranking stage requires storing local features for the re-ranked images. It would be good to add discussion and analysis of memory requirements for the proposed approach and in particular the memory requirements to store the local features for spatial re-reranking.
soundness: 3
strengths: - S1: The paper is clear and easy to read.

- S2: Simple, yet effective and robust method that improves the performance over the DINOv2 baseline and other state-of-the-art methods, while reducing the required dimensionality of the used features.

- S3: The method finetunes DINOv2 to achieve better performance but is still effective even in a zero-shot setting.
summary: The paper introduces a method that uses DINOv2 foundation model to address the Visual Place Recognition task. This is achieved by retrieving the top-K most similar candidates in the first stage by measuring the similarity of (finetuned) DINOv2’s [CLS] tokens between the query and the candidate images. In the second stage, the proposed approach re-ranks the retrieved candidates using local features computed using the Queries, Keys and Values from the self-attention module of an intermediate DINOv2 layer. The method outperforms other state-of-the-art zero-shot approaches, as well as supervised methods when the last layers of the DINOv2 network are finetuned. Moreover, the proposed method achieves results competitive with the supervised methods even when using only the first stage of the method.
weaknesses: - W1 Even though the results of the three proposed variants of the method (EffoVPR-ZS, EffoVPR-G and EffoVPR-R) are provided, the paper is missing their comparison in a single table, to directly see how large is the effect of finetuning and re-ranking. It would be good to see this comparison in a single table to understand (and compare) the direct benefits of the different components of the proposed approach. 

- W2 The notation for Queries, Keys and Values and corresponding local features would be more readable if the layer index would be consistently used always as superscript or always as subscript ($V_l$ versus $v^l_i$). Alternatively, since the Queries, Keys and Values are always taken from the same layer, maybe the layer index could be completely omitted for better readability.

- W3 What are the failure cases and limitations of the main presented method, EffoVPR-R? The paper contains only failure cases for the zero-shot setting.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: The paper mentions time evaluation, but only shows time evaluation on the first step and not the reranking.  From the look of the model, the time of reranking doesn’t seem to be particularly time efficient.  How fast is the reranking?  

The paper mentioned memory efficiency and reduced memory footprint.  Quantitatively, how much memory consumption does the modal overall take compared to the other models? 

On the ablation on T1 and T2, table only shows the result with different T1 and different T2, but didn’t specify what the other filter value is (ex. On table S5, When showing performance when T1 equals 0.05 and 0.10, it didn’t specify what T2 is when evaluating).  What is the value of T2 when evaluating for table S5 and what is the value of T1 when evaluating for table S6? 

How well does the model do on the Pittsburgh 250k dataset? 

Are there experimental results of EffoVPR trained on other datasets other than SFXL? 

In the Eigenplaces paper which the training process of EffoVPR seems to be heavily based on, the backbone was completely trained.  How well would the model perform if every layer of the backbone was trained? 

Paper mentions that only the “final layers” are trained.  What are the final layers that are trained? 

Paper calls itself a “foundational model”, which should be able to be used for multiple tasks, yet only same view matching capability is shown in the paper.  Can the model be used for other tasks like cross view matching or detection? 

Could the global descriptor be used also during the reranking phase to further filter out the nearest neighbors, perhaps to only neighbors that belong in the same predicted cell? 

How well does the model perform with anyloc as backbone rather than DINOV2?
soundness: 3
strengths: Performance: 
EffoVPR achieves SOTA performance by performing better than previous recent SOTA models such as SALAD and EigenPlaces.  In particular, EffoVPR performs significantly better on the more challenging datasets  
EffoVPR is able to fine tune on cities in SF XL but then perform very well on Nordland which is mainly rural and natural scenes taken from a train. 
EffoVPR has very impressive zero shot performance compared to other zero shot methods and some trained models, showing effectiveness of reranking model 
EffoVPR with only the first stage also performs very well  
Architecture: 
The architecture for EffoVPR is intuitive and simple to understand 
Efficiency: 
EffoVPR utilizing only reranking during evaluation makes training efficient and fast, as the reranking does not need to be trained 
Experimentation: 
Paper presents very detailed ablation studies on the choice of layer for reranking, the different thresholds, which layer to choose reranking features for, etc.   
Papers shows the model being performed on a wide range of datasets.
summary: The paper presents a model for visual place recognition, specifically aimed at achieving robust zero-shot retrieval performance. The model uses a Vision Transformer (ViT) with DINOV2 weights as its backbone to identify nearest neighbors based on global feature similarity, followed by a mutual nearest neighbor (NN) re-ranking strategy to pinpoint the closest reference image using local features. For training, the Eigenplace paradigm is applied to promote view invariance in the ViT. Only the final layers of the backbone are fine-tuned, with the rest of the model kept frozen. Fine-tuning was performed on the SF-XL dataset, and testing was conducted on several benchmarks: Pitts30k, Tokyo24/7, MSLS-val, and MSLS-challenge, as well as more challenging datasets, including Nordland, Amsterdam Time, SF-XL Occlusion, SF-XL Night, SVOX Night, and SVOX Rain.  For contribution, EffoVPR proposed a novel reranking architecture that does not require training and works well even with a small number of candidates.  The paper also claims to demonstrate the lack of need for aggregate methods to capture local features
weaknesses: Performance: 
Model does not perform better than other models by a significant margin on the less challenging datasets.  However, the less challenging datasets are already inferred very well by the other datasets so there’s not as much room for improvement.   
Paper is not entirely clear how much the reranking contributes to performance compared to DINOV2 
Novelty: 
In contribution, the paper claimed “We introduce a novel image matching method for VPR that relies on keypoint detection and descriptor extraction from the intermediate layers of a foundation model”.  However, R2Former also uses filtered local features from intermediate layers from the VIT backbone. Perhaps need to change wording to reflect true level of novelty. 

The paper mentions reduced memory footprint, but only shows dimension of global feature and not other aspects such as parameter size (ex. R2Former uses VIT-S whereas EffoVPR uses VIT-L but is not reflected anywhere on paper) 

Training: 
T1, T2, along with which layer to get feature map all being fixed hyperparameters means there’s more manual tuning involved to achieve optimal results

Meta-review: This paper introduces a novel method for visual place recognition (VPR) called EffoVPR, which utilizes the DINOv2 foundation model for image feature extraction. EffoVPR is claimed to achieve superior performance on several VPR benchmarks, including the SF-XL dataset, compared to previous state-of-the-art (SOTA) methods. The proposed method is evaluated under different conditions, including zero-shot settings, where it shows promising results without requiring fine-tuning.

All reviewers have given acceptance scores and unanimously agree that the responses during the rebuttal period addressed their concerns. Therefore, I ultimately recommend accepting the paper.

Decision: Accept (Poster)