Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognition
Authors: Feng Lu
Lijun Zhang
Xiangyuan Lan
Shuting Dong
Yaowei Wang
Chun Yuan
Venue: ICLR 2024 poster (2024)
Keywords: Visual Place Recognition
Parameter-efficient Transfer Learning
Global Adaptation
Local Adaptation
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recent studies show that vision models pre-trained in generic visual learning tasks with large-scale data can provide useful feature representations for a wide range of visual perception problems. However, few attempts have been made to exploit pre-trained foundation models in visual place recognition (VPR). Due to the inherent difference in training objectives and data between the tasks of model pre-training and VPR, how to bridge the gap and fully unleash the capability of pre-trained models for VPR is still a key issue to address. To this end, we propose a novel method to realize seamless adaptation of pre-trained models for VPR. Specifically, to obtain both global and local features that focus on salient landmarks for discriminating places, we design a hybrid adaptation method to achieve both global and local adaptation efficiently, in which only lightweight adapters are tuned without adjusting the pre-trained model. Besides, to guide effective adaptation, we propose a mutual nearest neighbor local feature loss, which ensures proper dense local features are produced for local matching and avoids time-consuming spatial verification in re-ranking. Experimental results show that our method outperforms the state-of-the-art methods with less training data and training time, and uses about only 3% retrieval runtime of the two-stage VPR methods with RANSAC-based spatial verification. It ranks 1st on the MSLS challenge leaderboard (at the time of submission). The code is released at https://github.com/Lu-Feng/SelaVPR.

Published as a conference paper at ICLR 2024
TOWARDS SEAMLESS ADAPTATION OF PRE-TRAINED MODELS FOR VISUAL PLACE RECOGNITION
Feng L Lijun Zhang Xiangyuan Lan Shuting Dong', Yaowei Wang", Chun Yuan
'Tsinghua Shenzhen International Graduate School, Tsinghua University Peng Cheng Laboratory University of Chinese Academy of Sciences {1 (1f220mail) 1s,, yuancesz). ts tsinghua. edu. cn lanxyêpcl.ac.cm
ABSTRACT
Recent studies studies show that vision models pre-trained in generic visual learning tasks with large-scale data can provide useful feature representations for a wide range of visual perception problems. However, few attempts have been made to ex- ploit pre-trained foundation models in visual place recognition (VPR). Due to the inherent difference in training objectives and data between the tasks of model pre-training and VPR, how to bridge the gap and fully unleash the capability of pre-trained models for VPR is still a key issue to address. To this end, we propose a novel method to realize seamless adaptation of pre-trained models for VPR. Specifically, to obtain both global and local features that focus on salient landmarks for discriminating places, design hybrid adaptation method achieve both global and local adaptation efficiently, which only lightweight adapters are tuned without adjusting the pre-trained model. Besides, to guide effective adaptation, we propose mutual nearest neighbor local feature loss, which ensures proper dense local features produced for local matching and avoids time-consuming spatial verification in re-ranking. Experimental results show that our method outperforms the sfate-of-the-art methods with less train- ing data and training time, and uses about only 3% retrieval runtime of the two- stage VPR methods with RANSAC-based spatial verification. It ranks 1st on the MSLS challenge leaderboard (at the time of submission). The code is released at btsps://github com/ Lu- Sae--Feg//eeeeee SelaVPR
INTRODUCTION
Visual place recognition (VPR), also known as image localization (Liu et 2019) visual geo- localization (Berton et al. 2022b), aims at coarsely estimating the location of a query place image by searching for its best match from a database of geo-tagged images. VPR has long been studied in robotics and computer vision communities, motivated by its wide applications in mobile robot local- ization (Xu et al. 2020) and augmented reality (Middelberg et al. 2014), etc. The main challenges of the VPR task include condition (e.g.. illumination and weather) changes, viewpoint changes, and perceptual aliasing (Lowry et al., 2015) (hard to differentiate similar images from different places).
The VPR task is typically addressed by using image retrieval and matching approaches (Arand- jelovic et al. 2016 Cao et al. 2020) with global or/and local descriptors to represent images. The aggregation algorithms like VLAD (Jégou et al. 2010 Lowry & Andreasson 2018 Khaliq et al.) 2019) are usually used to aggregate/pool local features into a vector as the global feature. Such compact global features facilitate fast place retrieval and robust against viewpoint variations. However, these global features neglect spatial information, making VPR methods based on them prone perceptual aliasing. A promising solution (Cao al.] 2020 Hausler al., [002 Wang et al. 2022a). i.c., two-stage VPR, is to retrieve top-k candidate results in the database using global features, then re-rank these candidates by matching local features. Moreover, VPR model training follows the "pre-training then finetuning" paradigm. Most VPR models are initialized using model parameters trainrd on ImageNet (Deng et al., (009) and fine-tuned on the VPR datasets, such as MSLS (Warburg et al. 2020). As models and training datasets continue to expand, the training becomes more costly both computation and memory footprint.
"Corresponding authors.

---

Published as a conference paper at ICLR 2024
Building
Tree V
Tree
Tree
Building
(a) Input image
(b) Result of pre-trained model
(c) Result of method
Figure 1: Attention map visualizations of the pre-trained foundation model (DINOv2) and our model. The pre-trained model pays attention to some regions (e.g. dynamic riders) that are use- less to identify places. Our method focuses discriminative regions (buildings and trees).
Recently, foundation models (Radford et al. 202 Yuan et al. 2021 Oquab al. 2023) achieved re- markable performance on many computer vision tasks given their ability to produce well gell-generalized representations. However, the image representation produced by the pre-trained model is susceptible to useless (even harmful) dynamic objects (c.g. pedestrians and vehicles), and tends to ignore some static discriminative backgrounds (e.g. buildings and vegetation), shown in Fig. (b). A robust VPR model should focus on the static discriminative landmarks (Chen et al. 2017b) rather than the dynamic foreground. This results in a gap between pre-training and VPR tasks. Mcanwhile, full fine-tuning the foundation model on downstream datasets might forget previously learned knowl- edge and damage the excellent transferability, i.e., catastrophic forgetting. An effective method to address this issue is parameter-effie efficient transfer learning (Houlsby et al., 2019 Lester et al. 2021), which has been studied in the VPR area. Besides, most foundation models do not directly pro- duce (dense) local features, which is typically required in the re-ranking of two-stage VPR methods.
In this paper, we propose novel method to realize Seamless adaptation of pre-trained founda- tion models for the VPR task, named SelaVPR. By adding few tunable lightweight adapters to the frozen pre-trained model, we achieve an efficient hybrid global-local adaptation to get both global features for retrieving candidate places and local features for re-ranking. Specifically, the global adaptation is achieved by adding adapters after the multi-head attention layer and paral- lel to the MLP layer in each transformer block. The local adaptation is implemented by adding up-convolutional layers after the entire transformer backbone to upsample the feature map. Addi- tionally, we propose mutual nearest neighbor local feature loss, which can be combined with the commonly used triplet loss to optimize the network. The proposed SelaVPR feature representation focuses on the discriminative landmarks, which is critical identifying places. Furthermore, we can directly match the local features without spatial verification, making the re-ranking much faster than mainstream two-stage VPR methods. Our main contributions are highlighted follows:
1) We propose a hybrid global-local adaptation method to seamlessly adapt pre-trained foundation models to produce both global and local features for the VPR task. The proposed SelaVPR feature representation can focus on discriminative landmarks and ignore the regions irrelevant to distin- guishing places, thus closing the gap between the pre-training and VPR tasks.
2) We also propose mutual nearest neighbor local feature loss to train the local adaptation module, which combined with global feature loss for fine-tuning fne-tuning. The obtained local features can be directly used in cross-matching for re-ranking, without time-consuming geometric verification.
3) Our method outperforms state-of sate-oofthhe-an methods on several VPR benchmarks (ranks 1st on MSLS challenge leaderboard) using less training data and training time. And it only consumes 3% retrieval runtime of the mainstream two-stage methods with RANSAC-based geometric verification.
RELATED WORK
Visual Place Recognition:
The traditional VPR approaches perform nearest neighbor search using global features to find the most similar place. The global features are commonly produced using aggregation algorithms, such as Bag of Words (Angeli al. 2008) and VLAD (Jégou al. 2010), to process the hand-crafted features like SURF (Bay et al. 2008). With the advancement of deep
learning techniques, many works (Sünderhauf et al. 2015 Jin Kim al. 2017 Chen al. 2017a
2

---

Published as a conference paper at ICLR 2024
Naseer et al. 2017: : Garg et al. 2017 2018 Xin al. 2019 Yin et al. 2019 Lu et al. 2021 Leyva-
Ali Bey et al. 2023) have employed a variety of deep features for the VPR task. Some works integrated the aggregation methods into neural networks (Arandjelovic al.
Vallina et al. 2021 2023|
2016
Yu et al. 2019| Peng et al. ), and improved training strategies (Gie et al. 2020] [Berton al. 2022a; Ali-bey et al. 2022), to achieve better performance. Nevertheless, most one-stage (i.e. global retrieval) VPR approaches are prone to perceptual aliasing due to the use of aggregated features while neglecting spatial information. One recent work (Keetha et al. (203)) also used pre- trained foundation models for the VPR task. However, this work did not perform any fine-tuning, making it difficult to fully unleash the capability of these models for VPR.
Recently, the two-stage (i.e. bierarchical) VPR methods (Hausler & Milford 2020 Garg & Milford
2021| Hausler al.] [202]] [Berton et al. 2021; Shen et al. [022] Wang et al. 2022a Lu et al. [023]
Shen et al.) 2023 Zhu et al.) 2023) have become popular. These approaches typically retrieved
top-k candidate images over the whole database using compact global feature representation, such
as NetVLAD (Arandjelovic et al. 2016) Generalized Mean (GeM) pooling (Radenović et al. 2018). then re-ranked candidates by performing local matching between the query image and each candidate using local descriptors. However, most of these methods required geometric consistency verification after local matching (Cao et al.) 2020 Hausler et al. [2021] Wang et al. (022a) or taking
into account spatial constraints during matching (Berton et al. 2021 Lu et al. 2023 Zhu et al., 2023), which greatly increases the runtime burden. The recent visual foundation model (Oquab) al. 2023) has shown an excellent ability to match similar semantic patch-level features across domains. In this work, we attempt to match local features produced by the foundation model for re- ranking without spatial verification, thereby significantly speeding up the retrieval process in VPR.
Parameter-efliiient Transfer Learning: Recent work (Radford et al. 2021: Caron et al. 2021
Wang et al., 2022b Oquab et al.] 2023) demonstrated the visual foundation model can produce powertul feature representation and achieve excellent performance on multiple tasks. These works commonly trained the ViT (Dosovitskiy et al. 2020) model or its variants with large quantities of pa- rameters on huge amounts of data. The parameter-efficien transfer learning (PETI.) (Houlsby et al.) 2019). first proposed in natural language processing, is an effective way to adapt foundation models to various downstream tasks, which can reduce training computation costs and avoid catastrophic forgetting. The main PETL methods fall broadly into three categories: adding task task-specific adapters Adaptation (LoRA) (Hu 2021). We follow the first in this work. Although several adapter- based approaches (Jie &
(Houlsby al. 2019), prompt tuning (Lester et al. (021), and
ct al.
Deng 2022 Chen al. 2022 Pan et al. [022] Yang et al., 2023 Khan & Fu, (2033 Xu al. 2023] 2023) have been proposed to perform various computer vision tasks, to the best of our
Park et al.
knowledge. this work is among the first to use the hybrid global-local adaptation to produce both
global features and local features, and apply them to address the challenges in VPR.
PROPOSED METHOD
This section describes the proposed SelaVPR for two-stage VPR. We first introduce ViT and its use to produce place image representation. Then, propose the global adaptation, local adaptation, and local matching re-ranking to achieve two-stage VPR. Finally, we present the loss for fine-tuning.
3.1 PRELIMINARY
The Vision Transformer (ViT) and its variants have proven to be powerful for a variety of computer vision tasks including VPR. In this work, we adapt the ViT-based pre-trained foundation model DINOv2 (Oquab et al. 2023) for VPR, so here we give a brief overview of ViT.
Given an input image, ViT first slices it into N patches and linearly projects them D-dim patch embeddings Tp E R.NDD, then prepends a learnable [class] token to Ipp as [Eidasss xpl R(++1)DD, After adding positional embeddings to preserve the positional information, xo is fed into a series of transformer blocks to produce the feature representation. A standard transformer block mainly includes multi-head attention (MHA), multi-layer perceptron (MLP), and LayerNor- malization (LN) layers, as shown in Fig. 2 (a). For the input token sequence, its change process passing through transformer block is: The MHA is first applied compute attentional features,
3

---

Published as a conference paper at ICLR 2024
Tuned
Froz
(Parallel Adapter
MLP
MI.
Layer
Layer Norm
Project
Up Project
(Ser hall
dapter
MHA
Down Project c
MLP
Rei.U
+ MHA
+
Down Project
Layer
3 Layer Norm
MHA
(a) Transfomer Block
(b) Serial Adepter
(c) Parallel Adapter
(d) Global Adeptation
Figure 2: Illustration of the global adaptation. We add serial adapter (b) after the MHA layer and
parallel adapter (c) in parallel to the MLP layer in each standard transformer block (a) to achieve global adaptation.
then MLP is utilized realize the feature nonlincarization and dimension transformation. It is formulated as:
z MHA(LN(=-1I) 21-1
3i MLP(LN(2))) + +a/,
(2)
where ø1-1 and xy are the output of the (1 - 1) and /-th transformer block.
For the feature map output by CNN models, a common practice of the two-stage VPR method is to use NeIVLAD or GeM to aggregate it into a global feature for candidate retrieval and also treat it as dense local (patch) features for re-ranking. For the VIT model, the output consists of one class token and N patch tokens, where the class token can be directly used as the global feature to represent places. Meanwhile,
patch tokens can also be reshaped as a feature map (similar to CNN) to restore spatial position. In this work, instead of using the class token the global feature, we use GeM to pool the feature map into the global feature. The reason is explained in Appendix C
3.2 GLOBAL ADAPTATION
Although pre-trained foundation models are capable of powerful feature representation, direct use of them in VPR cannot fully unleash their capability due to the gap between the pre-training and VPR tasks. To address it, we introduce the global adaptation to adapt the pre-trained model so that the feature representation can focus on the static discriminative regions that are beneficial to VPR.
Inspired by previous adapter-based parameter-efffcient fine-tuning works (Houlsby et al. 2019
Chen et al. 2022 Yang et al. 2023). we design our global adaptation as shown in Fig. 2 (d).
Specifically, we add two adapters in each transformer block. Each adapter is a bottleneck module,
which first uses the fully connected layer to down-project the input to a smaller dimension, then applies a ReLU activation and up-projects back the original dimension. The first adapter is a serial adapter that is added after the MHA layer and has skip-connection internally. The second adapter is a parallel adapter that connected in parallel to the MLP layer multiplied by scaling factor The computation of each global adapted transformer block can be denoted as
z = AdapterI (MHA(LN(zz-11) + 21-1
(3)
T MLP(LN(2())
Adapter2(LN(xd)) + 2/-
(4)
The output of the last transformer block is fed into LN layer the final output of the entire global adapted ViT backbone. We discard the class token and reshape patch tokens as the produced feature map mm. The n.2-normaizzed GeM global feature used to retrieve candidates can be written as
f L2(GeM(fm)).
(5)
The global adapted foundation model can produce feature representations that focus on discrimi- native landmarks and ignore dynamic interference. This bridges the gap between the model pre- training and VPR tasks, and greatly boosts the performance of foundation models in the VPR task.
4

---

Published as a conference paper at ICLR 2024
GeM pooling
Global features
Retrieving top-k candidates
Global
Reshape Patch Tokens
Re-ranking
Adapted
VIT Backbone
Dense local features
A query some reference images
ReLU
Local
up-conv 3x3
Adaptation
Local matching for re-ranking
Module
Figure 3: Illustration of the local adaptation and our two-stage VPR pipeline. The global adapted VIT backbone is applied extract the feature map. We first use GeM pool the feature map into the global feature for candidate retrieval. The local adaptation module after the backbone is achieved using up-conv layers, which upsample the feature map yield dense local features. Then we cross- match the local features between the query image and each candidate for re-ranking.
3.3 LOCAL ADAPTATION
Two-stage VPR methods typically match dense local features to re-rank candidate places for boost- ing performance. In the above introduction, we have obtained the feature map output by the global adapted ViT backbone, which can also be regarded as boare--raiined patch-level features. How- in order achieve promising performance improvements through (local matching) re-ranking, more fine-grand one-grained dense local features are required. To achieve this, we propose the local adapta- tion, which is achieved by up-sampling module after the VIT backbone, as shown in Fig. B To be specific, this module consists two up-convolutional (up-conv) layers and a ReLU layer in the middle. The height and width of the feature map will approximately double after passing through each up-conv layer, while the channel dimension will be reduced. Finally, for the input image of size 224x224 pixels, this module adjusts the 16x 16x 1024-dim feature map output by ViT-L/14 backbone to 61 1x61 x 128-dim, and performs L2 normalization in the channel dimension (intraL2) to yield dense local features, i.e., dense x61 grid of 128-dim local features ft. Formally, it can be represented as
f LocallAdaptation(/m) =intal22lppccco/22aaaaaapppoo (fm)))
3.4 LOCAL MATCHING FOR RE-RANKING
Our two-stage place retrieval pipeline is shown in Fig. 3 After obtaining the global features and lo- cal features, we first compute L2 distance to perform the similarity search in the global feature space the datahase get the top-k most similar candidate images. For the local features matching between the query image and a candidate image c, we search for mutual nearest neighbor matches by cross-matching. Since local features are L2 normalized, the inner product equivalent to cosine similarity is used to measure local feature similarity. That is
Sye (i. (0) f(G) i,j 21, N., (N' 61 61),
where (1) is the i-th local feature in query image f!()) is the j-th local feature in candidate image c, and (i. the local feature similarity between them. The mutual nearest neighbor matches set M is defined
M {1,, u) arg max (i v), arg max (u,
That is, the u-th feature in image g and the v-th feature image care the best matches for each other.
The obtained nearest neighbor matches set in previous VPR work (Hausler et al. [021] Wang al.) 2022a) exists a large number of false matches. So they apply geometric verification (e.g. RANSAC Fischler & Bolles 1981)) to remove outliers (i.e. false matches) and use the number of inliers the
image similarity score, which is time-consee consumnnn Due to the powerful representation ability of the foundation model and the superiority of the ViT in capturing long ding-distance feature dependencies, the number false matches in this work is actually not enough to affect the performance of re-ranking. So directly use the number of matches (i.e., [M)) the image similarity score for re-ranking.
5

---

Published as a conference paper at ICLR 2024
3.5 LOSs
For the loss designed to optimize the model to produce global features (denoted as global loss Lg) follow the triplet loss used in the previous works (Arandjelovic et al. 2016 Wang et al. 2022a):
bg = E1(1 f21 +
115 fü, I),
where ((x) = max(x, 0), i.e. hinge loss. m the margin. fă Sg. and fäj are the global features of query, positive, and hard negative samples, respectively.
However, what we use to measure image similarity in local matching re-ranking is the number of mutual matches, which discrete integer variables within deep learning model because of its non-differentiabiliyy So compromise to optimize the network so that the resulting mutual matching local features are more similar, and design mutual nearest neighbor local feature loss L1
discrete integer. It is intractable directly optimize loss function of
Σ L = Σ 1( EiwwjeM (pp(u, v) + E(ee Syn, (u', 22),
(10)
[M]
IM
This local loss maximizes the average local feature similarity in the mutual matches set M of the query image and positive image, and minimizes that in the matches set M' of the query and negative images. It makes the produced local features more suitable for local matching, We obtain the final loss L through combining the global loss Lg and local loss L, by weight as
I.g Ala-
(11)
EXPERIMENTS
4.1 DATASETS AND PERFORMANCE EVALUATION
Several VPR benchmark datasets mainly including Tokyo24/7, MSLS, and Pitts30k are used in our ex- periments. Table [ summarizes their main informa- tion. Tokyo24/7 Torii et al. 2015) includes about 76k database images and 315 query images captured from ur- ban scenes with drastic illumination changes. Mapillary Street-Level Sequences (MSLS) (Warburg al., 2020)) consists of more than 1.6 million images collected in ur- ban, suburban and natural scenes over 7 years. We assess
Table 1: Summary of the main evalua- tion datasets.
Dataset
Deseription
Number
Datahise Queries
Tokyo2477
urban, day/night
76k
315
MSLS-va
urban, suburban
740
MSLS challenge
long temm
39k
27092
Pins test
urhan, panoramna
6816
models on both MSLS-val and MSLS-challenge (an online test set without released labels) sets. Pittsburgh (Pitts30k) (Tori et al., 2013) contains 30k reference images and 24k query images in the train, val and test sets, and exhibits severe viewpoint changes. More details are in Appendix
We evaluate the recognition performance using Recall@N (R@N), which the percentage of queries for which least one of the N retrieved images is the right result. The threshold is set to 25 meters and 40 for MSLS, 25 meters for Tokyo24/7 and Pitts30k, following standard evaluation procedure (Warburg et al. 2020 Arandjelovic et al. 2016).
4.2 IMPLEMENTATION DETAILS
We use the DINOv2 based on ViT-L/14 as the foundation model and conduct all experiments on an NVIDIA GeForce RTX 3090 GPU using PyTorch. Fed a 224x224 image, the model produces a 1024-dim global feature and a dense grid of 128-dim local features. The bottleneck ratio of the adapters VIT blocks is 0.5 and the scaling factor Eq. is set to 0.2. We use 3x3 up-conv with stride=2 and padding=1 in the local adaptation module. The output channels of the first and second up-conv layers are 256 and 128, respectively. Following other two-stage methods, we re- rank the 100 candidates to yield final results. We train models using the Adam optimizer with the learning rate set as 0.00001 and batch size set as 4. When the R@5 on the validation set does not have improvement within epochs, the training is terminated. For MSLS, we set epoch

---

Published as a conference paper at ICLR 2024
Table 2: Comparison to state-of-the-ar methods on benchmark datasets. The best is highlighted in bold and the second is underlined.
Tokyo24/7
MSLS-val
MSLS-challenge
3kkts33k-test
Method
R@1
R@5
R@
10
R@1
R@5
R@10
10
R@1
R@5
R 10
R@1
R@5
R@ 10
NetVLAD
60.6
68.9
74.6
53.1
66.5
71.1
35.1
47.4
51.7
81.9
91.2
93.7
SERS
81.0
88.3
92.4
69.2
80.3
83.1
41.6
52.0
56.3
89.4
94.7
95.9
CosPlace
81.9
90.2
92.7
82.8
89.7
92.0
61.4
72.0
76.6
88.4
94.5
95.7
MixVPR
85.1
91.7
94.3
88.0
92.7
94.6
64.0
75.9
80.6
91.5
95.5
96.3
(glooP
81.9
94.9
96.5
87.7
95.8
96.6
69.6
86.9
90.1
90.2
96.1
97.1
SP-SuperGlue
88.2
90.2
90.2
78.1
81.9
84.3
50.6
56.9
58.3
87.2
94.8
96,4
Patch-NetVLAD-s
78.1
83.8
87.0
77.8
84.3
86.5
48.1
59.4
62.3
87.5
94.5
96.0
Patch-NetVI.A--p
86.0
88.6
90.5
79.5
86.2
87.7
48.1
57.6
60.5
88.7
94.5
95.9
TPR
79.0
82.2
85.1
86.8
91.2
92.4
63.9
74.0
77.5
89.0
94.9
96.2
StructVPR
88.4
94.3
95.0
69.4
81.5
85.6
90.3
96.0
97.3
RR Former
88.6
91.4
91.7
89.7
95.0
96.2
73.0
85,9
88.8
91.1
95.2
96.3
SclaVPR (ours)
94.0
96.8
97.5
90.8
96.4
97.2
73.5
87.5
90.6
92.8
96.8
97.7
as passing 30k queries, whereas Pitts3kk passing 5k queries. In model training, we define the potential positive images as the reference images that are within 10 meters from the query image, while the definite negative images are those further than 25 meters. Two hard negative images from 1000 randomly chosen definite negatives are used in the triplet loss. We empirically set the margin m = 0.1 in Eq. 9 the weight À = in Eq. 11 For the experiments in Section 4.3 (i.e. comparisons with other methods), we fine-tune further fine-tune it on Pitts30k-train to test on Pitus30k-test and Tokyo24/7 (as in Fo (Zhu al. 2023)). For the ablation experiments in other sections, the models tested on Pitts30k-test and To/yo24/7 are directly fine-tuned on titts33k-train by default.
model
MSLS-train to test MSL.Sval/Chaallene,, and
4.3 COMPARISONS WITH STATE-OF-THE-AR METHODS
In this section, we compare the proposed SelaVPR method with several state-of-the-art (SOTA) VPR methods, including four one- ote-stage methods using global feature retrieval: NetVLAD (Arandjelovic let al. 2016). SFRS (Ge et al.) 2020). CosPlace (Berton et al.) 2022a) and MixVPR (Ali-Bey et al. 2023 as well as five two-stage methods with re-ranking: SP- SP-SuperGlue (DeTone et al.
2018 Sar-
lin et al. 2020). Patch-NerVLLAU (Hausler et al., (022)) TransVPR (Wang et al.] 2022a). Struct VPR
(Shen et al. 2023) and R² Former (Zhu et al. 2023). The details of these methods are Appendix Note that CosPlace and Mix VPR are trained on individually constructed large-scale datasets.
K
VPR and Former are transformer-based methods. Fororrer ranked first the leaderboard of the MSLS place recognition challenge before our method was proposed. We also show the global retrieval result without re-ranking using the proposed SelaVPR, and denote it as SelaVPR(globar The quantitative results are shown in Table 2 Since the code for StructVPR has not yet been re- leased, we use the results reported in the paper (Shen et al. 2023). The proposed method achieves the best R @ I/R @ 5/R @ 10 on all datasets.
When using only global features for direct retrieval, SelaVPR(global) significantly outperforms other one-stage methods all datasets for R@5 and R@ 10, including SFRS using complex train- ing strategies and CosPlace/Mixx VPR trained on purpose-built large-scale datasets. SelaVPR(global) also outperforms all other two-stage methods Tokyo24/7, MSLS-val, and MSLS-challenge for R R@5 and R@ 10. This fully demonstrates that adapting the foundation model can provide pow- erful feature representation, which is novel way to achieve SOTA one-stage VPR. Although Sela (elgloR(gll does not achieve very good R@ on Tokyo24/7 where the lighting changes drasti- cally, the complete SeR method outperforms other methods by a large margin after local feature re-ranking. This illustrates the necessity of using local matching (local adaptation) for VPR in ex- treme environments. The complete SPR method significantly outperforms SOTA methods on Tokyo24/7 and Pitts30k with absolute R@1 improvement of 5.4% and 1.3% respectively. Mean- while, it also ranks 1st on the leaderboard of MSLS place recognition challenge (see Appendix L. Comparisons to SOTA methods more datasets are in Appendix F (SelaVPR achieves near- perfect results on St. Lucia). Fig. 4 qualitatively demonstrates that our approach is highly robust in challenging scenes. Benefiting from the visual foundation model and sensible adaptation, Sela SPR achieves absolute performance advantages on a variety of datasets without complex training strate- gies or purpose-buie built large-scale training datasets.
7

---

Published as a conference paper at ICLR 2024
Query
PP (Ours)
NetVLD
SFRS
CosPlace
Patch-NetVLAD
Trans VPR
Light changes
Perceptual aliasing
Light
changes
Viewpoint
changes
Danamic
objects
Viewpoint
changes
Percepiual
alissing
Figure 4: Qualitative results. In these challenging examples (containing condition changes, view- point changes, dynamic objects, etc.), the proposed SelaVPR successfully returns the right database images, while all other methods produce incorrect results.
Table 3:
The single query runtime comparison of two-
93
SP SuperGilue Patch-NetVLAD: Patch-MetVLAD-p
stage methods on Pitts3Ok-test.
92
Recalla (%)
Extraction
Matching Time (s)
Total Time (s)
91
TransVPR
Method
SelaVPR (ours)
Time (s)
90
SP-SuperGlue
0.042
6.639
6.681
89
Patch-NetVIAA--s
0.186
0.551
0.737
88
PacD--NeVLAD-pp
0,412
10.732
11.144
87
0.008
3.010
3,018
8 10
Total Runtime (s)
ScR (ours)
0.027
0.085
0.112
Figure 5: R@@ 1-runtime comparison.
Efficiency is another metric for evaluating the VPR methods. In Table 3 we compare the run- time (feature extraction time and matching/retrieval time) of our method with other two-stage meth- ods on Pitts30k-test. Match-NetVLADD. and Trans VPR are representative two-stage methods using RANSAC for spatial verification in re-ranking. SP-SuperGlue uses neural networks to match local features. TPR is fast at extracting features, while SelaVPR is slower (but faster than other methods) due to the use the ViT/L. backbone. However, for matching/retrieval, since our method does not require time-consuming spatial verification in re-ranking, its runtime is less than 3% of TransVPR and only about 1% of SP-SuperGlue and Patch-NetVLAD-pp The total runtime of our method is less than 4% of TransVPR Although Patch-NetVLAD-s uses Rapid Spatial Scoring method for fast verification, it is still significantly slower than ours. Fig. 5 simultaneously shows the total runtime and @@1 1 Pitts30k. Due to the absolute advantages in both performance and efficiency, our SeR method is able to pave the way for real-world large-scale VPR applications.
4.4 ABLATION STUDY
In this section, we perform a series of ablation experiments to demonstrate the necessity of fine- tuning and verify the effectiveness of the proposed global adaptation and local adaptation:
DINOv2-GeM: Using the pre-trained DINOv2 backbone (freeze parameter) and GeM pooling, i.e., baseline.
Tuned-DINOv2-GeMM. full fine-tuned DINOv2-GeM.
Global-Adaptationn Global adapted DINOv2-GeM with our global adapation. Local-Adaptation: Using the DINOv2-Gen (freeze parameter) to retrieve candidates, and adding the local adaptation module after it to produce local features for re-ranking. Sela The complete (hybrid global-local adaptation) method.
The results shown in Table 4 Note that we directly the model fine-tuned Pitts30k-train to test on 30tts30k and Tokyo2477 in this section. The results are slightly lower than that in Table 2 (but still SOTA).
8

---

Published as a conference paper at ICLR 2024
Table 4: Comparison of different ablated versions.
Pitts30k-rr test
Tokyo24/7
MSL.Sval
Ablated version
R@1
R@5
R@ 10
R@1
R@5
R@
10
R@1
R@5
R
10
DINOv2-GeM
81.3
91.0
93.8
67.3
85.1
89.8
44.7
55.9
59.3
85.3
92.7
94.7
65.7
78.1
83.8
79.7
90.3
92.2
Global-Adaptation
87.3
94.6
96.6
77.8
87.6
91.7
87.4
95.9
96.9
Local-Adaptation
87.2
93.9
96.1
87.6
94.6
95.9
67.2
75.0
76.6
SelaVPR
91.4
96.5
97.8
93.3
95.2
95.6
90.8
96.4
97.2
The pre-trained DINOv2-GeM achieves decent results on Pitts30k, which has few dynamic objects on the place image. The performance improved after full fine-tuning (i.e. Tuned-DINNOv2-Gel indicating that fine-tuning is necessary to make the produced feature representation more suitable for the VPR task even without dynamic interference. However, Tuned-DINOv2-GeMM performs worse than DINOv2-GeM on Tokyo24/7. It is because there is generalization gap between Tokyo24/7 and Pitti30k. and full fine-tuning damages the excellent transferability of the pre-trained founda- tion model. Compared with Tuned-DINOV2--Gel Global-Adaptation only tunes few newly added adapters (with backbone frozen), which reduces the training consumption and always improves per- formance (retaining the generalization ability of the foundation model). Global-Adaptanl absolute R@ 1 improvement of 10.5% on Tokyo24/7. Besides, Local-Adaptation only adds local adaptation module (with the proposed local loss for training) after the frozen DINOv2 to produce local features for re-ranking. It can also obviously improve performance, especially on Tokyo24/7 (20.3% absolute @ improvement), which shows day-night changes. The complete Sela VPR method combines the advantages of Glocal-Adaptation and Local-Adaptation, and achieves absolute R @ improvement of 10.1 % on Pitts30k and 26.0% on Tokyo24/7.
achieves
MSLS is a dataset with many dynamic objects (e.g. vehicles and pedestrians), on which the pre- trained DINOv2-GeM gets terrible results. By adding a local adaptation module after DINOv2 (Local-Adaptation) to get dense local features for re-ranking, we can get 22.5% absolute per- formance improvement for R@ 1, which is still not ideal. Using full fine-tuning can even bring significant performance improvement (35.0% for R@ 1), and the improvement yielded by Global- Adaptation is more obvious (42.7% for R@ I). More importantly, the complete SelaVPR method achieves 2 x higher @
Overall, full fine-tuning the pre-trained foundation model on these datasets usually results in better performance (unless there is a domain gap between the training and test set), indicating that the first problem should be solved when applying a foundation model to the VPR task is to make the output feature representation suitable for VPR (focusing regions that help differentiate places), ie. bridge the gap between pre-training and VPR tasks. The second is the catastrophic forgetting that will be encountered in fine-tuning the pre-trained models, which can be addressed by parameter- efficient fine-tuning instead of full fine-tuning. Additionally, the local matching for re-ranking is also necessary, especially on datasets that show drastic condition variations (e.g. Tokyo24/7). The proposed global adaptation and local adaptation can work together to solve these problems well.
More experiments and analyses are in the Appendix, e.g., the comparisons of tunable parameters, data efficiency, and training time.
5 CONCLUSIONS
In this paper, we introduced a novel hybrid adaptation method to seamlessly adapt pre-trained foun- dation models for the VPR task, which is composed of the global adaptation and local adaptation. The feature representation produced by the adapted foundation model is more focused on discrim- inative landmarks to differentiate places, thus bridging the gap between the pre-training and VPR tasks. The local adaptation enables the model to output proper dense local features, which can be used for local matching re-ranking in two-stage VPR to greatly boost performance. The experimen- tal results demonstrated that the proposed Sela method outperforms previous SOTA methods on VPR benchmark datasets (with a large margin on Tokyo24/7 and Pitts30k). Meanwhile, since our approach eliminates the reliance time-consume time-consuming spatial validation in re-ranking, it costs only 3% retrieval time of RANSAC baseed two-stage methods. We believe that the proposed SeRaVPR method provides promising way to address the VPR task in real-world large-scale applications.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Key R&D Program of China (202YFB44014400400014444 SSTIC Grant (K12D200009211110000)) Shenzhen Key Labora- tory (ZDSYS202 10200200222000000000000 Beijing Key Lab Networked Multimedia, and the Project of Peng Cheng Laboratory (PCL2023A08).
REFERENCES
Amar Ali-bey, Brahim Chaib-draa, and Philippe Giguère. Gsv-cities: Toward appropriate supervised visual place recognition. Neurocomputing, 113:194-003, 2022.
Amar Ali-Bey, Brahim Chaib-Draa, and Philippe Giguere. Mixvpr: Feature mixing for visual place recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2998-99989 2998-3007, 2023.
Adrien Angeli, David Filliat, Stéphane Doncieux, and Jean-Arcady Meyer. Fast and incremental method for loop-closure detection using bags of visual words. IEEE transactions on robotics, 24 (5):1027-1037, 2008
Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Cnn ar- chitecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5297-5307, 2016.
Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Lue Van Gool. Speeded-up robust features (surf). Computer vision and image understanding, 1 11003):346-3355 2008.
Gabriele Berton, Carlo Masone, Valerio Paolicelli, and Barbara Caputo. Viewpoint invariant dense matching for visual geolocalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12179- 2021.
Gabriele Berton, Carlo Masone, and Barbara Caputo. Rethinking visual geo-localization for large- scale applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4878 4888, 2022a.
Gabriele Berton, Riccardo Mereu, Gabriele Trivigno, Carlo Masone, Gabriela Csurka, Torsten Sattler, and Barbara Caputo. Deep visual geo-localization benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5396-5407, 2022b.
Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local and global features for image search. In European Conference Computer Vision, pp. 726-743. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678. 2022.
Zetao Chen, Adam Jacobson, Niko Sünderhauf, Ben Upcroft, Lingqiao Liu, Chunhua Shen, Ian Reid, and Michael Milford. Deep learning features scale for visual place recognition. In 20/7 IEEE international conference on robotics and automation (ICRA), pp. 3223-222332333 3230.. IEEE, 2017a.
Zetao Chen, Fabiola Maffra, Inkyu Sa, and Margarita Chli. Only look once, mining distinctive landmarks from convnet for visual place recognition. In 2017 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pp. 9-16. IEEE, 2017b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
10

---

Published as a conference paper at ICLR 2024
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and deseription. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp.
2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x 16 words: Transformers for image recognition at scale. International Confer- on Learning Representations, 2020.
Martin A Fischler and Robert Bolles. Random sample consensus: a paradigm for model fitting with applications image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
Sourav Garg and Michael Milford. Seqnet: Learning descriptors for sequence-based hierarchical place recognition. IEEE Robotics and Automation Letters, 6(3):4305 4312, 2021.
Sourav Garg. Adam Jacobson, Swagat Kumar, and Michael Milford. Improving condition-ane enviromment-ivariian place recognition with semantic place categorization. In 2017 IEEE/RSI International Conference Intelligent Robots and Systems (IROS), pp. 6863-6870. IEEE, 2017.
Sourav Garg, Niko Suenderhauf, and Michael Milford. Don't look back: Robustifying place cate- gorization for viewpoint-and condition-invarian place recognition. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3645-3666 3652, 2018.
Yixiao Ge, Haibo Wang. Feng Zhu, Rui Zhao, and Hongsheng Li. Self-supeiiiiing fine-grained region similarities for large-scale image localization. In European conference on computer vision, pp. 369-386 Springer, 2020.
Arren Glover, William P Maddern, Michael J Milford, and Gordon Wyeth. Fab-map+ ratslam: Appearance-basse slam for multiple times of day. In 2010 IEEE international conference robotics and automation, pp. 3507-3512. IEEE, 2010.
Stephen Hausler and Michael Milford. Hierarchical multi-process fusion for visual place recogni- tion. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3327- 3333. IEEE, 2020.
Stephen Hausler, Sourav Garg, Ming Xu, Michael Milford, and Tobias Fischer. Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14141-14152, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
arXiv:2106.0966 2021.
Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into compact image representation. In 2010 IEEE computer society conference on computer vision and patlern recognition, pp. 3304-3311. IEEE, 2010.
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint ar arXiv:2207.0700 2022.
Hyo Jin Kim, Enrique Dunn, and Jan-Michael Frahm. Learned contextual feature reweighting for image geo-localization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2136-2145, 2017.
Nikhil Keetha, Avneesh Mishra, Jay Karhade, Krishna Murthy Jatavallabhula, Sebastian Scherer, Madhava Krishna, and Sourav Garg. Anyloc: Towards universal visual place recognition. arXiv preprint 2r3i2:230 000888008..88........ 2023.
11

---

Published as a conference paper at ICLR 2024
Ahmad Khaliq, Shoaib Ehsan, Zetao Chen, Michael Milford, and Klaus Se-oonald-Miie A holistic visual place recognition approach using lightweight cnns for significant viewpoint and appearance changes. IEEE transactions on robotics, 36(2):561-569, 2019.
Zaid Khan and Yun Fu. Contrastive alignment of vision to language through parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficieet prompt tuning. arXiv preprint arXiv:2 104. 104.08691, 2021.
María Vallaa,,, Nicola Strisciuglio, and Nicolai Petkov. Generalized contrastive optimization of siamese networks for place recognition. arXiv preprint arXiv:2 10..06603336633 2021.
María Leyva-Vallina, Vallina, Nicola Strisciuglio, and Nicolai Petkov. Data-efficient large scale place recog- nition with graded similarity supervision. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 23487-23496, 2023.
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2041-2050, 2018.
Liu Liu, Hongdong Li, and Yuchao Dai. Stochastic attraction-repulsion embedding for large scale image localization. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2570-2579, 2019.
Stephanie Lowry and Henrik Andreasson. Lightweight, viewpoint-invariint visual place recognition in changing environments. IEEE Robotics and Automation Letters, 3(2):957-964, 2018.
Stephanie Lowry. Niko Sünderhauf, Paul Newman, John J Leonard, David Cox, Peter Corke, and Michael Milford. Visual place recognition: A survey. ieee transactions on robotics, 32(1):1-19, 2015.
Feng Lu, Baifan Chen, Xiang-Dong Zhou, and Dezhen Song. Sta-vpr: Spatio-temporal alignment for visual place recognition. IEEE Robotics and Automation Letters, 6(3):4297 2021.
Feng Lu, Lijun Zhang, Shuting Dong, Baifan Chen, and Chun Yuan. Aanet: Aggregation and alignment network with semi-hard positive sample mining for hierarchical place recognition. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11771-11778. IEEE, 2023.
Sven Middelberg, Torsten Sattler, Ole Untzelmann, and Leif Kobbelt. Scalable 6-dof localization on mobile devices. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part II 13, pp. 268-283. Springer, 2014.
Tayyab Naseer, Gabriel L Oliveira, Thomas Brox, and Wolfram Burgard. Semantics-aware visual localization under challenging perceptual conditions. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 2614-2620. IEEE, 2017.
Daniel Olid, José M Fácil, and Javier Civera. Single-view place recognition under seasonal changes. arXiv preprint arXiv: 1808.06516..66 06516,656661 2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXh::2304.07199. 2023.
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning, Advances in Neural Information Processing Systems, 35:26462- 26477, 2022.
Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Dual-path adaptation from image to video trans- formers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 2203-2213, 2023.
12

---

Published as a conference paper at ICLR 2024
Guohao Peng, Jun Zhang, Heshan Li, and Danwei Wang. Attentional pyramid pooling of salient visual residuals for place recognition. In Proceedings of the IEEE/CVF International Conference on Computer Viston, pp.
2021.
Filip Radenović, Giorgos Tolias, and Ondiej Chum. Fine-tuning emn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence, 41(7):1655- 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 4938-4947, 2020.
Yanqing Shen, Ruotong Wang, Weiliang Zuo, and Nanning Zheng. Tel: Tightly coupled learning strategy for weakly supervised hierarchical place recognition. IEEE Robotics and Automation Letters, 7(2):2684-2611, 2022.
Yanqing Shen, Sanping Zhou, Jingwen Fu, Ruotong Wang, Shitao Chen, and Nanning Zheng. Structvpr: Distill structural knowledge with weighting samples for visual place recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11217-11226, 2023.
Niko Sünderhauf, Sareh Shirazi, Feras Dayoub, Ben Uperoft, and Michael Milford. On the perfor- mance of convnet features for place recognition. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 4297-4304. IEEE, 2015.
Akihiko Torii, Josef Sivic, Tomas Pajdla, and Masatoshi Okutomi. Visual place recognition with repetitive structures. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 883-890, 2013.
Akihiko Torii, Relja Arandjelovic, Josef Sivic, Masatoshi Okutomi, and Tomas Pajdla. 24/7 place recognition by view synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1808-1817, 2015.
Ruotong Wang, Yanqing Shen, Weiliang Zuo, Sanping Zhou, and Nanning Zheng. Transvpr: Transformer-based place recognition with multi-level attention aggregation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13648-13657, 2022a.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image foreign language: Beit pretraining for all vision and vision- laiiunnl tasks. arXiv preprint arXiv:2208. 10442, 2022b.
Frederik Warburg, Soren Hauberg, Manuel LotezeAnteu Pau Gargallo, Yubin Kuang, and Javier Civera. Mapillary street-level sequences: A dataset for lifelong place recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2626- 2635, 2020.
Zhe Xin, Yinghao Cai, Tao Lu, Xiaoxia Xing, Shaojun Cai, Jixiang Zhang, Yiping Yang, and Yan- qing Wang. Localizing discriminative visual landmarks for place recognition. In 2019 Interna- lional conference on robotics and automation (ICRA), pp. 5979-5985. IEEE, 2019.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2945-2954, 2023.
Ming Xu, Niko Snderhauf, and Michael Milford. Probabilistic visual place recognition for hierar- chical localization. JEEE Robotics and Automation Letters, 6(2):311-318, 2020.
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. 2023.
13

---

Published as a conference paper at ICLR 2024
Peng Yin, Lingyun Xu, Xueqian Li, Chen Yin, Yingli Li, Rangaprasad Arun Srivatsan, Lu Li, Jianmin Ji, and Yuqing He. A multi-domain feature learning method for visual place recognition. In 2019 International Conference on Robotics and Automation (ICRA), pp. 119-324. IEEE, 2019.
Jun Yu, Chaoyang Zhu, Jian Zhang, Qingming Huang, and Dacheng Tao. Spatial pyramid-enhanced netvlad with weighted triplet loss for place recognition. IEEE transactions on neural networks and learning systems, 31(2):661-613 2019.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432. 2021.
Sijie Zhu, Linjie Yang, Chen Chen, Mubarak Shah, Xiaohui Shen, and Heng Wang. R2former: Unified retrieval and reranking transformer for place recognition. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19370-19380, 2023.
14

---

Published as a conference paper at ICLR 2024
Table 5: Tunable parameters, training epoch, and training time of different methods on Pitts30k.
Method
Tunable Param (M)
epoch
time (h)
R@1
R@5
ResNet50-GeM
1.63
82.3
91.9
Lumed-DNNOv2-GeM
304
2
0.62
85.3
92.7
SelaVPR
53
1
0.30
91.4
96.5
SelaVPR*
0.4
0.12
91.0
96.2
Table The results of using different adapters in the global adaptation.
P0nts30kttest
MSLS-val
Method
R@1 1
R@5
R@ 10
R@1
R@5
@@ 10
only serial adapter
90.9
96.2
97.4
90.9
95.8
96.9
only parallel adapter
91.3
96.6
97.7
89.7
96.2
96.6
both adapters
91.4
96.5
97.8
90.8
96.4
97.2
A COMPARISONS OF TUNABLE PARAMETERS, DATA EFFICIENCY, AND TRAINING TIME
In this section, we evaluate the tunable parameters, data efficiency, and training time of our model. We use the ResNet50 with GeM pooling the benchmark implementation (Berton al. 2022b) (ResNet50-GeMM and the full fine-tuning DINOv2-GeM experiment is conducted on Pitts30k and the results are shown in Table 5 The tunable parameters of our SeR are only about 1/6 of full fine-tuning DINOv2 TTune--DINOv2-GeMM)) Although our model has more tunable parameters than the ResNet50-GeM (CNN backbone), we require sig- nificantly less training data and time, and achieve 9.1% higher R@) Note that epoch only contains a part of the training set (5000 triples). To further study the data efficiency of our model, train it with only 2000 triples (i.e., 0.4 epoch) and denote it as SelaVPR* # It still achieves SOTA performance.
Lamed-DINOv2-Geell as reference. This
B ADDITIONAL ABLATION EXPERIMENTS FOR THE GLOBAL ADAPTATION
In this section, further conduct ablation experiments to study the effect of reducing the number of tunable parameters in the global adaptation. Table the results of using different adapters, iee. only serial adapter, only parallel adapter, both adapters. Using both adapters achieves the best results overall, but we can still get good performance with just either adapter. Table 7 shows the results of setting different bottleneck ratios in each adapter. Setting the ratio 0.5 gets the best results overall, but we can still achieve good performance with a 0.25 bottleneck ratio. These experimental findings show that promising results can be obtained even if the number of tunable parameters in the global adaptation is reduced. We can choose appropriate settings in the global adaptation according our needs.
Table The results different bottleneck ratios of the adapters.
Pitts30k-test
MSLS-val R@5
ratio
R@1
R@5
@ 10
R@1
R@
10
0=255
91.0
96.2
97.4
91.2
96.6
96.9
r=0.5
91.4
96.5
97.8
90.8
96.4
97.2
r=0.75
90.9
95.9
97.2
91.1
96.4
96.6
15

---

Published as a conference paper at ICLR 2024
Table 8: Performance of different global features without or with re-ranking.
Pitts30k-test
MSLS-val
Method
R@1
R@5
R@ 10
R@1 1
R@5
R 10
SelaVPR-clsglobaa))
89.2
96.2
97.6
89.3
96.2
97.2
SelaVPR-Ge.(gglbbal)
87.2
95.1
97.2
87.7
95.8
96.6
SelaVPR-cls(re-rank)
90.1
96.0
97.4
83.5
92.2
94.1
SelaVPR-GGeM(re-rank
91.4
96.5
97.8
90.8
96.4
97.2
Table 9: The results of using "local" features different granularities for re-ranking.
Pitts30k-test
MSLS-val
Method
R@l
R@5
R@ 10
R@1
R@5
R @ 10
global adaptation for global retrieval
87.3
94.6
96.6
87.4
95.9
96.9
coarse patch tokens for re-ranking
89.8
95.4
96.9
82.8
92.0
94.9
dense local features for re-ranking
91.4
96.5
97.8
90.8
96.4
97.2
C PERFORMANCE OF DIFFERENT GLOBAL FEATURES
This section compares the performance of using class token or GeM as global features in our SelaVPR method. As shown in Table the performance of direct global retrieval (without ranking) using class token as the global feature, i.e. Sela SelaVPR-cls(globa)) is better than that of GeM. However, SelaVPR-GeMer features. That is, the improvement of SelaVPR-cls after re-ranking is significantly lower than that of SelaVPR-GeM,, and the performance of SelaVPR-cls after re-ranking even decreased on MSL.S- val. This shows that when we use local features in conjunction with global features, GeM is better compatible with local features, whereas class token not. Therefore we finally choose GeM as the global feature in SeR.
is better than SelaVPR-cls(re-rank after re-ranking using local
D PERFORMANCE OF "LOCAL"" FEATURES AT DIFFERENT GRANULARITIES,
To illustrate the necessity of using dense local features in reranking, we provide a comparison of reranking using coarse 16x 16 patch tokens (directly produced by the global adapted backbone, and treated coarse-grained "local" features) and our dense local features. The results are shown in Table 9 Although re-ranking with coarse patch features provides an improvement (but as good as ours) Pitts30k compared to only global adaptation for global retrieval (without re-ranking), performs significantly worse than global retrieval on MSI.S-val, where VPR methods are more susceptible to perceptual aliasing. That is, it will bring negative effects and damage the results of global retrieval. However, using dense local features (produced by local adaptation) for re-ranking can always provide significant improvements compared to global retrieval.
E PERFORMANCE OF RE-RANKING DIFFERENT NUMBERS OF CANDIDATES
We show the results of our SelaVPR for re-ranking different numbers of candidates in Table I0 The performance roughly reaches saturation when re-ranking top-100 candidates, which is our rec- ommended setting for optimal recognition performance. Our approach also achieves excellent per- formance when only the top-20 candidates are re-ranked. This setting can reduce the re-ranking runtime by approximately 80%.
F COMPARISONS ON MORE DATASETS
This section provides comparisons with other methods on more datasets. We compare the proposed method with other two-stage methods the Nordland and St. Lucia datasets. As shown in Table II
16

---

Published as a conference paper at ICLR 2024
Table 10: The results of re-ranking different numbers of candidates. Note that the model tested on Tokyo24/7 and Pitts30k-test is fine-tuned on MSLS-train and then further fine-tuned on Pitts30k- train (same as compared with the SOTA methods, but different from the ablation experiments).
Tokyo24/7
Pitus30k-test
MSLS-val
Candidates
R@D
R@5
@@ 10
R@D
R@5
@@ 10
R@D 1
R@5
@@ 10
Top-20
93.0
96.2
97.1
92.7
96.6
97.5
90.7
96.4
97.0
Top-50
93.7
96.5
96.8
92.8
96.8
97.6
90.8
96.4
97.0
Top-100
94.0
96.8
97.5
92.8
96.8
97.7
90.8
96.4
97.2
Top-200
94.0
96.8
97.5
92.8
96.9
97.8
90.7
96.5
97.3
Table 11: Comparison to two-stage methods on Nordland-test and St. Lucia. The threshold is set to +2 frames for Nordland-test and 25 meters for St. Lucia. The model the same as that used for MSLS in Table 2
Nordland-test
St. Lucia
Method
R@1 1
R@5
R
10
R
R@5
R@ 10
SuperGGee
25.8
35,4
38.2
86.5
92.1
93.4
Patch-NerVLAA.s
44.2
57.5
62.7
90.2
93.6
95.0
Patch-NetVLAD-p
51.6
60.1
62.8
93.9
95.5
96.2
TPR
61.3
71.7
75.6
98.7
99.0
99.2
Sela SelaVPR(globa))
72.3
89.4
94.4
99.4
99.9
100.0
SelaVPR
85.2
95.5
98.5
99.8
100.0
100.0
our method surpasses other methods even without re-ranking, i.e., SelaVPR(global), The SelaVPR outperforms other methods by a wide margin and achieves near-perfect results on St. Lucia (99.8% R@I and 100% R@5). We also provide the results of our SelaVPR on Pitts250k Table 12 VPR achieves the previous best results on Pitts250k, and SeRaVPR outperforms it.
G QUALITATIVE RESULTS OF LOCAL MATCHING
We further illustrate that method makes the produced features by foundation models more suitable for local matching in this section, and show the qualitative local matching results of our SelaVPR method and the pre-trained DINOv2 in Fig. maps output by the ViT backbone of the two models are used for local matching. Our SeR gets more correct matches (i.e. matching point pairs) than the pre-trained DINÓv2 between two images from the same place. For two images from different places, all local matches are wrong and we expect as few matches as possible. SelaVPR produces fewer wrong matches than pre-trained DINOv2.
To make fair comparison, the feature
Table 12: The results of MiX and our method on Pitts250k. The model is the same that used for Pitts30k in Table 2
Pitts250k-test
Method
@
R@5
R@1
10
Mix
94.3
98.2
98.9
Sela VPR
95.7
98.8
99.2
17

---

Published as a conference paper at ICLR 2024
Produced by SeR (60 matches) (a) Local matching between two images from the same place
Produced by DINOv2 (57 matches)
Produced by our SelaVPR (47 matches)
Produced by DINOv2 (55 matches) (b) Local matching between two images from different places
Figure 6: Comparison of local matching between our SelaVPR and pre-trained DINOv2. (a) shows the local matching between images from the same place. The more matches the better, and the matches produced by ours are more than those of DINOv2. Note that some matches produced by DINOv2 are clearly inappropriate. (b) shows the local matching between images from different places. The fewer (wrong) matches the better, and our method has fewer (wrong) matches than pre- trained DINOv2.
H ADDITIONAL ATTENTION VISUALIZATION
Fig. the main paper has shown that the feature representation of our method can precisely focus on image regions that are helpful for place recognition. Here, Fig. demonstrates more challenging examples. Compared to the pre-trained model, which is disturbed by objects such as dynamic foreground, our method always pays attention to all discriminative regions (buildings and vegetation).
I ADDITIONAL QUALITATIVE RESULTS
Fig. 4 in the main paper has shown a small number of qualitative results. Here, Fig. Fig. and Fig.
10 show more qualitative results Tokyo24/7, MSLS, and Pitts30k, respectively. These examples
show challenging cases such as severe condition changes, viewpoint changes, dynamic interference, and only small regions of discriminative landmarks almost no landmarks. Our method obtains correct results, while other methods produce incorrect results.
J DATASET DETAILS
Tokyo24/7 Torii et al. 2015). The Tokyo24/7 dataset includes 75984 database images and 315 query images captured from urban scenes. The query images selected from 1 1125 images taken at 125 distinct places with different viewpoints and at different times of day. This dataset mainly exhibits viewpoint changes and drastic condition changes (day-nigh) changes).
Mapillary Street-Level Sequences (MSLS) (Warburg et al. 2020). The MSLS dataset a large- 1.6 million images labeled with GPS coordinates and compass angles, captured from 30 cities in urban, suburban, and natural scenes over seven years. cov- various challenging visual changes due illumination, weather, season, viewpoint, well as
scale VPR dataset containing
18

---

Published as a conference paper at ICLR 2024
Input images
Results of DINOv2
Results of ours
(1)
2)
(4)
Figure 7: Additional attention map visualizations of the pre-trained model (DINOv2) and our model. We compute the mean in the channel dimension of the output feature map and display it using the heat map. In the last example, although our method is also affected by the bus, this because is closer and takes up larger region of the image, while the buildings and vegetation are far away and occupy a smaller region. However, our method can still avoid missing any discriminative landmark (building and vegetation) to help retrieve correct results, while the pre-trained model cannot.
dynamic objects, and includes subsets of training, public validation (MSLS-val) and withheld test (MSLS-an -hallechallnnge). Following several related works (Hausler et al. 202T] Wang et al., 2022a Zhu et al. 2023), the S-al-- and MSL.S-challeng sets are used to evaluate models.
19

---

Published as a conference paper at ICLR 2024
Query
SelaVPR (Ours)
NaVLAD
SFRS
CosPlace
Patch-Nete NEtVLAD VLAD
TransVPR
Figure 8: Qualitative results on Tokyo24/7. These challenging examples exhibit severe condition (lighting) changes and viewpoint changes.
Query
SelaVPR (Ours)
NeVLAD
SFRS
CosPlace
Patch-Nat NaD
TransVPR
Figure 9: Qualitative results on MSLS. The first and second examples are dominated by vegetation (which tends to change over time). Most methods are prone to perceptual aliasing in these examples. The third example is extremely dark. All other methods return nighttime images but are wrong, only method gets the correct image (but during daytime).
Query
SelaVPR (Ours)
NaVLAD
SPRS
CosPllace
Patch NaVVAD
TransV
GEG
Figure 10: Qualitative results on Pitt33k.. In the first and second examples, most of the image area is sky, and our method retrieves correct results using distinguishable landmarks that occupy only small portion of the image region. The third example shows large viewpoint changes. Our approach is robust to these challenges.
20

---

Published as a conference paper at ICLR 2024
Pittsburgh (Torii et al. 2013). The Pittsburgh dataset is collected from Google Street View panora- mas, and provides 24 images with different viewpoints at each place. The images in this dataset exhibit large viewpoint variations and moderate condition variations. The Pitts30k dataset is sub- set of Pitts250k (but more difficult than Pitts250k) and contains 10k database images each in the training, validation, and test sets.
Nordland Olld et al. 2018). The Nordland dataset primarily consists of suburban and natural place images, captured from the same viewpoint in the front of train across four seasons, which allows the images to show severe condition (e.g., season and light) changes but no viewpoint variations. Its ground truth is provided by the frame-level correspondence. Following the previous works (Olid et al.] 2018)] Wang et al. 2022a), we use the dataset partition first presented in (Olid et al. (2018) for our experiments. The summer (reference) and winter (query) images of the down-sampled version (224x of the test set (3450 images per sequence) are adopted to evaluate models.
St. Lucia (Glover et al., 2010; Berton al. 2022b). The St. Lucia dataset comprises ten video sequences captured from the same suburban roadway in Brisbane. Following visual geo-localization benchmark (Berton et al. 2022b), the first and last sequences are used as the reference and query data in our experiments, and only one image is selected every 5 meters. Thus, the reference and query sequence include 1549 and 1464 images, respectively.
K DETAILS OF THE COMPARED METHODS
2016). NetVLAD is a classic one-stage VPR method with differen- tiable VLAD layer that can be integrated into neural networks. We use the pytorch implementatior with the released VGG16 model trained on the Pits30k dataset.
NetVLAD (Arandjelovic et al.
SFRS GG et al., 2020 2 To train a more robust NetVLAD-basee model, this work utilizes self- supervised image-to to-region similarities mine hard positive samples. The official model trained on Pitts30k is used in our experiments.
CosPlace (Berton al., 2022a This work introduces an extra large-scale dataset SF-XL and applies the training technique proposed in the classification task to train the VPR models. The official VGG16 model is used for comparisons.
MixVPR (lli-Bey et al., 2023 This work presents novel holistic feature aggregation method that takes feature maps from pre-trained backbones as global features, and uses a stack of Feature- Mixer to iteratively incorporate global relationships into each individual feature map. MixVPR is a SOTA one-stage VPR method. We use the best configuration (ResNet50 with 4096-dim output features) for comparisons.
SP-SuperGlue (DeTone et al. 2018 Sarlin et al. 2020 Hausler et al. (022)) first used this pipeline for the VPR task in the PatchNetVLAD work. SP SP-SuperGlue first retrieves candidate im- ages using NetDLAD. Then, the SuperGlue (Sarlin et al., 2020) (a feature matcher based on graph neural network) is used to match SuperPoint Tone et al. 2018) features for re-ranking. The official implementation with the model trained on Megalepthh (Li & Snavely 2018) is adopted.
Patch-NetVLAD (Hausler al. 2021 This approach also retrieves candidates with NetVLAD, then re-ranks the candidates using the NetVLADDbased multi-scale patch-level features. We use the speed-focused and cefformmann--ooussee configurations for evaluation. The official model trained on Pitts30--train is tested on Pitts30k-test and Tokyy24/7, and the model trained on MSLS is tested on other datasets.
Trans (Wang et al. 2022a This work first achieves candidate retrieval using the global
features produced by integrating multi-level attentions from vision transformer, then utilizes
21

---

Published as a conference paper at ICLR 2024
Resuits
Entrics
recali85
anonymous02
09/17/23
(1)
05/14/23
(2)
06/23/23
11
07/27/22
0.82 (4)
05/18/28
01/21/22
0.80 (6)
09/26/23
10
03/01/22
0.77 (8)
10/16/22
10
10/16/22
0.76 (10)
10/10/22
17
20
10/22/22
0.74
04/04/23
0.74
05/23/28
0.73 (14)
02/20/22
0.71 (15)
Figure 11: The snapshot of MSLS place recognition challenge leaderboard. Our SelaVPR method named "anonymous02" (for double-blin policy) ranks 1st at the time of submission.
tention mask filter feature maps to get key-patch descriptors for re-ranking. The official model trained on Pitts30k- train is evaluated on Pitts30k-test and Tokyo24/7, while that trained on MSLS is tested on others.
StructVPR (Shen et al. 2023). To improve feature stability in changing environment, StructVPR utilizes the segmentation images to enhance structural knowledge in global features and applies knowledge distillation to avoid online segmentation in testing. This method is combined with SP- SuperGlue to form a two-stage VPR method, achieves great performance. Since the code not released,
the results reported in the original paper.
R Former (Zhu et al. 2023 This work addresses both the retricval and re-ranking in two-stage VPR using novel transformer model. Its re-ranking module takes feature correlation, attention value, and coordinates into account, and does not require RANSAC-based geometric verification. Former is the SOTA two-stage VPR method. It ranked first in the MSLS challenge leaderboard before our method was proposed.
L THE SNAPSHOT OF MSLS LEADERBOARD
MSLS place recognition challenge (Huusler et al. 2021 Zhu et al. 2023) is an authoritative com- petition for VPR with over 100 participants. Fig. 11 shows the snapshot of the MSLS challenge leaderboard at the time of submission. The proposed SelaVPR method (named "anonymous02" for double-blind policy) ranks 1st.
hhttp///ghhaa..cmmmhccmmn//// com/hyyedaacc/22Fomme
hhups//codaaab. lisn upsselaayfr/ccompettiooo////
22


Reviews:
[{"review_id": "oEDBgEKMBw", "reviewer": "Reviewer_LtSJ", "summary": "The authors propose a method to adapt foundation models to the task of visual place recognition, arguing that the object-centric focus of the training of foundation models does not align with the background/static-object attention needed in the VPR task. Thus, they propose, instead of fine-tuning the visual transformers, to extend the transform block with two mechanisms (added MLP) that operate as adapter for the global feature computation. Another local adaptation is done for reranking, similar to geometric verification of retrieved images.", "questions": "- How would the adapters work with other (smaller) models?\n- What are the reasons, and interpretaions (with evidences) of why the proposed adapters work?\n- How the adapter parameter space influences the improve of performance, and how does it relate with 'smaller' backbones?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ concept is straightforward: the idea behind the adaptation, and the motivation why it is needed to adapt from foundation models is clearly presented.\n+ experiments on relevant datasets: results are very good, improving on existing approaches\n+ the paper is easy-to-read", "weaknesses": "_very limited or non-existing insights_\nthe paper is built solely around overcoming the results of existing methods, while insights and evidence-based contributions are not provided. After reading the paper I am left with a big question: \"why this method works and what do I learn that can perhaps use to design methods in different applications?\". Would the adaptation work also for CNNs pre-trained on ImageNet (as they share the same object-centric bias)? \nI would expect (at ICLR) a thorough analysis of reasons why the performance are much higher, what the implications of doing adaptation are, and what are the real scientific contributions behind this work (not just that the method gets better results than sota methods).\n\n_design choice weakly explained_\nno motivations or justification of why the adapters are designed in a certain way, and what the difference w.r.t. existing approaches for adaptation of foundation models are. What is the hypothesis behind this kind of design, and what explanations can be given (with experimental evidence) about their working principle?\n\n_data-efficiency not elaborated upon_\nas data-efficiency is a key argument about using foundation models, the authors indeed mention it but do not provide substantiable experimental evidence about how it benefits their approach. \n\n_parameter difference not well-analyzed_\nThe adaptation mechanisms proposed are still requiring the fine-tuning of +50M parameters, which is much more than other methods train. Summed up with the +300M parameters of the foundation backbones, these models account for much more capacity than whatever method used previously. The authors do not provide any discussion about this point, or experiments with adaptation of other (smaller) models.\n\nA missing reference:\nLeyva-Vallina et al., Data-Efficient Large Scale Place Recognition With Graded Similarity Supervision; CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I1dLlCfPBc", "reviewer": "Reviewer_S5xw", "summary": "1.\tVisual place recognition (VPR) is a fundamental task for applications in robot localization and augmented reality. This paper aims to bridge the gap between the tasks of pre-training and VPR, thus fully unleashing the capability of pre-trained models for VPR.\n2.\tThe authors introduce a hybrid adaptation method to get both global features for retrieving candidate places and dense local features for re-ranking. \n3.\tMeanwhile, a novel local feature loss is designed to guide the production of proper local features for local matching without geometric verification in re-ranking.", "questions": "1.\tWhy is L2 distance used to measure global feature similarity, but dot product used to calculate local feature similarity?\n2.\tIs it feasible to re-rank top-k candidate images directly using coarse patch tokens from ViT? and how is the performance?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThis work is novel and solid. The proposed SelaVPR is a well-designed method and achieves very fast two-stage retrieval.\n2.\tThis paper makes two technically strong contributions: closing the gap between the pre-training and VPR tasks, and outputting proper dense local features for VPR task using DINO v2. The extensive ablation experiments and visualization results show that the proposed method well adapts the pre-trained model to the VPR task. The produced dense local features also perform well in local matching re-ranking.\n3.\tThis method achieves better performance than the SOTA methods with less training data and training time.", "weaknesses": "1.\tThis work adapts the pre-trained model to the VPR task. The global and local features produced by this hybrid adaptation seem to be useful for more visual tasks. Expanding the use of this method can make the contribution of this paper more obvious.\n2.\tThe clarity of the paper could be further improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sVeQqVGpqA", "reviewer": "Reviewer_mSaw", "summary": "The paper presents a hybrid global and local adaptation method to adapt pre-trained foundation models to two-stage visual place recognition. The global adaptation is achieved by adding parallel and serial adapters in each ViT block. The local adaptation is implemented by adding up-sampling layers after ViT backbone to produce dense local features. A novel mutual nearest neighbor local feature loss is proposed to train the local adaptation module. This architecture achieves fast two-stage place retrieval and outperforms several SOTA methods. It is ranked first on the MSLS challenge leaderboard.", "questions": "1. Will re-ranking more candidate places achieve better performance or hurt the results?\n2. This work finetunes the models on the MSLS dataset and further finetunes them on Pitts30k to test on Pitts30k and Tokyo24/7, which is the same as R2Former. However, the R2Former work provides the result on Pitts30k of the model that only trained on MSLS, which can prove the model's transferability to the domain gap. Can the proposed SelaVPR still outperform R2Former on Pitts30k using only MSLS for training?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and presents a good overview of the related work.\nThe approach is simple and easy to follow. \nThe experimental datasets are sufficient (conducted on 6 VPR benchmark datasets) and the results are excellent (outperform previous SOTA methods by a large margin).\nThis method can bridge the gap between the tasks of model pre-training and VPR using only a small amount of training data and training time. The two-stage retrieval runtime on Pitts30k is less than 0.1s (about 3% of the TransVPR method). This makes contributions to use pre-trained foundation models for real-world large-scale VPR applications.", "weaknesses": "This method achieves significantly better performance than other methods on several VPR datasets, and the authors qualitatively demonstrate some challenging examples. However, the motivation of the proposed method is not demonstrated well.  In particular, the gap of the tasks of model pre-training and VPR is not very clear to me. In addition, this paper does not show failure cases, which can inform future research in VPR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m2QE36Fkd9", "reviewer": "Reviewer_Gibf", "summary": "In this paper the authors propose a global-local adaptation method to seamlessly adapt the pre-trained DINOv2 model to produce both global and local features for the visual place recognition task. The proposed feature representation can focus on discriminative landmarks and eliminate dynamic interference. The output local features are used in local matching for re-ranking to further boost performance. This method outperforms other state-of-the-art methods on multiple datasets with high computational efficiency.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper is very well written and clearly presented.\n2.\tRecapitulation of related work is good.\n3.\tThe authors design a hybrid adaptation method to seamlessly adapt pre-trained foundation model to the VPR task. The method is novel and interesting, and the authors did not over-complicate it.\n4.\tThe experimental results are really good.", "weaknesses": "1.\tPitts250k is also a common VPR dataset. The proposed approach has shown excellent results on multiple benchmark datasets. Providing the results on the Pitts250k dataset might make the experiment more complete.\n2.\tRe-ranking top-100 candidates seems a common setting for two-stage VPR methods. However some works also show the performance with different numbers of re-ranking candidates [1], which can help other researchers choose the optimal number of candidates when using this method. I think it is also necessary to show the performance of different numbers of candidates.\n\n[1] Zhu, Sijie, et al. \"R2former: Unified retrieval and reranking transformer for place recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)