Title: Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual Place Recognition

Image Descriptors

Aggregation
Abstract: Visual place recognition (VPR) is typically regarded as a specific image retrieval task, whose core lies in representing images as global descriptors. Over the past decade, dominant VPR methods (e.g., NetVLAD) have followed a paradigm that first extracts the patch features/tokens of the input image using a backbone, and then aggregates these patch features into a global descriptor via an aggregator. This backbone-plus-aggregator paradigm has achieved overwhelming dominance in the CNN era and remains widely used in transformer-based models. In this paper, however, we argue that a dedicated aggregator is not necessary in the transformer era, that is, we can obtain robust global descriptors only with the backbone. Specifically, we introduce some learnable aggregation tokens, which are prepended to the patch tokens before a particular transformer block. All these tokens will be jointly processed and interact globally via the intrinsic self-attention mechanism, implicitly aggregating useful information within the patch tokens to the aggregation tokens. Finally, we only take these aggregation tokens from the last output tokens and concatenate them as the global representation. Although implicit aggregation can provide robust global descriptors in an extremely simple manner, where and how to insert additional tokens, as well as the initialization of tokens, remains an open issue worthy of further exploration. To this end, we also propose the optimal token insertion strategy and token initialization method derived from empirical studies. Experimental results show that our method outperforms state-of-the-art methods on several VPR datasets with higher efficiency and ranks 1st on the MSLS challenge leaderboard. The code is available at https://github.com/lu-feng/image.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel global descriptor for visual place recognition made of the concatenation of \"learnable tokens\" / registers stacked to the visual tokens of a visual-transformer-network.\nThe \"regular\" visual tokens are discarded.\n\nThe design of such global descriptor contrasts with existing architectures that follow a 2 step approach: extract visual features with a backbone and aggregate them with an aggregator network.\nOne issue that can arise then is any domain shift in the backbone feature extract hinders the performance of the aggregation network.\nDefining the global descriptor from the backbone network only addresses this issue.\n\nThe additional tokens can be stacked to the regular ones at different levels of the transformer network.\nThe paper motivates and demonstrates through experiment that the optimal results are obtained when the tokens are inserted after the last frozen layer.\nThe global descriptor is best when the learnable tokens are initialized based on the frozen layer outputs: clustering the frozen layer's feature space and using the centroid to initialize the learnable tokens leads to the best results.\n\nThe proposed global descriptor is evaluated following the visual place recognition guidelines (evaluation datasets, metrics, relevant and exhaustive baselines).\nThe results support the claim of the paper in that the proposed aggregation-free and register-based descriptors are powerful representations for visual place recognition and perform better or on than other transformer-based descriptors.\n\nSTRENGTHS AND WEAKNESSES:\nS1. Novel and interesting design of the global descriptor that demonstrate how visual transofrmers can learn global descriptors through register tokens.\nThe design is well motivated and the experiments demonstrate that it is suitable for place recognition.\n\nS2. Exhaustive experimental evaluation that support the claim of the paper.\n\nS3. Relevant related work.\n\nS4. Well written paper.\n\n\nW1. The paper claims to address the \"limitations of the previous backbone-plus-aggregator\" but the justifications are not sound so this claim does not hold.\nIt is best to simply state that the proposed descriptors is novel because it learns global tokens within the transformer architecture, which is a good contribution on its own.\n\nL81: \"This aggregator-free VPR paradigm addresses the limitations of the previous backbone-plus-aggregator paradigm by:\"\n- a) \"unifying feature extraction and aggregation toward global optimization, and\"\n    - Aggregation-based method enable global optimization by making the aggregation differentiable (e.g. NetVLAD, BoQ, ...) so this sentence is incorrect.\n- b) \"progressively implementing aggregation in cascaded transformer blocks, thus achieving the correction and refinement of image representations.\"\n    - \"correction and refinement of image representations\" is undefined. If this means improving the feature representations with each layers, all networks (CNN, VIT ...) do so\n\nL91: \"This new paradigm effectively and efficiently overcomes the limitations of the previous works\".\n  - Same comments as before: the design of previous works, i.e., aggregation-based descriptors, is not a limitation per se\n\nQUESTIONS:\n- In Eq3, is there an interpretation to the term $Q_z K_a^T V_a$ that also reprensets interactions between the aggregation token / registers and the image embeddings?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact of their work (in the supp. material)\n\n### Misc. Comments\n\nUndefined terms:\n- L46: \"global optimiality\" of a global descriptor\n- L48: \"correction or refinement\" of a global desctiptor. Is this supposed to mean \"finetuned\"?\n\n\nUnjustified claims:\n- L50 \" With regard to specific methods like NetVLAD, there exist some problems, such as [...] the loss of position information of the original patch features.\". Why is that an issue? If this is related to re-ranking, there are works that enable re-ranking from the global descriptors only (e.g. \"Global Features are All You Need for Image Retrieval and Reranking\", ICCV23)\n\n- L354: \"These centers, obtained via unsupervised clustering of descriptors from randomly sampled training images, tend to capture common visual patterns (e.g., buildings, vegetation, vehicles, pedestrians, etc).\"\n   - how is it proved that these descriptors capture human level semantics?\n\nReferences:\n- L103: BoW's correct reference is not [4] but \n\nSivic, and Zisserman. \"Video Google: A text retrieval approach to object\nmatching in videos.\" Proceedings ninth IEEE international conference on\ncomputer vision. IEEE, 2003.\n\n- L30: [52] is a generic place recognition method so it is not a relevant citations related to the existing challenges in VPR", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ImAge, a framework for visual place recognition that eliminates the need for explicit aggregators. Instead, it leverages the self-attention mechanism of a transformer backbone by inserting learnable aggregation tokens at a specific depth.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\nThe paper is well-written, with clearly articulated ideas that are easy to follow.\n\nWeaknesses：\n\n[1] The paper tends to rely heavily on storytelling, while the proposed method is essentially a form of prompt tuning. Compared to existing classical prompt tuning approaches (e.g., VPT [R1]), it lacks substantive innovation, offering only engineering-level adjustments to the number, position, and initialization of prompt tokens. Applying this method to visual place recognition (VPR), a classification task, is fairly intuitive. However, the authors only compare their method against other VPR-specific approaches, deliberately avoiding comparisons with established prompt tuning methods in the broader computer vision community.\n\n[2] The comparative experiments are not sufficiently rigorous. As shown in Table 6, the model's performance is highly sensitive to the number of aggregation tokens. Nevertheless, the ablation studies presented in Tables 4, 5, and 7 are only conducted under the setting where the number of agg tokens is 8. Therefore, the conclusions drawn from these ablations may lack generality and robustness.\n\n[3] The proposed initialization strategy for cluster centers is inspired by NetVLAD, but its rationale is not well justified. In fact, the mechanism for computing global representations in the two methods differs fundamentally: NetVLAD constructs global descriptors using residual statistics between patch features and cluster centers, whereas ImAge directly flattens the cluster centers to form the global representation. This conceptual gap is not clearly explained, which undermines the credibility of the design.\n\n[4] Moreover, the proposed method is highly sensitive to multiple hyperparameter choices, including the insertion strategy, initialization, and number of aggregation tokens. Due to the lack of theoretical justification, determining these key hyperparameters in new practical scenarios remains a challenge, which may significantly affect the model’s stability and generalization.\n\nReference：\nJia, Menglin, et al. \"Visual prompt tuning.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\nQUESTIONS:\nPlease see the above weaknesses.\n\nLIMITATIONS:\nThe discussion of the limitations in this paper is insufficient and lacks transparency. The authors claim that “ImAge is broadly applicable to a wide range of image retrieval tasks,” but this assertion appears somewhat inverted in logic. In fact, prompt tuning methods have already been widely applied to mainstream vision tasks such as image retrieval, classification, and object detection. The proposed ImAge method essentially transfers the prompt tuning strategy to the relatively niche task of visual place recognition (VPR). However, the authors do not provide a systematic comparison between ImAge and existing prompt tuning approaches in other vision domains, limiting their analysis and validation to the VPR setting alone.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ImAge, an innovative implicit aggregation paradigm for visual place recognition (VPR) that leverages the self-attention mechanism of transformers to eliminate the need for explicit aggregators. \nThis approach addresses critical limitations of traditional \"backbone+aggregator\" architectures, such as suboptimal two-stage optimization and loss of positional information. \n\nThe core innovation lies in:\n1. Introducing learnable aggregation tokens that interact with patch tokens via self-attention within transformer blocks.\n2. Proposing an optimal token insertion strategy and initialization method to enhance representation robustness.\n3. Demonstrating state-of-the-art (SOTA) performance across diverse VPR benchmarks with higher efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Paper writing is good and clear to show the motivation and comparison.\n2. Thorough ablation studies and fair comparisons validate the method's robustness and efficiency. The approach ranks 1st on the MSLS challenge leaderboard (Fig. 7), demonstrating real-world applicability.\n3. The visualization of the attention weights of agg tokens to patch tokens shows the convincing training proces.\n\nWeakness:\n1. No GPU memory and computational efficiency comparison has been shown in quantitative experiments while the authors repeatedly stressed “leading to significant GPU memory and computational overhead”.\n2. Although this paper has shown comparison of different aggregators based on standard transformer-backbones ViT and DINO-V2，I am curious about the results of transformer-based VPR methods with agg tokens. For example, R2Former.\n3. Such agg tokens are not novel enough for transformer/VLM era. Of course, it's kind of novel for VPR.\n\nQUESTIONS:\n1. Plz provide the memory and computational efficiency comparison results.\n2. Plz provide more ablation studies on ViT-based VPR methods to show the additional performance gains and show more significance of this work in VPR field.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach to adapt vision transformers for visual place recognition. Specifically, the novelty focuses on the pooling/aggregation mechanism.\nRather than using an additional separate pooling mechanism on top of the backbone, this work proposes to use additional \"aggregation\" tokens that use the same self-attention mechanism already in place within the transformer. The output embeddings of these tokens can then be used as image descriptors for place recognition.\nThis approach outperforms previous visual place recognition approaches.\n\nSTRENGTHS AND WEAKNESSES:\n(+) = strength, (-) = weakness\n\n**Quality**:\n1. (+) The solution is technically sound, and the design choices are well validated through ablation studies.\n2. (+) The results appear strong compared to existing methods and cover multiple datasets.\n3. (+) The approach remains very simple by using an existing mechanism in transformer models, yet is very powerful.\n4. (-) I find the way some motivations, properties, and results are presented to be potentially misleading:\n- 4.1. While I agree in the value of this work, I struggle to understand how it resolves the claimed issues about standard two-stage approaches. The authors claim that two-stage approaches may not achieve global optimality, and that specific approaches such as NetVLAD might suffer from cluster centres learned on the training set not being suitable for the test set. However, two-stage approaches can also be fully differentiable, and I do not see why ImAge would be more likely to achieve global optimality. Moreover, I also do not see any reason or results indicating that ImAge would not also suffer from differences between train and test settings.\n- 4.2. The claims on the approach being aggregator-free are a bit misleading. While I agree this could be considered an implicit aggregation mechanism, as it is already part of the original model, I still believe that calling the method \"aggregator-free\" is not completely correct and confusing. Similarly, it doesn't really have 0 aggregator parameters, as the shared attention weights are fine-tuned to learn the aggregation. \n- 4.3. While the results in Table 2 are interesting, they are not based on a fair comparison, as mentioned in the text. I believe this should be made clearer by specifying it in the caption as well.\n- 4.4. The results of figures 4, 5, and 6 do not seem very representative of overall performance. The baseline results on the datasets presented were still strong, and usually just slightly below the performance of ImAge, and it is not specified how these examples were selected. As such, I believe these figures might only be highlighting cherry-picked samples where ImAge succeeded and the baselines did not, but I do not believe we can make many conclusions based on this. If the samples are actually randomly chosen, it should be clarified. Based on the overall metrics, there are likely cases where ImAge also fails and some baselines succeed. I would expect a more complete quantitative study on the performance with different types of scenarios (e.g., dynamic objects, viewpoint variations, etc.) if the authors want to claim that ImAge consistently retrieves the correct results where others don't (Fig. 4 caption).\n- 4.5. In Table 2, CricaVPR and SuperVLAD are not included in the bold/underlined results only on Pitts30k, but it would be more consistent to either not include them on any datasets, or include them on all.\n\n**Clarity**: \n1. (+) The introduction of the idea of the paper relative to existing works is very clear, and the figures illustrating the approach are easy to understand.\n2. (-) The text in the paper is very dense, with figure captions sometimes getting really close to the text (e.g., Fig. 3). This negatively affects how easy it is to read the paper, and breaking down long paragraphs with more structure would help improve clarity.\n3. (-) The details of the k-means clustering used for initialising tokens remain unclear to me (see question 2).\n\n**Significance**:\n1. (+) I believe that the results from this paper can affect what future research on the topic of visual place recognition will focus on, potentially shifting away from designing new aggregation mechanisms and focusing on best adapting existing pre-trained models and relying on their existing aggregation mechanisms.\n\n**Originality**: \n1. (+) While the technical solution proposed relies on existing mechanisms, the strength of the results combined with the simplicity of the approach makes this work original enough in the context of the visual place recognition literature.\n\nQUESTIONS:\n1. Could the authors please address my concerns on misleading statements in the paper detailed in my comment on quality #4? These include how the solution tackles claimed limitations of two-stage approaches, clarifications on the description of the method being “aggregator-free” and having “0 aggregator parameters”, experimental details in the caption of table 2, and the conclusions drawn from figures 4, 5 and 6.\n2. Can the authors please clarify how the clustering for initialisation works? Specifically, which features are extracted and used for clustering? Is this a limitation for transferring to new settings?\n3. Could the authors please discuss potential limitations from moving away from a two-stage approach? I would expect that this would lead to more challenges if one wants to keep the backbone frozen (e.g., to get good generalisation in new domains, as discussed in [40], or if the backbone is extremely expensive to fine-tune).\n4. Can the authors please expand more on the statement from line 244-245: \"each added agg token representing an object category\". It is unclear to me why we would be able to make that assumption, and why tokens would end up with separated categories to represent.\n5.  Can the authors please clarify why the proposed aggregation tokens are not classified as \"output-oriented tokens\"? They seem to follow the same purpose as class tokens, just with different design choices on location and initialisation.\n\nLIMITATIONS:\nThe authors discuss some limitations in the appendix, but not in the main paper. Moreover, some additional limitations would be interesting to discuss (see my question 3).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}