SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As depicted in Table 1, our method notably surpasses SP, GCRANSAC [5], ASMK, and VLAD across all settings. It significantly outperforms SP by  $8.6\%$  and  $11.7\%$  on the challenging ROxford under middle and hard settings, respectively. This underlines the feasibility and effectiveness of our novel adaptation of the topological model in RANSAC. Given the widespread usage of SP [26, 33, 35, 34] and its competitiveness with other instance recognition approaches in terms of accuracy [28, 7, 24], this improvement is particularly noteworthy. Our results represent the top non-fine-tuning performance on ROxford and RParis. While GCRANSAC excels at predicting homography or fundamental matrices [5], it does not surpass standard RANSAC in our recognition task. This may be due to the spatial model's incompatibility with recognition tasks, suggesting that enhancing spatial-based RANSAC may not improve landmark
Table 2: Results (\% mAP) of HP on the ROxford.
Method	M	H
SP (hop1)	80	61
Our TP (hop1)	81	63
SP (hop2)	85	69
Our TP (hop2)	86	71
recognition performance. In an ablation study conducted to assess the impact of using SP to filter the hypothesis set, we found that incorporating SP surprisingly led to a decrease in final accuracy. Despite our Python-based method averaging 1.23s per image pair, slower than C-implemented SP at 0.53s, we anticipate considerable speed improvements with a C-based implementation of our method.
Our TP outperforms notable pre-trained models, including CLIP [30] and Superpoint [10], without requiring any training. Although CLIP delivers fair performance, it falls short of both SP and TP. These results underscore the competitive edge of the hypothesis-testing strategy for landmark recognition in highly noisy situations, even against large-scale pre-training methods. Despite Superpoint's superiority to SIFT in SP and its enhancement of inference performance, it remains less effective than our TP. These results attest to our TP's ability to circumvent the inherent issues of the spatial model for recognition.
Our novel TP method demonstrates competitive results against the renowned GeM [29], fine-tuned on the large cleaned Google landmarked dataset (GLD)[46], without any pre-training or fine-tuning. This comparison, while unconventional due to TP being a reranking method and GeM an initial ranking method, is meaningful. GeM, a state-of-the-art retrieval baseline, marked a milestone as the first metric learning method to surpass non-fine-tuned SP approaches [28], following a series of improvements from selective search[41] to R-MAC [43]. Since then, non-fine-tuning techniques have seen limited advancements. However, TP breaks this trend, achieving
Figure 3: SP results, TP results, and some TP steps for verifying a correct image pair (upper) and a wrong image pair (down) using SIFT features. The threshold  $\alpha$  is set as 0.2.
comparable results to GLD-trained GeM. This is not to say SIFT is enough for this task but
Figure 4: Hypergraph propagation results using traditional SP (upper) and our method (down). In each triplet, the first image and third image are different. Red lines show SP's wrongly connected local features between the first image and the third image. The orange and green regions show the verified HRs between each image pair. Our method correctly separates the local features of the first and the third images. HR is the union of many verified patches. We only show the outline of key points in HR for ease of observation.
to emphasize the potential of hypothesis testing when combined with a topological model for non-fine-tuning image retrieval.
We acknowledge that advancements in fine-tuning features for ASMK or SP, such as SuperFeature and DELG [45, 7], have emerged recently. While it is possible to train features for our TP, we leave this for future work, as this work mainly focuses on the non-fine-tuning retrieval. However, to show our method's generalization capabilities, we still apply TP to fine-tuned DELG local features and Hypergraph Propagation (HP) [3]. HP is a fast reranking method that combines verification and diffusion on local features. It runs the heavy verification offline and thus achieves fast, low-memory, and high-accuracy reranking online. Table 2 demonstrates that our method outperforms SP on HP, irrespective of propagating among hop1 or hop2 neighbors; hop1 or hop2 is the truncation hyperparameter in diffusion [3]. This improvement is noteworthy, considering it does not necessitate any additional training. This is the SOTA retrieval performance on ROxford to the best of our knowledge.
We understand that readers may have concerns about the practical value of a RANSAC-based method in retrieval, given its slower processing time compared to using dot products with learned embedding vectors. However, it is important to note that if RANSAC can provide reliable labels without fine-tuning, these labels can be used directly to train the embedding space for any database, eliminating the need for a specially collected fine-tuning set. As we shown in Table 2, it can also directly improve HP accuracy. Because verification in HP is offline, the slower speed of RANSAC is acceptable. Furthermore, SP is employed not only in retrieval but also in other tasks on mobile devices where model size is crucial, such as loop-closure detection. Our TP, which shares the lightweight nature of SP without involving millions of parameters like models such as ResNet [15], offers enhanced robustness and accuracy. Most importantly, our method is highly explainable, which we will discuss in the following section.
4.3 Qualitative result
In addition to its accuracy, our method exhibits high explainability, a crucial aspect for real-life applications. The explainability of our method can be understood in three aspects:
1. We can observe how the HRs are constructed step by step, find a correspondence patch for each patch in HR, and check how each candidate patch pair is accepted or rejected by the fovea function  $F$ , as shown in Figs. 2 and 3. These characters offer a clear understanding of how our method works.
2. The TP detected HRs and matched patches are easy to comprehend, whether using SIFT (Fig. 3) or DELG (Figs. 1 and 4). As shown in the upper row of Fig. 3, although SP can also give a high inlier number for the correct image pair, its found matchings can be wrong. Our TP gives more reasonable results. This issue of high inlier counts with incorrect matchings is also observed in DELG, suggesting that fine-tuning can improve the similarity of local features for the same object, but cannot guarantee the accuracy of their locations. Additional examples are provided in the supplementary material. Fig. 4 compares the qualitative performance of SP and our TP on HP. The first query and third image in each triplet do not depict the same building. Due to SP's erroneous matching of some local features (indicated by red lines), a search engine using SP mistakenly identifies the third
image as a positive match. In contrast, our method accurately matches the correct similar regions between two images and can correctly determine that the third image is unrelated to the query, thereby enhancing retrieval performance.
Figure 5: The false positive (FP) and false negative (FN) cases of our method. We draw the SIFT feature locations in the largest detected HRs for FP cases and the ratio test results for FN cases. Check more examples in the supplementary material. Check all the FP and FN cases from this link.
3. Our approach maintains explainability, even in cases of false positives and negatives, offering valuable insights into the effectiveness and potential issues at each stage. We have visualized all false negatives (FN) and positives (FP) in Fig. 5. For a more comprehensive review, we provide a link for readers to explore further. Interestingly, without context, many FP cases are challenging for the authors to distinguish, signaling the effectiveness of our fovea function F in generating reasonable patch similarity results. Upon examining FN cases, we recognize areas for enhancement. The primary issues lie in incorrect ratio test results and significant size changes. While our method considers size changes, it struggles with handling extreme variations. An easy solution is to multi-rescale each image and traverse all possible matching scales.
We refrained from this to avoid significantly reducing speed and to maintain a fair comparison with SP. However, future work could potentially apply this strategy or devise a better sampling method to manage extreme scale changes, thereby improving performance.
5 Discussion about the task value
This paper aims to advance explainable image retrieval in scenarios where a fine-tuning set is unavailable. This pursuit carries significant weight as acquiring a large fine-tuning set can often be impractical or costly, especially in open-world or private settings. Therefore, research in this domain has the potential to enhance the utility of AI systems across a variety of real-world contexts.
Our exploration into non-fine-tuned retrieval is further motivated by the observed gap between human-level performance and that of the state-of-the-art (SOTA) fine-tuned models [4, 40] on challenging datasets such as ROxford/ RParis [28] and PROxford/ PRParis [3]. Notably, these datasets provide invaluable user study results, as each positive image pair is verified to be identifiable by humans without the need for contextual visual information [28, 4]. Our recent study shows that annotators who are not familiar with European buildings can recognize, localize, and segment the target landmark of all the positive image pairs in ROxford and RParis. We propose the pixel-retrieval task and its first benchmarks PROxford and PRParis, to evaluate the pixel-level recognition [4]. Interestingly, humans achieve this remarkable recognition ability without extensive in-domain training, such as learning from the Google Landmarks Dataset (GLD) [46]. Yet, even the best fine-tuned models [7, 40] trained on the vast GLD with 4.1 million images have yet to reach human-level accuracy on ROxford/RParis and PROxford/PRParis. This discrepancy prompts us to question how humans excel in such recognition tasks without large-scale fine-tuning.
The process of recognition is a confluence of perception and cognition. The superior instance recognition of humans in the ROxford dataset might be more cognitively inclined, though concrete evidence remains elusive. Cognition entails reasoning based on existing knowledge for decision-making and the discovery of new knowledge [47, 13, 37]. When summarizing notable human instance recognition experiments [6, 16, 39], Greyson [1] defines perception as the direct link between an instance and its visual features, while cognition involves understanding the object's changes in viewpoint or state (e.g., background and illumination) [47, 1, 11, 22]. Machine learning methods that align with this definition of cognition include ASMK [42] and SP [26], which further motivates us to explore improvements to SP without fine-tuning. Future research discerning the nuances between perception and cognition could potentially enhance non-fine-tuning methods for real-world applications.
Our study highlights the limitations of the SPatial verification (SP) method, particularly its reliance on a spatial model for recognition. As a potential alternative, we propose the use of topological common sense, which significantly outperforms SP and achieves unprecedented performance in non-fine-tuning instance recognition. This underscores the potential of hypothesis-testing inference.
The inspiration for our saccade and fovea function comes from the human process of comparing two newly seen images. In this sense, our topological approach aligns more closely with the human visual system than the spatial approach. While humans don't compute transformation matrices like SP when comparing images, they intuitively understand the topological relations among observed objects—for instance, recognizing that a hand is connected to an arm. Notably, topological information is vital to the human visual system, as it is consistently preserved even when visual signals undergo significant deformation during transmission from the retina to the lateral geniculate nucleus and visual cortex [44, 18].
Importantly, our method doesn't compete with existing fine-tuning methods but complements them. It can directly integrate learned features, and future work could even fine-tune features specifically for this topological approach.
While our results are promising, they also pave the way for future exploration. Subsequent work could investigate the incorporation of more complex common sense concepts, both explicit, like our topological model, and implicit, derived from pre-training. Enhancements in these areas could potentially boost the robustness and adaptability of our method. Further study into the convergence properties of our method and refinement of the hypothesis sampling strategy are also warranted. Through addressing these areas, we aspire to advance object recognition and highlight the potential of hypothesis-testing approaches in artificial intelligence.
We would like to thank the anonymous reviewers for their valuable comments and insightful suggestions. Sung-Eui Yoon is the corresponding author of this paper. This work was supported by the Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (RS-2023-00237965, Recognition, Action and Interaction Algorithms for Open-world Robot Service) and the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2023-00208506).
[1] Greyson Abid. Recognition and the perception–cognition divide. Mind & Language, 37(5):770–789, 2022. 9
[2] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM, 54(10):105-112, 2011. 1, 2
[3] Guoyuan An, Yuchi Huo, and Sung-Eui Yoon. Hypergraph propagation and community selection for objects retrieval. Advances in Neural Information Processing Systems, 34, 2021. 7, 8, 9
[4] Guoyuan An, Woo Jae Kim, Saelyne Yang, Rong Li, Yuchi Huo, and Sun-Eui Yoon. Towards content-based pixel retrieval in revisited oxford and paris. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20507-20518, 2023. 9
[5] Daniel Barath and Jií Matas. Graph-cut ransac. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6733-6741, 2018. 6, 7
[6] Timothy F Brady, Talia Konkle, George A Alvarez, and Aude Oliva. Visual long-term memory has a massive storage capacity for object details. Proceedings of the National Academy of Sciences, 105(38):14325-14329, 2008. 9
[7] Bingyi Cao, André Araujo, and Jack Sim. Unifying deep local and global features for image search. In European Conference on Computer Vision, pages 726-743. Springer, 2020. 1, 2, 3, 6, 7, 8, 9
[8] Sunglok Choi, Taemin Kim, and Wonpil Yu. Performance evaluation of ransac family. Journal of Computer Vision, 24(3):271-300, 1997. 2
[9] Konstantinos G Derpanis. Overview of the ransac algorithm. Image Rochester NY, 4(1):2-3, 2010. 2