Learning Super-Features for Image Retrieval
Authors: Philippe Weinzaepfel
Thomas Lucas
Diane Larlus
Yannis Kalantidis
Venue: ICLR 2022 Poster (2022)
Keywords: image retrieval
landmark retrieval
mid-level features

Abstract: Methods that combine local and global features have recently shown excellent performance on multiple challenging deep image retrieval benchmarks, but their use of local features raises at least two issues. First, these local features simply boil down to the localized map activations of a neural network, and hence can be extremely redundant. Second, they are typically trained with a global loss that only acts on top of an aggregation of local features; by contrast, testing is based on local feature matching, which creates a discrepancy between training and testing. In this paper, we propose a novel architecture for deep image retrieval, based solely on mid-level features that we call Super-features. These Super-features are constructed by an iterative attention module and constitute an ordered set in which each element focuses on a localized and discriminant image pattern. For training, they require only image labels. A contrastive loss operates directly at the level of Super-features and focuses on those that match across images. A second complementary loss encourages diversity. Experiments on common landmark retrieval benchmarks validate that Super-features substantially outperform state-of-the-art methods when using the same number of features, and only require a significantly smaller memory footprint to match their performance. Code and models are available at: https://github.com/naver/FIRe.

Published as a conference paper at ICLR 2022
LEARNING SUPER-FEATURES FOR IMAGE RETRIEVAL
Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis NAVER LABS Europe, Grenoble, France
ABSTRACT
Methods that combine local and global features have recently shown excellent performance on multiple challenging deep image retrieval benchmarks, but their use of local features raises at least two issues. First, these local features simply boil down to the localized map activations of neural network, and hence can be extremely redundant. Second, they are typically trained with a global loss that only acts top of an aggregation of local features; by contrast, testing is based on local feature matching, which creates a discrepancy between training and test- ing. In this paper, we propose a novel architecture for deep image retrieval, based solely on mid-level features that we call Super-features. These Super- cuper-features are constructed by an iterative attention module and constitute an ordered ser in which each element focuses on a localized and discriminant image pattern. For training. they require only image labels. A contrastive loss operates directly at the level of Super-features and focuses on those that match across images. A second comple- mentary loss encourages diversity. Experiments on common landmark retrieval benchmarks validate that Super-features substantially outperform state-of-the-an methods when using the same number of features, and only require a significantly smaller memory footprint to match their performance.
Code and models are available at: https: /ttg/gttthbbbtt http://ggtttgttttett/taaaaeaette www..//////enneeeeeeeeeeee
INTRODUCTION
Image retrieval is a task that models exemplar-based recognition, i.e. class-agnostic, fine-grained understanding task which requires to retrieve all images matching a query image over an (often very large) image collection. It requires learning features that are discriminative enough for a highly detailed visual understanding but also robust enough to extreme viewpoint/pose or illumination changes. A popular image retrieval task is landmark retrieval, whose goal is to single out pictures of the exact same landmark out of millions of images, possibly containing different landmark from the exact same fine- tne-grained class (e.g. 'gothic-era churches with twin bell towers').
While early approaches relied handcrafted local descriptors, recent methods use image-level (global) or local Convolutional Neural Networks (CNN) features, see Csurka & Humenberger (2018) for a review. The current state of the art performs matching re-ranking using CNN-based local features (Noh et al., 2017; Cao et al., 2020; Tolias et al., 2020) and only learns with global (i.e. image-level) annotations and losses. This is done by aggregating all local features into global representation on which the loss is applied, creating a discrepancy between training and inference.
Attention maps from modules like the ones proposed by Vaswani et al. (2017) are able capture intermediate scene-level information, which makes them fundamentally similar to mid-level fea- tures (Xiao et al., 2015; Chen et al., 2019). Unlike individual neurons in CNNs which are highly localized, attention maps may span the full input tensor and focus more global or semantic pat- terns. Yet, the applicability of mid-level features for instance-level recognition and image retrieval is currently underwhelming;
argue that this is due to the following reasons: generic attention maps are not localized and may fire on multiple unrelated locations; at the same time, object-centric attentions such as the one proposed by Locatello et al. (2020) produce too few attentional features and there is no mechanism to supervise them individually. In both cases, methods apply supervision at the global level, and the produced attentional features are simply not discriminative enough.
In this paper, we present a novel image representation and training framework based solcly on at- tentional features call Super-features. We introduce iterative Local feature Integration Trans-

---

Published as a conference paper at ICLR 2022
Figure I: Super-features attention maps produced by our iterative attention module (LIT) for three images (left), with the first two that match, for five Super-features, They tend to consistently fire on some semantic patterns, egg. circular shapes, windows, building tops (second to fourth columns).
former (LIT), which tailors existing attention modules to the task of image retrieval. Compared to the slot attention (Locatello et al., 2020) for example, it is able to output an ordered and much larger set of features, as it is based on learned templates, and has simplified recurrence mechanism. For learning, we devise loss that is applied directly to Super-features, yet it only requires image- level annotations. It pairs a contrastive loss on a set of matching Super-features across matching images, with decorrelation loss on the attention maps of each image, to encourage Super-feature diversity. In the end, our network extracts for each image a fixed-size set of Super-features that are semantically ordered, i.e., each firing different types of patterns; see Figure 1 for some examples.
At test time, we follow the protocol of the best performing recent retrieval methods and use ASMK (Tolias et al., 2013), except that aggregate and match Super-features instead of local features. Our experiments show that the proposed method significantly outperforms the state of the art common benchmarks like ROxford and RParis (Radenovié et al., 201 8a), while requiring less memory. We further show that performance gains persist in the larger scale, i.e. after adding 1M distractor images. Exhaustive ablations suggest that Super-features less redundant and more discriminative than local features.
Contributions. Our contribution is threefold: (a) an image representation based on Super-features and an iterative module to extract them; (b) a framework to learn such representations, based on a loss applied directly on Super-feature yet only requiring image-level labels; (c) extensive evalua- tions that show significant performance gains over the state of the art for landmark image retrieval. We call our method Feature Integration- based Retrieval or FIRe for short.
BACKGROUND: LEARNING LOCAL FEATURES WITH GLOBAL LOSS
WXXD denote a convolutional neural network (CNN) backbone that encodes an input image E I into a (W x H x D)-sized tensor of D-dimensional local activations over a (W H) spatial grid. After flattening the spatial dimensions, the output of f can also be seen set of T. W. H feature vectors denoted by / [m E DRD : E 1 L); note that the size of this set varies with the resolution of the input image. These local features are then typically whitened and their dimension reduced, process that represent by function a(-) in this paper. Global representations, i.e. image-level feature vectors, are commonly produced by averaging all local features, e.g. via global average max pooling (Babenko & Lempitsky, 2015; Tolias et al., 2016; Gordo et al., 2016).
Let function ʃ
A global contrastive loss for training. Tolias et al. (2020) argue that optimizing global representa- tions is a good surrogate for learning local features to be used together with efficient match kernels for image retrieval. When building their global representation g(M), they weight the contribution of
2

---

Published as a conference paper at ICLR 2022
LIT
0000
first Iteration
Layer norm
Latta
soltmax 1 norm
CNN
0000
o(-)
0
8000
71
Q
DOOO
Q
matching mmges
match fibering
K
0t
MLP
Lupper
CNN
Lastn
L
LIT
FIRe
LIT
(a) The FIRe training process.
(b) The LIT module architecture.
Figure 2: An overview of FIRe. Given pair of matching images encoded by CNN encoder, the iterative attention module LIT (Section 3.1) outputs an ordered set of Super-features. A filtering process keeps only reliable Super-feature pairs across matching images (Section 3.2$ 1), which are fed into Super-feature-leve contrastive loss (Section 3.2$2), while a decorrelation loss reduces the spatial redundancy of the Super- features attention maps for each image (Section 3.2$3).
each local feature to the aggregated vector using its norm:
g(2)) (0()))
g(1)
0(21) Σ
uillz of o(w),
where ||-1|, denotes the 12 norm. Given database where each image pair is annotated as matching with each other or not, they minimize a contrastive loss over tuples of global representations. Intu- itively, this loss encourages the global representations of matching images to be similar and those of non-matching images to be dissimilar. Let tuple (?1, 21+ VT= ...., Vn represent the sets of local features of images (xr xt, ae+,,7 UT ...,, yn ). where z and x+ are matching images (i.e. a positive pair) and none of the images Vi :....... Un is matching with image x (i.e. they are negatives). Let [.]+ denote the positive part and μ margin syper-parameter They define contrastive loss over global representations as:
Lgtobal [s(!t) g((t))2+
+
[E 1900 - 2( 1(1 j-1
(2)
In HOW, Tolias et al. (2020) employ the global contrastive loss of (2) to learn a model whose local features then used with match kernels such as ASMK (Tolias et al., 2013) for image retrieval. ASMK is matching process defined over selective matching kernels of local features; it is much stricter and more precise matching function than comparing global representations, and crucial for achieving good performance. By learning solely using loss defined over global representations and directly using ASMK over local features U1 HOW achieves excellent image retrieval performance.
LEARNING WITH SUPER-FEATURE
Methods using a global loss for training but local features for matching have number of disadvan- tages. First, using local activations as local features leads to high redundancy, they exhaustively cover highly overlapping patches of the input image. Second, using ASMK on local features from model trained with global loss introduces a mismatch between training and testing: the local features used for ASMK are only trained implicitly, but are expected to individually match in the matching kernel. To obtain less redundant feature sets, propose to learn Super-features using loss function that operates directly on those features, and to also use the latter during retrieval; the train/testing discrepancy of the pipeline presented in Section 2 is thus eliminated.
In this section, we first introduce the Local feature Integration Transformer (LIT), an iterative at- tention module which produces ordered set of Super-features (Section 3.1). We then present framework for effectively learning such features (Section 3.2) that consists of two losses: con- trastive loss that matches individual Super-features across positive image pairs, and decorrelation
3

---

Published as a conference paper at ICLR 2022
loss on the Super-feature attention maps that encourages them to be diverse. An overview of the pipeline is depicted in Figure 2. We refer to our approach as Feature Integration- -aased Retrieval or FIRe for short, an homage to the feature integration theory of Treisman & Gelade (1980).
3.1 LOCAL FEATURE INTEGRATION TRANSFORMER (LIT)
Inspired by the recent success of attention mechanisms for encoding semantics from global context in sequences (Vaswani et al., 2017) or images (Caron et al., 2021), we rely on attention to design our Local feature Integration Transformer (LIT), a module that outputs an ordered set of Super-features.
RL*D EN Nxx that takes as input the set of local features 2/ and outputs N Super-features. We define LIT as an iterative module:
Let LIT be represented by function (2))
((U)) = 2
2²= ((;;
(3)
where & denotes the core function of the module applied T times, and 2 RNxd denotes set of learnable templates, i.e. a matrix of learnable parameters. Super-features are progressively formed by iterative refinement of the templates, conditioned on the local features from the CNN.
The architecture of the core function & is inspired by the Transformer architecture (Vaswani et al., 2017) and is composed of dot-product attention function rb, followed by a multi-layer perceptron (MLP). The dot-product attention function d receives three inputs, the key, the value and the query' which are passed through layer normalization and fed to linear projection functions K, V and Q that 1024.
project them to dimensions d,, de and dg. respectively. In practice, we set dz=d,=
The key and value inputs are set the local features un E U across all iterations. The query input is the set of templates 2 = {q E R n = 1 .). It is initialized as the learnable templates 2o for iteration and is set as the previous output of function & for the following iterations. After projecting with the corresponding linear projection functions, the key and the query are multiplied to construct a set of N attention maps over the local features, i.e., the columns of matrix a E TR LR while the L rows on of that matrix can be seen as the responsibility that each of the N templates has for each local feature and is given by²:
α
6 2i-1 ai
e M.
K(ui) Q(gh) vd
E ERINN ai
Min
(4)
-n e.
aLL
The dot product between keys and templates can interpreted as a tensor of compatibility scores between local features and templates. These scores are normalized across templates via softmax function, and are further turned into attention maps by I normalization across all L spatial locations. This is a common way of approximating joint normalization function across rows and columns, also used by Locatello al. (2020). The input value is first projected with V, re-ghiehted with the attention maps and then residually fed MLP to produce the output of function ó:
2¹ o(U;2t=1)
n(U;; 2) (LP(M Φ)) + (1; 2), (1U;; 2) V(U)) a 2. (5)
Following standard image retrieval practice, we further whiten and l2 normalze the output of ((!) to get the final set of Super-features. Specifically, let ((2)) : e Nxd be the raw output of our iterative attention module, we define the ordered set of Super-features as:
{ : 8n, 8n
o(8n)
n= 1,.,,..N? N).
(6)
[o($.)
where, as in Section 2, o(-) denotes dimensionality reduction and whitening. Figure 2 (right) illus- trates the architecture of LIT. Note that all learnable parameters of & are shared across all iterations.
What do Super-features attend to? Each Super-feature in S function of all local features in U. invariant to permutation of its elements, and can thus attend arbitrary regions of the image.
Thee term 'query' has a procise meaning for retrieval, yet, for this subsection only, we overload the term to refer to one of the inpots of the dot- product attention, concistently with the terninology from seminal works on attention by Vaswani et al. (2017).
"The attention maps presented in (4 are technically taken at iteration e, but we omir iteration superscripts for clarity. For the rest of the
paper and visualizations, we use attention maps to refor to the attention maps (4) after the final (T -th) iteration the itcrative module.
3 The NLP function consists of a layer nomn, a fully connected layer with half the dimensions of the features, ReL.U activation and fully-connected layer that projects features hack their initial dimension.
4

---

Published as a conference paper at ICLR 2022
To visualize the patterns captured by Super-features, Figure 1 shows the attention maps of the same five Super-features for three images, including two of the same landmark (first two rows). The type of patterns depends on the learned initialization templates 20; this explains why the Super-features form an ordered set, a property which allows to directly compare Super-features with the same ID. We observe the attention maps be similar across images of the same landmark and contain some mid-level patterns (such as a half-circle on the second column, or windows on the third one).
3.2 LEARNING WITH SUPER-FEATUREE
We jointly fine-tune the parameters of the CNN and of the LIT module using contrastive learning- However, departing from recent approaches like HOW (Tolias al., 2020) DELG (Cao et al., 2020) that use a global contrastive loss similar to the one presented in Eq. (2), we introduce a contrastive loss that operates directly Super-features, the representations we at test time, and yet only requires image-level labels. Given a positive pair (i.e. matching images of the same landmark) and set of negative images, select promising Super-feature pairs by exploiting their ordered nature and without requiring any extra supervision signal. We then minimize their pairwise distance, while simultaneously reducing the spatial redundancy of Super-features within an image.
Selecting matching Super-features. Since we are only provided with pairs of matching images, i.e. image-level labels, defining correspondences at the Super-feature level is not trivial. Instead of approximating sophisticated metrics (Liu et al., 2020; Mialon et al., 2021)-sce Section 5 for a discussion-, we leverage the fact that Super-features are ordered and we select promising matches and filter out erroneous ones only relying on simple, nearest neighbor- based constraints.
For any s E S, let i(s) be the function that returns the position/order or Super-feature ID, i.e. i(s:) i, Vs; E 8. Further let function n(s, 8) arg min || s:ll2 be the function that returns the nearest neighbor E s' 8" be two Super-features from their Super-feature sets 8, 8", respectively. In order for Super-feature pair (s, 8') to be eligible, all of the following criteria must be met: a) 8, of have to be reciprocal nearest neighbors, b) they need to pass Lowe's first-to-second neighbor ratio test (Lowe, 2004) and c) they need to have the same Super-feature ID. Let P be the set of eligible pairs and T the hyper-parameed parameter that controls the ratio test; the conditions above can formally be written
s from set S. Now, given a positive pair of images æ, ax, let
s n(s(,5)
(((s) = i(s') [|ss - "isss'll2 Is'-n(s' ((dd,)((s)))
and We set r = 0.9. Our ablations (Section 4.1) show that all criteria are important. Note that the pair selection step is oon-differentiable and no gradients are computed for Super-features not in P.. We further discuss this in Appendix B.4 and empirically show that all Super-features are trained.
(8,8')E
{ s'= ns, S*)
A contrastive loss on Super-features. Once the S of all eligible Super-feature pairs has been constructed, we define contrastive margin loss on these matches. Let pair (s. s E I be a p of Super-features, selected from a pair of matching images 2, t and let ()) = {n k = 1 n) for j = 1 N be the set of (y ..... ).)) The contrastive
with Super-feature ID) j extracted from negative images loss can be written as:
Lsuper
Σ (8.8
|
8
Σ WI- n nEN((())
(8)
JEB
where μl' is margin hyper-parameter and the negatives for each 8 are the Super-features from all n negative images of the training tuple with Super-feature ID equal to i(s).
Reducing the spatial correlation between attention maps. To obtain Super-features that are as complementary as possible, we encourage them to attend to different local features, i.e. different locations of the image. To this end, we minimize the cosine similarity between the attention maps of all Super-features of every image.
Specifically, let matrix a = [ā āN now be seen as column vectors denoting the N attention maps after the last iteration of LIT. The attention decorrelation loss is given by:
1 N(N -1) 1) Latta (x)
Σ
i E (1, NJ.
2
5

---

Published as a conference paper at ICLR 2022
reci- proc.
ratio test
same
SFM-120k
3OOxoord
(R Paris
0.7
ID
val
med
bard
med
hard
Match ratio
0.6
68.3
64.3
39.8
74.1
52.4
With same ID constraint
79.6
69.2
44.3
79.2
60.9
same constraint
0.5
80.8
70.7
45.1
80.3
61.9
0.4
75.9
63.8
35.1
77.3
56.5
89.7
81.8
61.2
85.3
70.0
0.3
50
100
150
200 Epochs
Ablation on matching constraints: Impact of removing constraints on reciprocity, Lowe's ratio test and the Super-feature ID.
Table 1:
Evolution of the matching quality mea- sured as the ratio matches coming from the positive pair over all pairs with the query in each training tuple.
Figure 3:
In other words, this loss minimizes the off-diagonal elements of the N x N self-correlation matrix of ã. We ablate the benefit of this loss and others components presented in this section in Section 4.1.
Image retrieval with Super-features. Our full pipeline, FIRe, is composed of a model that outputs Super-features, trained with the contrastive Super-feature loss of Eq)(8) and the attention decorre- lation loss of (9). As our ablations show (see Section 4), the combination of these two losses is required for achieving state-of-the-art performance, while adding third loss on the aggregated global features as in (2) does not bring any gain.
4 EXPERIMENTS
This section validates our proposed FIRe approach on standard landmark retrieval tasks. We use the SfM-120k dataset (Radenović et al., 2018b) following the 551/162 3D model train/val split from Tolias et al. (2020). For testing, we evaluate instance-level search on the ROxford (Philbin et al., 2007) and the RParis (Philbin et al., 2008) datasets in their revisited version (Radenović et al., 20), with and without the million distractor set called R IM. They both contain 70 queries, with 4993 and 6322 images respectively. We report mean average precision (mAP) on the Medium (med) and Hard (hard) setups, or the average over these two setups (avg).
Image search with Super-features. At test time, we follow the exact procedure described by Tolias et al. (2020) and extract Super-features from each image at resolutions/scales {2.0. 1.414, 1.0, 0.707, 0.5, 0.353, 0.25). We then keep the top Super-features (the top 1000 unless otherwise stated) according to their L2 norm and the binary version of ASMK with codebook of 65536 clusters. Note that one can measure the memory footprint of a database image x via the number of non-empty ASMK clusters, i.e. clusters with at least one assignment, denoted [C(x)].
Implementation details. We build our codebase on HOW (Tolias et al., 2020)* and sample tuples composed of one query image, one positive image and hard negatives. Each epoch is composed of 400 batches of tuples each, while hard negatives are updated at each epoch using the global features of (1). We train our model for 200 epochs on SIM-120k using an initial learning rate of 3.10-5 and random flipping each epoch, and use an Adam optimizer with weight decay of 10-4. We use ResNet50 (He et al., 2016) without the last convolutional block as backbone (R507). ). For LIT, we use D = d2 = dg d, d 1024. Following HOW, we reduce the dimensionality of features to 128 and initialize o(-) using PCA and whitening before training and keep it frozen. We pretrain the LIT module with the backbone on ImageNet-IK for image classification, see details in Appendix B.6. We use 2l 1.1 in (8) and weight L' Suuper and Lattn with 0.02 and 0.1, respectively.
data augmentation. We multiply the learning rate by factor of 0.99
4.1 ANALYSIS AND ABLATIONS
In this section, we analyze the proposed FIRe framework and perform exhaustive ablations on the training losses and the matching constraints. The impact of the number of iterations (T) and tem- plates (N) in LIT is studied in Appendix 1. For the rest of the paper, we set T and N 256.
Matching constraints ablation. Table reports the impact of the different matching constraints: nearest neighbor, reciprocity, Lowe ratio test and constraint on Super-fcatures ID. Adding reciprocity
hhttps:/// :/g thub. coomggollnn iottolias. has/hww

---

Published as a conference paper at ICLR 2022
254
L Glorat
Latta
Lapeer
SfM-120k
val
med
Okfoordd
hard
Paris
hand
low
79.0
64.3
38.0
75.4
51.7
87.7
75.8
51.2
79.0
57.0
88.4
57.2
61.7
59.3
32,8
69.9
47.3
89.7
81.9
61.5
85.3
70.1
high
21
205
Table 2: Ablation on loss components: Im- pact of removing Lamn, using either global loss gioba or loss directly on Super-features LLuper or combination of both.
Figure 4: Impact of aatnn on the correlation
matrix between attention maps (the darker, the lower the correlation) the last iteration of LIT when training with (left) and without (right). This averaged over the 70 queries of ROxford.
R ROxford (avg)
R Paris (avg)
1000
OP0O 1000
70
2000
75
2000
6
65
1000
70
o 1000
FIRe
65
FIRe
60
HOW
O- HOW
500
1,000
1,500
0
500
1,000
1,500
average (c())
average [C(x)]
Figure 5: Performance versus memory for HOW and FIRe. The x-axis shows the average number of clusters per image used in ASMK (proportional to memory usage). We vary the number of
features extracted per image before aggregation in (200,.... 2000, 2500, .., 5000); solid markers denote the commonly used settings (1000/2000). FIRe has at most 1,792 features (256 per scale).
and Lowe's ratio test significantly improves performance, which indicates that it reduces the number of incorrect matches. Keeping all pairs with the same ID yields lower performance: the ID constraint alone is not enough. One possible explanation is that two images from the same landmark may differ significantly, e.g. when some parts are visible in only one of the two images. In that case, the other constraints allow features attending non-overlapping regions of a positive image pair to excluded from P. Finally, combining the selective ID constraint with the others yields the best performance.
In order to better understand the impact of the Super-feature constraint, a constraint only applica- ble for Super-features due their ordered nature, we measure the quality of selected matches during training with and without it. Since there is no ground-trut for such localized matches on landmark retrieval datasets, we measure instead the ratio of matches coming from the positive pair, over all matches (from the positive and all negatives). Ideally, a minimal number of matches should come from the negatives, hence this ratio should be close 1. In Figure we plot this match ratio for all epochs; we observe that it is significantly higher when using the Super-features ID constraint.
Training losses ablation. We study the impact of the different training losses in Table 2. We start from global loss similar to HOW (first row). We then add the decorrelation loss (second row) and observe clear gain. It can be explained by the fact that without this loss, Super-features tend to be redundant (see the correlation matrices in Figure 4). Adding the loss operating directly on the Super-features further improves performance (third row). Next, we remove the global loss and keep only the loss on Super-features alone (fourth row) with the decorrelation loss (last row). The latter performs best (fourth vs last row). Figure displays the correlation matrix of Super- features attention maps with and without Lamn- Without it, we observe that most Super-features have correlated attentions. contrast, training with Lam leads to uncorrelated attention maps. This is illustrated in Figure which shows several attention maps focusing different areas.
Varying the number of features at test time, Figure 5 compares HOW (Tolias et al., 2020) with approach, as we vary the number of local features Super-features. The x-axis shows the average number clusters used in ASMK for the database images, i.e., which is proportional to the average memory footprint of image. We observe that our approach does not only significantly improve
7

---

Published as a conference paper at ICLR 2022
Scale of selected features (%)
Features selected per scale(())
Total nb. features per scale
0.6
features (%)
0.3
HOW
FIRe
features (%)
# features
4K
0.2
0.4
3K
2K
0.1
0.2
1K
0.25 05 0.7 1.0 1.4
2.0
0.35 05 0.7
1.0 144 2.0
0.25 0.5 0.7
1.0 144 2.0
Figure 6: Statistics on the feature selected across scales for HOW and FIRe, averaged over the 70 queries from ROxford. Left: Among the 1000 selected features, we show the percentage coming from each scale. Middle: For each scale, we show the ratio of features that are selected. Right: Total number of features per scale: we extract N 256 Super-features regardless of scale.
accuracy compared to HOW, but also requires overall less memory. Notably, FIRe matches the best performance HOW with a memory footprint reduced by a factor of 4. For both methods, performance goes down once feature selection no longer discards background features. The gain in terms of performance is even larger when considering a single scale at test time (see Appendix B.1).
Statisties the number of selected features per scale. The left plot of Figure 6 shows the percentage of the 1000 selected features that comes from each scale, for HOW and FIRe. For HOW most selected features come from the higher resolution inputs; by contrast, selected Super-feature featurfe are almost uniformly distributed across scales, Interestingly, Figure 6 (middle) shows that HOW keeps higher percentage of the features coming from coarser scales. Yet, the final feature selection for HOW is still dominated by features from higher scales, due to the fact that the number of local features significantly increases with the input resolution (see Figure 6 right).
4.2 COMPARISON TO THE STATE OF THE ART
We compare our method to the state of the art in Table 3. All reported methods are trained on SfM-1206 for fair comparison. 5 First, we observe that methods based on global descriptors (Tolias al.. 2016; Revaud et al., 2019a) compared with L2 distance tend be less robust than methods based on local features (Noh et al., 2017; Tolias et al., 2020). DELG (Cao et al., 2020) shows better performance, owing to a re-ranking step based on local features, the cost of re-extracting local features for the top-ranked images of given query, given that local features would take too much memory to store. Our FIRe method outperforms DELF (Noh et al., 2017) as well as HOW (Tolias et al., 2020) by a significant margin when extracting 1000 features per image. Importantly, our approach also requires less memory, as uses fewer ASMK clusters per image, as shown in Figure 5: for the whole R 1M set, HOW uses 469 clusters per image while FIRe uses only 383, average.
RELATED WORK
Image descriptors for retrieval. The first approaches for image retrieval were based on handcrafted local descriptors and ofg-of-words representations borrowed from text retrieval (Sivic & Zisserman, 2003: Csurka al., 2004), or other aggregation techniques like Fisher Vectors (Perronnin et al., 2010), VLAD (Jégou et al., 2010) or ASMK (Tolias et al., 2013). First deep learning techniques were extracting global vector per image either directly or by aggregating local activations (Gordo al., 2016; Radenović et al., 2018b) and have shown to highly outperform handcrafted local features, see (Csurka& & Humenberger, 2018) for a review. Methods that perform matching or re-ranking using CNN-based local features et al., 2019; Cao et al., 2020; Tolias et al., 2020). They are able to learn with global (i.e. image-level annotations. Most of them simple variants of attention mechanisms (Noh et al., 2017; Ng al., 2020) or simply the feature norm (Tolias et al., 2020) to weight local activations.
currently the state of the art in the area (Noh et al., 2017: Teichmann
Low-level features aggregation. Aggregating local features into regional features has long his- tory. Crucial to the success of traditional approaches, popular methods include selecting discrimi- native patches in the input image (Singh al., 2012; Gordo, 2015), regressing saliency map scores
The GLDv2 clean dataset (Yokoo et al., 2020) is sometimes used for training. Yet, there is significant overlap between its training classes Oxford query landmarks. departs standard image retrieval evaluation protocols. Appendix for details.
8

---

Published as a conference paper at ICLR 2022
method
FCN
Mcm (GB)
O
ROxford med
+ IM
Paris
Paris
+ IM
med
hard
hard
med
Global descriptors
RMAC (istacci
R101
7.6
60.9
32.4
39.3
12.5
78.9
54.8
28.0
AP-GeMf (Readd
R101
7.6
67.1
42.3
47.8
22.5
80.3
60.9
51.9
24.6
GeM+SOLAR ii
R101
7.6
69.9
47.9
53.5
29.9
81.6
64.5
59.2
33.4
Global descriptors + reranking with local features
DELG Dast 260
R50
7.6
75.1
54.2
61.1
36.8
82,3
64,9
60.5
34,8
DELG 200
R101
7.6
78.5
59.3
62.7
39.3
82.9
65.5
62.6
37.0
Local feutures ASMK mustching (max. 1000 features per image)
DELF (1
R50
9.2
67.8
43.1
53.8
31.2
76.9
55.4
57.3
26.4
DELF-R-AN R-AMMR (uittmes 20011
R50
27.4
76.0
52.4
64.0
38.1
80.2
58.6
59.7
29.4
HOW (bitsan 2100
R50
7.9
78.3
55.8
63.6
36.8
80.1
60.1
58.4
30.7
FIRe (ours)
R50
6.4
81.8
61.2
66.5
40.1
85.3
70.0
67.6
42.9
(standard deviation over 5 rans)
(+006)
(=1.0)
(+008)
(+111)
(+0.4)
(+006)
(+0,7)
(+008)
(mAP guins over HOW)
(1 (.5)
(1 5.4)
(1 2.9)
(1 3.3)
(1 5.2)
(T 9.9)
(1 9.2)
(1 12.2)
Table 3: Comparison to the state of the art. All models are trained on SfM- 120k. FCN denotes
the fully-convolutional network backbone, with R50 denoting ResNet-50 without the last block.
Memory is reported for the image representation of the full R 1M set (without counting local features for the global descriptors reranking methods). result from (Tolias et al., 2020). Bold denotes best performance, underlined second best among methods using ASMK, italics second best overall.
at different resolutions (Jiang et al., 2013), aggregating SIFT descriptors with coding and pooling schemes (Boureau et al., 2010), or mining frequent patterns in sets of SIFT features (Singh et al., 2012). More recently, Hausler et al. (2021) introduced a multi-scale fusion of patch features.
Supervision for local features. Several works provide supervision at the level of local features in the context of contrastive learning. Xie et al. (2021) and Chen et al. (2021) obtain several views of an input image using data augmentations with known pixel displacements. Similarly, Liu et al. (2020) train a model to predict the probability for a pair of local features to match, evaluated using known displacements. Wang al. (2020) and Zhou et al. (2021) obtain local supervision by relying on epipolar coordinates and relative camera poses. Positive pairs in image retrieval depart from these setups, pixel-level correspondences cannot be known. To build matches, Wang al. (2021) use standard nearest neighbor algorithm to build pairs of features, similarly to our approach, but without the use of filtering which is critical our final performance. Using the current model predictions to define targets is reminiscent of modern self- supervsede learning approaches which learn without any label (Caron et al., 2021; Grill al., 2020). The additional filtering step can be seen as a way to keep only the most reliable model predictions, similar to self-distillation in e.g. Sohn et al. (2020).
Attention modules. Our LIT module an iterative variant of standard attention (Bahdanau et al., 2015; Vaswani et al., 2017), adapted to map a variable number of input features to an ordered set of N output features, similar to Lee et al. (2019). The Perceiver model (Jaegle et al., 2021) has demonstrated the flexibility of such constructions by using it to scale attention- based deep networks to large inputs. Our design was heavily inspired by slot-attention (Locatello et al., 2020), but has some key differences that enable to achieve high performance in more complex visual environ- ments: a) unlike the slot attention which initializes its slots with i.i.d sampling, we learn the initial templates and therefore define ordering on the output set, a crucial property for selecting promis- ing matches; b) we replace the recurrent network gates with a residual connection across iterations. These modifications, together with the attention decorrelation loss enable our module go from a handful of object-oriented slots to much larger set of output features. For object detection, Carion al. (2020) rely on set of learned object queries to initialize a stack of transformer layers. Unlike ours, their module is not recurrent; Appendix A.1 experimentally shows substantial benefits from applying LIT times, with weight sharing, to increase model flexibility without extra parameters.
CONCLUSIONS
We present an approach that aggregates local features into Super-features for image retrieval, a task that has up to now been dominated by approaches that work at the local feature level. We design an attention mechanism that outputs an ordered set of such features that are more discriminative and expressive than local features. Exploiting their ordered nature and without any extra supervision, present a loss working directly on the proposed features. Our method not only significantly improves performance, but also requires less memory, crucial requirement for scalable image retrieval.

---

Published as a conference paper at ICLR 2022
REFERENCES
Artem Babenko and Victor Lempitsky. Aggregating deep convolutional features for image retricval. In Proc. ICCV, 2015.
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. ICLR, 2015.
Y-Lan Boureau, Francis Bach, Yann LeCun, and Jean Ponce. Learning mid-level features for recognition. In Proc. CVPR, 2010.
Bingyi Cao, André Araujo, and Jack Sim. Unitying deep local and global features for image search. In Proc. ECCV, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnacve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. Proc. ECCV, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proc. ICCV, 2021.
Kai Chen, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit-Yan Yeung. Multisiam: Self-supervised multi- instance siamese representation learning for autonomous driving. In Proc. ICCV, 2021.
Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalantidis. Graph- based global reasoning networks. Proc. CVPR, 2019.
Gabriela Csurka and Martin Humenberger. From handcrafted deep local invariant features. arXiv preprint arXiv: 10254 2018.
Gabriela Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. Visual categorization with bags of keypoints. In Proc. ECCV Worshops, 2004.
Albert Gordo. Supervised mid-level features word image representation. In Proc. CVPR, 2015.
Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global repre- sentations for image scarch. In Proc. ECCV, 2016.
Jean-Basee Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Kavukcuoglu Koray, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to supervied learning. In Proc. NeurIPS, 2020.
Stephen Hausler, Souray Garg, Ming Xu, Michael Milford, and Tobias Fischer. Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition. In Proc. CVPR, 2021.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016.
Martin Humenberger, Yohann Cabon, Nicolas Guerin, Julien Morat, Jérôme Revaud, Philippe Rerole, Noé Pion, Cesar de Souza, Vincent Leroy, and Gabriela Csurka. Robust image retrieval-based visual localization using kapture. arXiv preprint arXiv:2007. 13867, 2020.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver:
General pereeption with iterative attention. Proc. ICML, 2021.
Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into compact image representation. In Proc. CVPR, 2010.
Huaizu Jiang, Jingdong Wang, Zejian Yuan, Yang Wu, Nanning Zheng, and Shipeng Li. Salient object detec- tion: A discriminative regional feature integration approach. In Proc. CVPR, 2013.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer framework for
intuuuatiioninvvrain neural networks. In Proc. ICML, 2019.
Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. arXiv preprint 1r 136777, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dasovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proc. NeurlPS, 2020.
10

---

Published as a conference paper at ICLR 2022
David G Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004.
Grégoine Mialon, Dexiong Chen, Alexandre d' d'Aspremont, and Julien Mairal. A trainable optimal transport embedding for feature aggregation
relationship attention. Proc. ICLR, 2021.
Tony Ng, Vassileios Balntas, Yurun Tian, and Krystian Mikolajezyk. Solar: second-order loss and attention for image retrieval. In Proc. ECCV, 2020.
Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In Proc. ICCV, 2017.
Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. Large-scale image retrieval with compressed fisher vectors. In Proc. CVPR, 2010.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In Proc. CVPR, 2007.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Lost in quantization: Im- proving particular object retrieval large scale image databases. In Proc. CVPR, 2008
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondiej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Proc. CVPR, 2018a.
Filip Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning enn image retrieval with no human annotation.
IEEE Trans PAMI, 2018b.
Jerome Revaud, Jon Almazán, Rafael S Rezende, and Cesar Roberto de Souza. Learning with average precision: Training image retrieval with listwise loss. In Proc. ICCV, 2019a.
Jerome Revaud, Cesar De Souza, Martin Humenberger. and Philippe Weinzaepfel. R2D2: Reliable and repeat- able detector and descriptor. Proc. NeurIPS, 2019b.
Torsten Sattler, Will Maddemn, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Mare Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localization in changing conditions. In Proc. CVPR, 2018.
Saurahh Singh, Abhinav Gupta, and Alexei A Efros. Unsupervised discovery of mid-level discriminative patches. In Proc. ECCV, 2012.
Josef Sivic and Andrew Zisserman. Video google: A text retrieval approach to object matching in videos. In Proc. ICCV, 2003.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Ku- rakin, Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Proc. NeurlPS, 2020.
Marvin Teichmann, Andre Araujo, Menglong Zhu, and Jack Sim. Detect-to-retrieve: Efficient regional aggre- gation for image search. In Proc. CVPR, 2019,
Giorgos Tolias, Yannis Avrithis, and Hervé Jégou. To aggregate or not to aggregate: Selective match kernels for image search. In Proc. ICCV, 2013.
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max- pooling of enn activations. In Proc. ICLR, 2016.
Giorgos Tolias. Tomas Jenicek, and Ondiej Chum, Learning and aggregating deep local descriptors for instance- level recognition. In Proc. ECCV, 2020.
Anne M Treisman and Garry Gelade. A feature-inteeration theory of attention. Cognitive psychology, 1980.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurlPS, 2017.
Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning feature descriptors using camera pose supervision. In Proc. ECCV, 2020.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self- supervised visual pre-training. In Proc. CVPR, 2021.
11

---

Published as a conference paper at ICLR 2022
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale bench- mark for instance-level recognition and retrieval. Proc. CVPR, 2020.
Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. The application of two-level attention models in deep convolutional neural network fine- fine-grained image classification. In Proc. CVPR, 2015.
Zhenda Xie, Yutong Lin, Zheng Zhang. Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proc. CVPR, 2021.
Shuhei Yokoo, Kohei Ozaki, Edgar Simo-Serra. and Satoshi lizuka. Two-stage discriminative re-ranking large-scale landmark retrieval. In Proc. CVPR Worshops, 2020.
Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixe. Patch2pix: Epipolar-gust guided pixel-level correspondences. In Proc. CVPR, 2021.
12

---

Published as a conference paper at ICLR 2022
Appendix
Table of Contents
A Additional ablations
14
A.1 Impact of the iteration and the number of templates hyper- paree--armmterss
14
A.2 Impact of hard negatives
14
A.3 Impact of the update function
15
A.4 Extended ablation on matching constraints
15
A.5 Impact of the template initialization
B Analysis and discussions
16
B.1 Single-scale results
16
B.2 Consistency of attention maps across scales
16
B.3 Redundancy in Super-features
16
B.4 Are all Super-feature trained?
17
Using the Super-feature loss with local features
B.6 Pretraining backbone together with LIT
B.7 Computational
of Super-features extraction
18
B.8 Is Google Landmarks v2 clean appropriate training dataset for testing ROxford and R Paris?
18
C Application to visual localization
18
In this appendix, present additional ablations (Appendix A), deeper analysis of our framework (Appendix B) as well as results of the application our FIRe framework for image retrieval to the task of visual localization (Appendix C). We briefly summarize the findings in the following paragraphs.
Ablations. We study the impact of some hyper- spper-parrmeters of model, namely the size N of the set of Super-features, and the number of iterations T in the LIT module, in Appendix A. We show that 256 Super-features and iterations offer the best trade-off between performance and computational cost. We also show in Appendix A.2 that we obtain further performance gains of over 1% by increasing the number of negatives to 10 15. We then study the impact of replacing the residual connection inside the LIT module by a recurrent network (Appendix A.3). In Appendix A.4 we present an extended version of Table with further matching constraints and we finally study the impact of the template initialization on performance in Appendix A.5.
Properties of Super-features. We show single-scale results in Appendix B. In Appendix B.2, we display the attention maps of Super-features, at different scales for a fixed Super-feature ID. We further study the amount of redundancy in Super-features, compared to local features, in Ap- pendix B.3. Next, verify in Appendix B.4 that all Super-fcatures receive training signal, sanity check. We discuss the case of applying a loss directly on local features in Appendix B.5 and give details about the pretraining average extraction time for the proposed Super-features, while in Appendix B.8 we discuss the fact that there is overlap between the queries from the common ROxford and R Paris datasets and the Google Landmarks-v2-clean dataset that is commonly used as training set for retrieval on ROxford and RR Paris.
ImageNet in Appendix B.6. In Appendix B.7 report the
Application to visual localization. We further evaluate our model using a visual localization setting on the Aachen Day-Night v1.1 dataset (Sattler al., 2018) Appendix C. To do so, we leverage retrieval local feature matching pipeline, and show that it is beneficial to use our method, especially in hard settings.
13

---

Published as a conference paper at ICLR 2022
A ADDITIONAL ABLATIONS
A.1 IMPACT OF THE ITERATION AND THE NUMBER OF TEMPLATES HYPER-PARAMETERS.
In this section, we perform ablations on the number of Super-features and the number of iterations T in the Local feature Integration Transformer. We first study the impact of the number N of Super- features extracted for each scale each image Figure 7a. We observe that the best value is 256 on both the validation and test sets (ROxford and R Paris). We then study the impact of the number of iterations T in Figure 7b. While the performance decreases when doing or iterations compared to just 1, a better performance is reached for 6 iterations, after which the performance saturates while requiring more computations. We thus use T
(ovg)
90
90
80
80
mAP
mAP
70
70
60
60
128
256
512
1024
3
12
number of templates (N)
number of iterations (T)
(a) Impact of the number of Super-features
(b) Impact of the number of iterations
Figure 7: Varying the number of Super-features N and of iterations T in the Local feature Integration Transformer.
A.2 IMPACT OF HARD NEGATIVES
Each training tuple is composed of one image pair depicting the same landmark, and 5 negative images (i.e. from different landmarks). We plot the performance when varying this number in Fig- ure We observe that adding more negatives improves overall the performance on all datasets, i.e. by more than 1% when increasing it to 10 or 15 negatives, but at the cost of a longer training time as more images need be processed at each iteration. This is why we use negatives the paper offers a good compromise between performance and cost.
rest of
SIM-120K vol. - D0Odod (ovg) - 3 Paris (ong)
90
80
mAP
70
2
4
5
7
10
15
number of negatives per tuple
Figure Varying the number of hard negatives per training tuple.
14

---

Published as a conference paper at ICLR 2022
A.3 IMPACT OF THE UPDATE FUNCTION
The formula for db in Equation (5) sums the previous 2 with the output of the attention component V(U) a, i.e., with residual connection. This is a different choice than the one made in the object-centric slot attention of Locatello et al. (2020), which proposes to use a Gated Recurrent Unit: 15(U; 2) GRU(V(?)) a, 9 ). We thus compare the residual connection we use to a GRU function and report results in Table with T = 3. We observe that the residual connection reaches a better performance in all datasets while having the interest of not adding extra parameters.
update function
SfM-120k val
ROxford hard
R Paris hard
med
med
residual
79.9
71.3
43.9
78.7
59.4
GRU
74.7
66.7
41.4
77.2
57.8
Table 4: Impact of the update function in the LIT module. We compare the performance of the residual combination of the previous template value with the cross-attention tensor compared GRU as used slot attention (Locatello et al., 2020). this experiment, we use T 3.
In summary, we propose the LIT module to obtain a few hundred features for image retrieval while slot attention handles handful of object attentions. Technical differences to the slot attention include: a) we use a learned initialization of the templates instead of i.i.d. sampling, which allows to obtain an ordered to clear gain, see Table and Figure 3, b) to handle a larger number of templates, we also add decorrelation loss on the attention maps, which has clear benefit, see Table 2 and Figure we use a residual connection with 6 iterations instead of a GRU, leading to improved performance for our task (see Table 4).
of features, and thus apply the constraint on the ID for the matching, leading
A.4 EXTENDED ABLATION ON MATCHING CONSTRAINTS
reci- proc,
ratio
same ID
SfM-120k val
ROxford
Paris
med
hard
med
hard
68.3
64.3
39.8
74.1
52.4
79.6
69.2
44.3
79.2
60.9
89.9
81.9
61.1
85.1
69.6
73.5
64.6
39.0
75.5
55.0
84.2
75.0
49.8
79.4
61.1
V
80.8
70.7
45.1
80.3
61.9
75.9
63.8
35.1
77.3
56.5
89.7
81.8
61.2
85.3
70.0
Table 5: Extended ablation on matching constraints. We study the impact of removing constraints on reciprocity, Lowe's ratio test and the Super-feature ID.
We show in Table an extended version of Table of the main paper, where we evaluate all possible combinations of constraints among reciprocity, Lowe's ratio test and the Super-feature ID. We ob- serve that the reciprocity constraint and the Super-feature ID constraint are the two most important ones, while the Lowe's ratio only only brings a small improvement.
A.5 IMPACT OF THE TEMPLATE INITIALIZATION
In the LIT module, initial templates 2 E R xd are learned together with the LIT module. We can therefore assume that they are adapted to the task at hand. To explore the sensitivity to the initial templates, we run a variant of FIRe where the initializations are not fine-tuned, but instead frozen to the values after pretraining on ImageNet. We report performance in Table 6. We observe that the two variants perform overall similarly. This ablation suggests that the initial templates are up to some point transferable to other tasks.
15

---

Published as a conference paper at ICLR 2022
Template Initialization
SfM-120k
ROxford
R Paris
val
med
hard
med
hard
Frozen from ImageNet pretraining Fine-tuned for landmark retrieval
89.5
81.3
59.6
85.3
70.2
89.7
81.8
61.2
85.3
70.0
Table 6: Fine-tuning the initial templates. Comparison where we either fine-tune the initial tem- plates Q of LIT (bottom row, as in the main paper) or keep them frozen after ImageNet pretraining (top row).
(ROxford (avg)
R Paris (avg)
60
55
200
200
50
50
200
200
40
40
0- HOW
6- HOW
35
FIRe
FIRe
30
50
100
150
200
250
0
50
100
150
200
250
average [C(x)]
average [C(x)]
Figure 9: Performance versus memory when varying the number of selected features at single scale for HOW and FIRe. The x-axis represents the average number of vectors per image in ASMK, which is proportional to the memory, when varying the number of selected features in (25, 50, 100, 150, 200, 300, 400, 500, 600), FIRe limited 256 features.
B ANALYSIS AND DISCUSSIONS
B.1 SINGLE-SCALE RESULTS
Similar to Figure 5 of the main paper, we perform the same ablation in Figure 9 when extracting features single scale (1.0). We observe that FIRe significantly improves the mAP on the two datasets compared to HOW. The average number of clusters, i.e. the memory footprint of images, remains similar for both methods the multi-scale case where our approach allows to save memory (about 20%), which we hypothesize due to the correlation of our features across scales that we discuss below.
same number of selected features. This stands contrast to
B.2 CONSISTENCY OF ATTENTION MAPS ACROSS SCALES
At test time, we extract Super-fcatures at different image resolutions. We propose here to study if the attention maps of the Super-features across different image scales are correlated. We show in Figure 10 the attention maps at the last iteration of LIT for the different image scales. We observe that they fire the same image location at the different scales. Note that the attention maps are larger/smoother at small scales (right columns), as for visualization, we resize lower resolution attention maps to the original image size.
B.3 REDUNDANCY IN SUPER- SUPER-FEATURES
To evaluate the redundancy of Super-features versus local features, Figure 11 displays the aver- age cosine similarity between every local feature Super-feature and its nearest local features / Super-features from the same image, for different values of K. We observe that Super- super-features significantly less correlated to the most similar other ones, compared to local features used in HOW.
16

---

Published as a conference paper at ICLR 2022
image 0.25
0.353
0.5
0.707
1.0
1.414 2.0
Figure 10: Attention consistency across scales. For three images, we select one Super-feature per image and show the attention maps of the latest iteration LIT for different image scales, with from left to right 0.25, 0.353, 0.5, 0. 707, 1.0, 1.414 and 2.0. We clearly observe that attention maps are correlated. They show larger regions at small scales as for visualization, we resize the lower resolution attention the original image size.
Avg. similarity to K nearest
0.5g
local features from HOW
Super features from FIRe
0.4
0
0.3
0.2
50
100
150
200
250
300
350
400
450
300
550
600
K
Figure 11: Measuring Super-feature redundancy. We compute the average cosine similarity be- tween every feature (local feature from HOW or Super-feature from FIRe) and its K most similar features from the same image, for varying K. Results are averaged over the 70 query images of the ROxford dataset. Only 256 Super-features can be extracted per image, explaining its maximum value.
B.4 ARE ALL SUPER-FEATURES TRAINED?
The loss in Equation (8) only operates subset of the Super-feature pairs that pass the criteria of Equation (7); if a Super-feature ID is not matched, it receives no training signal. To investigate all of the IDs contribute to the loss, we monitored how many times each ID is matched each epoch and report the percentage over all 2000 training tuples mean, standard deviation and minimum number of matches for the N Super-feature ID. We clearly observe that all Super-features receive training signals regularly and each Super-feature is matched about one quarter of the time on average.
epoch. In Figure 12 we report
B.5 USING THE SUPER-FEATURE LOSS WITH LOCAL FEATURES
It is worth noting that the loss in Equation (8) could also theoretically be used over local feature activations. It could be computed on pairs of local features that pass the conditions described in Equation (7), after removing the constraint on having the same Super-feature ID. Unfortunately, were unable to get any gains over HOW (Tolias et al., 2020) when appending such a loss side-by- side with the global loss over local features. Empirically, adding the constraint on the Super-features ID, which is only possible with ordered feature sets, is key to the success of our approach, and significantly improves the quality of the matching.
ablations (see Section 4.1) show that
17

---

Published as a conference paper at ICLR 2022
matches per Super-feature
0.3
0.2
0.1
average
ming
20
40
60
80
100
120
140
160
180
200
Epochs
Figure 12: Are all Super-features trained? As a sanity check, this plot shows the average number (in percentage) of times (with standard deviation) that the Super-features receive training signal, across training epochs. We also plot the minimum value, which is always significantly positive, showing that each Super-feature
receives training signal. Thus our loss Super proposed Equa- tion (8) does not lead to degenerate solutions where some IDs are never selected and never trained.
B.6 PRETRAINING THE BACKBONE TOGETHER WITH LIT
Before training on landmark retrieval, pretrain the backbone network, with the proposed LIT module appended after the last layer, for image classification on ImageNet-1K, Given that we re- move the last convolutional block similar HOW, to avoid overfitting we further append a classifi- cation head composed of one fully-connected layer, batch norm, leaky ReLU, dropout and another fully-connected layer at the end of the network and train for epochs using a standard cross- entropy loss and in addition the decorrelation loss Lam weighted by 0.1, the same weight as during fine-tuning. We start with learning rate of 0.01 and divide it by 10 every 20 epochs. This new archi- tecture reaches top-1 (resp. top-5) accuracy of 73 33.48 (resp. 91.54%) on the ImageNet validation set. Note that this performance a bit lower than a standard ResNet-50: this is because we have removed the last convolutional block which contains most parameters in the ResNet architecture.
B.7 COMPUTATIONAL COST OF SUPER-FEATUREE EXTRACTION
We report the time required for extracting multi-scale features for 5000 images, for HOW and FIRe. On our server, took 1.57 seconds for HOW and 172 for FIRe, i.e. extraction for Super-features only requires 10% more wall-clock time.
B.8
Is GOOGLE LANDMARKS v2 CLEAN AN APPROPRIATE TRAINING DATASET FOR TESTING ON ROXFORD AND RPARIS?
In the comparison the state of the art (Table 3), all reported methods trained the SfM- 120k dataset. Several recent works have also released a model trained on the Google Landmarks v2 dataset (Weyand al., 2020) its clean version (Yokoo al., 2020) with excellent performance on RParis. However, we find out that several of the query landmarks from Raris and ROxford were present the training set of the cleaned version Google Landmarks v2, such as 'La Defense", 'Eif- fel Tower', 'Sacré Coeur' or 'Hotel les Invalides' for R Paris such as 'Mary Madgalen', 'Bodleian Library', Souls College' or 'Radcliffe' ROxford.
C APPLICATION TO VISUAL LOCALIZATION
In this section, we evaluate FIRe for the task of visual localization, where retrieval is used as first-stage filtering and before more precise, local feature-based geometric matching. To this end, we follow the pipeline proposed by Kapture (Humenberger et al., 2020) on the Aachen Day-Night v1.1 dataset (Sattler et al., 2018), In this scenario, global Mancture-from-Motiee map is built
htttrer://tttttccco///eee/////ee loca zation
18

---

Published as a conference paper at ICLR 2022
Day images
Night images
Retrieval method
0.25m, 2
0.5m, 5
5m, 10
0.25m,
0.5m, 5
5m, 10
AP-GeM (Revaud al., 2019a)
88.8
96.6
99.6
72.3
86.9
97.9
HOW (Tolias et al., 2020)
90.8
96.2
99.6
72.8
90.1
97.9
FIRe (ours)
90.7
96.5
99.5
74.3
90.1
98.4
Table 7: Visual localization results. Percentage of successfully localized images on the Aachen Day-Night v1.1 dataset when changing the retrieval method in the Kapture pipeline from Humen- berger et al. (2020). In the most challenging scenario, i.e. night images at strictest localization threshold, using FIRe yields 2% improvement compared to AP-GeM and 1.5% improvement compared to HOW. Bold number denotes the best performance, underlined indicates performance within a 0.1 margin to the best one.
from the training images using R2D2 local descriptors (Revaud et al., 2019b). At test time, given query image localize, image retrieval is used retrieve the top-50 nearest images. On these retrieved images, R2D2 local features are extracted and matched with the ones on the query image, and this is used estimate the position of the camera. The percentage of successfully localized images within three levels of thresholds is then reported on the day or night images, following the visual localization benchmark protocol', the latter ones being more challenging as training images are taken during daytime.
We compare our retrieval method AP-GeM (Revaud et al., 2019a) and HOW (Tolias et al., 2020) and report results in Table 7. AP-GeM is the default method used in Kapture (Humenberger et al., 2020). We observe that using FIRe leads to better visual localization, specially in the most chal- lenging scenario of night image localization and the strictest localization threshold: Performance improves by 2% compared to AP-GeM and by 1.5% compared to HOW on night images at a thresh- old of 0.25m and 2 Overall, for either day night images, FIRe either improves performs par to both methods compared.
"https://wwa visu local zza on. net.
19


Reviews:
[{"review_id": "oswQrYColew", "reviewer": "Reviewer_uoYN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== Strengths ===\n\n- LIT is an interesting new method for local feature extraction from images using a transformer-like architecture.\n- While supervision is only needed on the image level, the loss is applied on the local feature level. In contrast, previous work on deep learning based local features applied the loss on aggregated local features.\n- The ordered property of SuperFeatures helps interpret what each feature represents semantically.\n- Rigorous ablation experiments clearly motivate the design and parameter choices.\n- SuperFeatures consistently outperform the state of the art on the ROxford and RParis datasets.\n- A good selection of relevant related work is cited.\n- The paper is well-written, well-organized and easy to follow.\n\n=== Weaknesses ===\n\n- Some inconsistencies in the mathematical formulation of the method make it difficult to understand its details and would also make it difficult to reimplement the method. Details on this in the “Questions and suggestions on Sec. 3.1” section below.\n- The computational cost of extracting SuperFeatures from an image is not discussed. This would be important for real-time image retrieval systems where features of the query image have to be extracted on-the-fly. It would be interesting to compare the speed of SuperFeature extraction to, e.g., DELG or HOW.\n\n=== General questions and suggestions ===\n\n- In the paper, image retrieval is only performed using ASMK. Have the authors tried other methods too? In particular, do SuperFeatures lend themselves to traditional spatial verification using RANSAC? If yes, it would be interesting to see results on that. If not, it would be helpful to clarify in the paper why this is not possible.\n- Have the authors performed an ablation where the loss is applied on an aggregation of all local features? This could further motivate why applying the loss on the local feature level is beneficial.\n- How sensitive is the performance of SuperFeatures to the initial set of templates? Are templates specific to a certain domain, e.g., landmarks or can they are transferred to, e.g., products or artwork?\n- In Tab. 1, it would be interesting to also have the following rows: ”reciprocal+same”, “ratio only” and “same+ratio”.\n- What is the value of N (number of SuperFeatures) in the experiments?\n- The ablations show that matching only SuperFeatures with the same ID during training increases performance. However, in retrieval with ASMK, the ordered property of SuperFeatures is not taken advantage of. Have the authors tried an image retrieval method that matches only identical SuperFeature IDs?\n- In Fig. 5, what are the numbers (1000, 2000) next to the curves?\n- In the Related Work section, consider renaming the first subsection to “Image descriptors”\n- In Related Work on image retrieval, consider citing GeM (Radenovic2018b), since it’s a popular method for feature aggregation that SuperFeatures are also compared against.\n- In the second to last row of page 4, it seems a word is missing. Maybe it should be “the attention maps *tend* to be similar”\n\n=== Questions and suggestions on Sec. 3.1 ===\n\n- Are attention maps (\\alpha_l) the rows or the columns of the matrix \\alpha? Put differently, does an attention map tell me how well a single SuperFeature responds to each image location (like in Fig. 1), or do they tell me how much a single image location excites each SuperFeature?\n- In the 1st equation in Eq. 4, should the final index be \\alpha_N instead of \\alpha_L? This seems to be inconsistent with the row above that says that there are N attention maps.\n- It would help if the paper could specify the linear projection dimensions d_k, d_v and d_q.\n- What are the dimensions of the three matrices in the second equation of Eq. 5? As far as I understand: \\alpha is an (L x N) matrix as per Eq. 4, V(U) is an (N x L) matrix because it needs to have as many rows as \\alpha has columns, Q is a (d x N) matrix since it has N templates of dimension d. However, if these are true, then the addition would not work since an (L x L) matrix would be added to a (d x N) matrix. If I understand correctly, the output of \\psi(U, Q) needs to be a (d x N) matrix, meaning \\alpha would need to have d rows and V(U) would need to have N columns. I’d appreciate it if the authors could point out where I’m wrong here.\n- In the same equation, V(U) is a linear projection function applied to the set U. I think it is implied here that the elements of the set are concatenated into a matrix before applying the function. It would help if this were clarified in the text.\n- The 2nd equation of Eq. 4 seems to divide vectors by vectors. I think there is a 2nd index missing in the divisor, i.e. it should be \\alpha_{l,i}\nEquations 4 and 5 skip the “t” superscript, as explained in footnote 2. I personally think that having the superscript in these equations would help with clarity.\n- What is the motivation for applying softmax and L1 normalization to \\alpha?\n- In the 1st equation of Eq. 5, I’d suggest adding “Q^{t+1} = \\phi(...)“\n- I’d suggest using n instead of j as the running index for the templates, which would be consistent with using l as the running index for the local features.\n- I’d suggest putting footnote 3 in the main text since it is an important detail.", "summary_of_the_paper": "The paper proposes Local Feature Integration Transformer (LIT), a new architecture to extract mid-level local features from images that are used for image retrieval. The resulting features are called SuperFeatures and are shown to yield state-of-the-art performance on two standard image retrieval benchmarks. The paper further proposes a learning framework, called Feature Integration-based Retrieval (FIRe), to train LIT using a contrastive loss with only image-level supervision. In contrast to other deep learning based image features that apply the loss on the image embedding or aggregated local features, FIRe applies the loss on the local features directly. An interesting property of SuperFeatures is that they are ordered, meaning a SuperFeature with the same index i in two matching images will be localized on similar structures.", "summary_of_the_review": "I am confident to suggest the paper for acceptance since it introduces a novel and interesting way of using transformers to train local image features that outperform the state of the art. The method is easy to train since only image-level labels are required, however the loss is applied in the local feature level, which is an interesting novelty.  Detailed ablations motivate the design of the method. My main concern are inconsistencies in the mathematical formulation of the method, which I am sure can be resolved for the camera-ready version.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SfARGadtCqC", "reviewer": "Reviewer_hp4Y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The experimental results are solid and relevant. The authors' approach outperform the baselines by a large margin, not only in mAP but also in its memory requirements. The ablation studies are complete and contribute to understanding the role of each part of the approach. This reviewer finds very relevant the fact that a global loss is irrelevant, as it is the dominant approach in most papers in the literature.\n+ Although it is not its strongest point, several details on the architecture and the losses are novel. This might be a critical point in a machine learning conference as ICLR: there are a few small contributions but not groundbreaking advances in this paper. In the opinion of this reviewer, the novel contributions plus the significantly improved performance is enough for acceptance.\n+ The research is excellently written and presented.\n- I miss Patch-NetVLAD [Hausler et al. CVPR21] in the list of references and the baselines. I am not expecting it to outperform the authors' approach, as Hausler et al. report worse performance than DELG, that is outperformed by this research. But I find it very related to this research, as it also refers to local features.\n- I cannot understand why the authors refer to their approach as \"mid-level features\". I could not find anywhere in the paper an explanation on why the features after the transformer should be called like that. My understanding is that the LIT will leverage **all** the local features to produce the super-features. The super-features will incorporate then global context, and I wouldn't call a feature that incorporates information from all the image as mid-level... The authors should explain why this naming or remove it.", "summary_of_the_paper": "The paper proposes a novel model, denoted as super-features, for image retrieval, with the following specific contributions: 1) features are extracted using an iterative attention module (the Local Feature Integracion (LIT) Transformer). Although similar modules have been proposed before in the literature [Locatello 2020], the authors reference them properly and their proposal have several modifications (decorrelation loss, learning the initial templates, residual connections). 2) The model is trained using contrastive losses on the local features, opposed to the more spread training procedure using constraints on the global embeddings. The paper presents solid experimental results in public datasets and compare them against reasonably chosen baselines. ", "summary_of_the_review": "Summing up, although the paper does not have a very strong novelty, its several contributions together with the solid experimental results in a relevant application domain are sufficient for acceptance. I found two minor issues that I think the authors should fix, but the rest is excellently written and presented, which is also valuable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S3Z9A_NNhD-", "reviewer": "Reviewer_1P44", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n1. The paper has a clear motivation. The concept of super features is very intuitive.\n2. The majority of the design is reasonable. A transformer might be the best bet currently if we want to aggregate local features.\n3. Some parts of the experiments are interesting. Fig 6 clearly show how FIRe differs from HOW in selecting local features from all scales. The major results in Tab. 3 are quite promising.\n\n## Cons/Questions\n1. In Eq. (4), the attention maps are normalized twice alongside the axes of both L and N. In conventional attention, normalizing the attention along the key entries (counting to L) would be enough. So why should we do it again along the query entries?\n2. I think LIT implements a set transformer [a], at least a part of the set transformer, in the way of reducing and aggregating features according to a set of dummy queries in a QKV manner. I haven't seen any discussion on this. So maybe the authors can elaborate more on this and find more related articles. They should be very careful when claiming the novelty of LIT.\n3. The proposed model follows HOW to measure image similarity using ASMK. Does it have to be ASMK? As the authors mentioned in Sec. 3.1, S is ordered (which is quite different from the case of HOW). Simply comparing two super features from two sets on the same entries is reasonable as they reflect the same latent topic defined by Q. Aggregating the similarities on all entries only requires a time complexity of $O(Ld)$. I believe ASMK undergoes a higher time complexity of $O(1000^2d)$ according to the implementation. Why should we go for a more complex solution besides 'a fair comparison'?\n\n## Optional Questions\n4. $L_{attn}$ seems to be the solver in this paper regarding the oversmoothness problem. This design is just too empirical. Do we have a better option here?\n5. Do we have a better option to get multi-scale features than rendering the resized images multiple times? It seems this solution requires too much memory.\n\n------\n[a] Lee, Juho, et al. \"Set transformer: A framework for attention-based permutation-invariant neural networks.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "This paper introduces the concept of super features into image retrieval. The whole framework is named FIRe. The idea is to aggregate a set of features on the feature maps at different scales according to a set of query vectors representing different latent concepts. Hence, the proposed model somehow implements a transformer, which is called LIT in this paper.\n\nTo train this model, the authors define two losses. The first one is a triplet-like one executed on the super-feature granular. Some empirical criteria are established to select positive and negative super features. The second one decorrelates the super features from a single image.\n\nASMK becomes the measurement to compare two images when testing, which is a conventional option in comparing two groups of local features.\n\nWe see experiments on ROxford and RParis. The results are quite promising.", "summary_of_the_review": "I conservatively give a borderline before the discussion phase. The motivation and intuition of this paper are quite nice, but some parts of the details are questionable. I think the overall quality of this paper is about ok, but it doesn't hit. Some parts of FIRe are not absolutely novel. The authors need to defend these designs to convince the readers.\n\n------------------------------------------------------------------------\n\nThe authors' response satisfies me. Hence, I have changed my final rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EQifl_pDom", "reviewer": "Reviewer_9CXB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1) A new method called Feature Integration-based Retrieval or FIRe for short. \n2) Paper is well written.\n3) A comparisson with other state-of-the-art methods. Authors show that their methods permers better on selected databases.\n4) Reference section is updated.\n\nWeaknesses:\n\nNot particularly found.", "summary_of_the_paper": "Authors describe an architecture for deep image retrieval. For this, they use mid-level features that they call super-features (SF). They construct thes SF by an iterative attention module. They build them as an ordered set in which each element focuses on a localized yet discriminant image pattern. Auhtors show that for training, only image labels suffice. Authors present a set of experiments on common landmark retrieval benchmarks to validate that their SF substantially outperform state-of-the-art methods when using the same number of features. They also show that this requires a significantly smaller memory footprint to match their performance.", "summary_of_the_review": "Authors describe an architecture for deep image retrieval. For this, they use mid-level features that they call super-features (SF). They construct thes SF by an iterative attention module. They build them as an ordered set in which each element focuses on a localized yet discriminant image pattern. Auhtors show that for training, only image labels suffice. Authors present a set of experiments on common landmark retrieval benchmarks to validate that their SF substantially outperform state-of-the-art methods when using the same number of features. They also show that this requires a significantly smaller memory footprint to match their performance.\n\nMain contribution: A method called: Feature Integration-based Retrieval or FIRe for short. It can decomposed into 3 main parts:\n\n1) an image representation based on mid-level features (super-features) and an iterative module to extract them.\n2) a framework to learn such representations. It is based on a loss applied directly on SF yet only requiring image-level labels.\n3) a set of extensive evaluations that show significant performance gains over the state of the art for landmark image retrieval. \n\nSome possible works that could be added to reference section:\n\nIntellige: A User-Facing Model Explainer for Narrative Explanations\nJ Yang, D Negoescu, P Ahammad - arXiv preprint arXiv:2105.12941, 2021 - arxiv.org\n\nNext generation 3D pharmacophore modeling\nD Schaller, D Šribar, T Noonan, L Deng… - Wiley …, 2020 - Wiley Online Library\n\nAutomated cyberbullying detection in social media using an svm activated stacked convolution lstm network\nTA Buan, R Ramachandra - Proceedings of the 2020 the 4th …, 2020 - dl.acm.org\n\nAUTOMATIC MODULATION CLASSIFIER\nHA Rasool - Iraqi Journal of Information & Communications …, 2020 - ijict.edu.iq\n\nNext generation 3D pharmacophore modeling\nD Šribar, T Noonan, L Deng… - WIREs …, 2020 - refubium.fu-berlin.de\n\nSynergistic Target Combinations Against Obesity: Focus on MCHR1/H3R Modulation\nDA Schaller - 2020 - refubium.fu-berlin.de\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a model for large-scale image retrieval. Unlike previous work that rely on local features, the proposed method aggregates local features into the so-called Super-features to improve their discriminability and expressiveness. To do so, the method proposes an iterative attention module (Local Feature Integracion Transformer, LIT), that outputs an ordered set of such features. By exploiting the fact that features are ordered, the paper proposes a contrastive loss on Super-features that match across images. The paper presents a thorough empirical evaluation on several publicly available datasets including relevant baselines.

Overall the paper is well written and the empirical results are strong (including detailed ablations that motivate the design of the method). All reviewers and the AC appreciate the idea of applying the contrastive training at local feature level while only requiring image-level labels.

Reviewer hp4Y points out that the proposed LIT is not particularly novel, but previous work are properly cited. Also this is not a major issue given that the motivation is very clear, it is well executed and the empirical results are strong. 

Reviewer uoYN had initial concerns regarding inconsistencies in the mathematical formulation of the method, which were resolved in a detail (and constructive) discussion with the authors.

All reviewers recommend accepting the paper, three of which consider the contribution to be strong. The AC agrees with this assessment and recommends accepting the paper.
Decision: Accept (Poster)