Contextual Similarity Aggregation with Self-attention for Visual Re-ranking
Authors: Jianbo Ouyang
Hui Wu
Min Wang
Wengang Zhou
Houqiang Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Image retrieval
re-ranking

Abstract: In content-based image retrieval, the first-round retrieval result by simple visual feature comparison may be unsatisfactory, which can be refined by visual re-ranking techniques. In image retrieval, it is observed that the contextual similarity among the top-ranked images is an important clue to distinguish the semantic relevance. Inspired by this observation, in this paper, we propose a visual re-ranking method by contextual similarity aggregation with self-attention.  In our approach, for each image in the top-K ranking list, we represent it into an affinity feature vector by comparing it with a set of anchor images. Then, the affinity features of the top-K images are refined by aggregating the contextual information with a transformer encoder. Finally, the affinity features are used to recalculate the similarity scores between the query and the top-K images for re-ranking of the latter. To further improve the robustness of our re-ranking model and enhance the performance of our method, a new data augmentation scheme is designed. Since our re-ranking model is not directly involved with the visual feature used in the initial retrieval, it is ready to be applied to retrieval result lists obtained from various retrieval algorithms. We conduct comprehensive experiments on four benchmark datasets to demonstrate the generality and effectiveness of our proposed visual re-ranking method.

Contextual Similarity Aggregation with Self-attention for Visual Re-ranking
Jianbo Ouyang" Hui Wu!* Min Wang2t Wengang Zhou Houqiang Li
ICAS Key Laboratory of Technology in GIPAS, EEIS Department
University of Science and Technology of China
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center louy jb, wh241 @maiil.use ustt eduu wangmin@iai ustc. edu. en, (zhwg, Lihq/ @uste.... edu..eeddc.eo en
Abstract
In content- image retrieval, the first-round retrieval result by simple visual feature comparison may be unsatisfactory, which can be refined by visual re-ranking techniques. In image retrieval, it is observed that the contextual similarity among the top-ranked images is important clue to distinguish the semantic relevance. Inspired by this observation, this paper, we propose a visual re-ranking method by contextual similarity aggregation with self-attention. In approach, for each image in the top-K ranking list, we represent it into an affinity feature vector by comparing with set of anchor images. Then, the affinity features of the top-K K images are refined by aggregating the contextual information with a transformer encoder. Finally, the affinity features are used recalculate the similarity scores between the query and the top- K images for re-ranking of the latter. To further improve the robustness of our re-ranking model and enhance the performance of our method, new data augmentation scheme is designed. Since our re-ranking model is not directly involved with the visual feature used in the initial retrieval, it is ready to be applied to retrieval result lists obtained from various retrieval algorithms. We conduct comprehensive experiments on four benchmark datasets to demonstrate the generality and effectiveness of our proposed visual re-ranking method.
Introduction
In instance image retrieval, the goal is to efficiently identify images containing the same object or describing the same scene with the query image from a large corpus images. Towards this goal, many works have emerged in recent years [42, 21, 36, 26. 24]. With the development of deep learning, a great number methods leverage convolutional neural network (CNN) as feature extractor [38, 3, 44, 4, 14, 34, 25, 35, 45] to replace combine with the classic SIFT feature [23]. Generally, the performance of the initial retrieval results by simple comparison of visual feature may not be satisfactory. To refine it, many visual re-ranking techniques have been proposed [9, 19, 48).
Popular visual re-ranking techniques include query expansion, geometric context verification, kNN- based re-ranking, and diffusion-basee methods, etc. Query Expansion (QE) [9, 10, 39, 34, 30] computes an average feature the query and the top-ranked images to update the original query
"The first and second authors contribute equally to this work. "Correppodingg authors: Min Wang and Wengang Zhou Code will he released at https: : / thub. com MCC (000-OHHCCO WH/CSA.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

feature, which is then used for a second-round retrieval. Geometric Context Verification (GCV) [30, 12, 20,3 2] leverages the geometric context of local features to remove the false matches in the original results. Besides, if two images belong to the k-reciprocal nearest neighhors (kNN) of each other, these two images have a high probability of being relevant. Based on this observation, lots of k-Nearest Neighbors (kNN) based re-ranking methods thus emerge and achieve outstanding performance [19, 48, 50J. Diffusion-
methods [51, 11, 18] consider the global similarity of all images and perform similarity propagation iteratively. Most of the above methods do not involve training, thus can be quickly equipped with various features. The recent learning-basee methods such as LAtQE| [16] and GSS [22], achieve better performance but require training specific model for each kind of feature.
Different from the methods mentioned above, in this paper, we propose novel visual re-ranking method by contextual information aggregation. The initial retrieval results generated by feature comparison only consider pairwise image similarity, but ignore the rich contextual information contained in the ranking list. As illustrated in [29], if two images are mutually relevant, they share similar distances from set of anchor images. Thus, we directly select the top-L retrieval results as anchors for each query to fully model the contextual information in the ranking list. We define an affinity feature for each image in top-K candidates by computing the similarity between it and the anchor images. To further promote the re-ranking quality, we propose a new data augmentation method.
Moreover, inspired by Query Expansion (QE) which employs the information of top-ranked images to update the query feature, we propose to update affinity features of each top-K candidates by aggregating the affinity features of other candidates to promote the re-ranking performance with transformer encoder [46]. For each image in the top-K K candidates, our re-ranking model dynamically aggregates the affinity features of other candidates based on the importance between candidates by computing their similarities using the affinity features. The output of our re-ranking model can be regarded as the refined affinity features for top-K candidates which contain more contextual information.
To train the re-ranking model, two loss functions are introduced. Firstly, we use a contrastive loss to restrain the updated affinity features so that the relevant images have large similarity, and vice versa. Besides, we exploit a Mean Squared Error (MSF) loss to reserve the information in original affinity features by restricting the difference between the original affinity features and the refined affinity features. During the inference time, we compute the affinity features for the top- candidates and utilize the transformer encoder to refine the features. Then we re-rank these candidates by computing the cosine similarity between the refined affinity features. The rank of images outside the top-K ranking list remains unchanged.
Note that our re-ranking model is not directly involved with the original visual feature. Instead, computes the affinity features for l images, which serve as the input in our method. Therefore, it can be combined with various existing image retrieval algorithms including representation learning and re-ranking methods, and further enhance the retrieval performance with low computational overhead. We conduct comprehensive experiments on four benchmark datasets to prove the generality and effectiveness of our proposed visual re-ranning method. Besides, the time and memory complexity of our re-ranking method is lower than the state-of-the-an re-ranking methods.
2 Related Work
In this section, we review the related works on visual re-ranking including query expansion, geometric context verification, ANN based methods, and diffusion- -ased methods.
Query Expansion. Query Expansion (QE) uses the information of the top-ranked images obtained from the initial retrieval to refine the representation of the original query for further retrieval. Average query expansion (AQE) [9] aggregates the features of top-K K retrieval results by average pooling. Average query expansion with decay (AQEwD) [15] proposes a weighted average aggregation, where the weights decay over the rank of retrieved images. Discriminative query expansion (DQE) [1] regards the top-ranked images as positive images, and regards the bottom-ranked images as negative images to train linear SVM. Then DQE sorts images according to their signed distance from the decision boundary. In [34], a-weighted query expansion (aQE) weights the top- K images by their
2

---

First Retrieval Stage
Contextual Similarity Aggregation
Re-ranking
MSE Loss
Alfinity Feature
Query
Retrieval
1st
FC
2nd
Top-K images
MLP
2nd
Dotabase
Transformer Encoder
Contrastive Loss
3rd
Shared
4th
4th
Training
Top-L Anchor Images
Testing
Initial
Refined Affinity Fenture
Results
Final
The pipeline of our method. In the first retrieval stage marked in blue, given a query, we perform the first-round retrieval get the initial ranking list. In the second stage marked in green, we first obtain the affinity features for the query (orange cuboid) and top-K K candidates (blue cuboids) with top L images as anchor images. The transformer encoder is used to refine the affinity features. The network is trained by minimizing the contrastive loss and MSE loss using the refined affinity features. In the re-ranking phase marked orange, we recalculate the similarity scores between the query and the top K candidates using the refined affinity features. Best viewed in color. cosine similarity with the query. LAtQE [16] proposes an ceantion-bascd model to learn the weights of aggregation.
Figure 1:
Geometric Context Verification. Geometric Context Verification uses the geometric context of local features to remove the false matches. Some approaches [9, 30, 37] estimate the transformation model to verify the local correspondences by RANSAC-based methods. RANSAC [12] generates hypotheses on random sets of correspondences and then identifies geometric model with maximum inliers. Spatial coding [52] proposes to represent the relative spatial locations among local features into binary maps for efficient local matching verification. DSM[40] leverages the MSER regions which are detected on the activation maps of the convolutional layer. Then DSM regards these regions as local features for spatial verification.
k-NN-basee Re-ranking. The k-reciprocal nearest neighbors of an image are considered as highly relevant candidates [49, 32, 39]. Contextual Dissimilarity Measure (CDM) [19] iteratively regularizes the average distance of each point to its neighbors to update the similarity matrix. Visual Rank [48] determines the centrality of nodes on similarity graph according to the link structures among images to rank images. In CRL [27]. lightweight CNN model trained explore the contextual information and learn the relevance between images. GSS [22] proposes unsupervised method for re-ranking based on graph neural network neighbor encoding and pairwise similarity separation loss. In [50], the k-reciprocal feature is designed by encoding the k-reciprocal nearest neighbors into single vector to perform re-ranking.
Diffusion-based Re-ranking, Diffusion technique is initially designed for ranking on manifolds [51], 1], and it has been successfully applied to many computer vision tasks, such image classification, object detection, image retrieval, etc. In image retrieval, lot of research concentrates on propagating similarity through the &NN graph [11]. Diffusion usually used as a re-ranking method, and has achieved tatte-of-the-att performance on many benchmarks [28, 47, 5, 18, 33, 8, 7, 6, 13, 28]. Diffusion considers the underlying manifold structure and explores the internal relationships between images. As one of the representative methods, DFS [18] proposes a regional diffusion mechanism to further improve the recall of small objects based on neighborhood graphs. Then employs conjugate gradient method to compute the closed-fome solution of diffusion as well as maintaining time efficiency and accuracy.
Method
3.1 Framework Overview
In image retrieval, the database is defined as D = (1, I2..... Jw where T, denotes the ith database image and N is the database size. Usually, images are first mapped to high-dimensional vectors
3

---

through a feature extractor ( We define the feature representation of images as f = o() E R Given a query, we obtain an initial ranking list of the top- images R 12. .Tk by a retrieval algorithm, where ri denotes the ID of the ith image. For convenience of the following discussion, we assume the query image is returned at the first position in the ranking list, otherwise we directly insert it in the front of the ranking list. The features of top- K images in the ranking list Frx] E Rdd xx
are described as FkK- = [fr, fra.
Our framework is shown in Fig. 1. With the initial retrieval ranking list obtained by a query, we propose to represent each of the top-K candidates into an affinity feature vector by computing the cosine similarity between it and a set of anchor images. Then the affinity features are refined by transformer encoder to aggregate the contextual information in the ranking list.
Affinity Feature. Generally, if two images are relevant to the query, they shall be relevant to each other [43]. Moreover, as revealed in [29], if two images are mutually relevant, their distances set of pre-defined anchor images shall be similar. Actually, it is a non-trivial issue to select a proper set of images as anchors. Generally, if the selected anchor images are far from the top-K candidates in the ranking list, the variance of its distance with candidates is relatively small, which is difficult to provide useful information distinguish the candidates. In other words, a better alternative is to choose the top-L. L images in ranking list as the anchor images. With anchor images, we define the affinity features for the ith image in the rank list as follows:
fn, S., fra: ,
(1)
After that, we take the affinity features of the K images in R as sequence, which is fed into our re-ranking model for refinement. Note that the input of our proposed method is affinity feature sequence and is not directly related the original feature, which endows our method with good generality.
Contextual Similarity Aggregation with Transformer. Although the affinity features have taken into account the contextual information in the ranking list, there may be inconsistency in the initial affinity features obtained by directly comparing with anchor images. In order to make the affinity features of related images more consistent, we propose a new re-ranking model by contextual information aggregation to dynamically refine the affinity features of different images. Specifically, with transformer encoder as the core component of our re-ranking model, we first map the affinity feature a; E R2 for the ith image to a' E R² using a learnable projection matrix W,, E R and then feed the sequence a', i 1,2,... K into transformer encoder consisting of Multi-Head Attention (MHA) and
Networks (FFNs).
In MHA. each sequence element is updated by weighted sum of all other elements based on the scaled dot-product similarity. The scaled dot-product attention first maps affinity features a (i 1, 2, ... K) Queries (Q E RKxd,), Keys (K Rfxxd)) and Values (V R&xd.) with three learnable projection matrices. After that, the similarity between Q and K is aggregated with V together,
QKT
Attention AttentionQ,,K,, V) Softmax
VT. V
(2)
which can be seen as using the similarity between images as weights to aggregate affinity features. The head structure concatenates and fuses the outputs of multiple scaled dot-product attention modules using the learnable projection WM. MHA is defined as follows:
MHA(Q, K, V) = Connaaenation (/1, f2. ANN WM,
(3)
where
h; = Attention(Q:. K,,
.NH.
(4)
and NH is the head number. Note that to update the affinity feature of an image, the affinity features of all the relevant candidates should be considered as contextual information. A single-head self-attention layer limits the ability of the model to focus without affecting other equally important candidates. This can be achieved by projecting the original affinity features into different representation subspaces. Specifically, in MHA, different projection matrices for Q K, and V are used for different heads, and these matrices can project affinity features into different subspaces. The output is normalized via Layer Normalization (LN) and finally added to S' to form a residue connection,
or more relevant candidates simultaneously
S' + +LN(MHA(Q, K, V)).
(5)

---

In addition to MHA, the transformer encoder layer has position-wise feed-forward network.
We stack transformer encoder layers obtain more consistent affinity features. The refined affinity features yi, i = 1,2, ................................ K come from the output of the last layer of our re-ranking model.
Data Augmentation. According to the description above, represent the top- K images into affinity features, which serve as the input to our model. To improve the robustness of the re-ranking model and avoid overfitting, it is necessary to increase the size of the training sets. To this end, we design data augmentation scheme oriented to our framework.
Considering that our training data comes from retrieval results, we propose to use different visual features to obtain the ranking lists of the same query. With different types of features create the affinity features, the network will learn more information about the contextual information. For
query, we employ M different visual feature extractors (é(-), (6()),,(()22. 2......... ((.))r)) } to obtain different ranking lists, and compute their corresponding affinity features according to the previous section. The obtained M affinity feature sequences for the same query are used as training samples. this
way, we enlarge the training set size to M times of the original size and improve the robustness of our model.
3.2 Objective Functions
Contrastive Loss. The relevant images should have larger cosine similarity and vice versa, thus we design contrastive loss for representation learning, which is defined follows:
Lc = -log E exp(sim((y /T) 1(1 and rs are relevant)
(6)
Ei= exp( (1m U.)/T)
with the cosine similarity:
(7) where ||-|| is Le norm, I(-) is the indicator function, J is a temperature hyper-paramene parameter, and y is the refined affinity feature of the query because we assume that query is on the top of the ranking list. The output affinity features of other images in ranking list are represented as y:, 2 S i I K. In the numerator, the sum is over relevant images, and in the denominator, the sum is over all top-K' images in the ranking list. Contrastive loss minimized when query is similar to its relevant images and dissimilar to all other irrelevant images.
sim(ys. yj) y. y, /(lealllllll
MSE Loss. order to preserve the information contained in the original affinity features, we propose to use Mean Squared Error (MSE) loss between the original affinity features and the refined affinity features. MSE loss is defined as follows:
CM
aa: - MLP(y:)1
where and yi is the original and refined affinity feature for the ith candidate, respectively. The Multilayer Perceptron (MLP) containing two layers with GELU non-linearity projects the refined affinity features back to the original affinity feature space to compute the MSE loss.
Combining Eq. (6) and Eq. (8), we define the final objective function of the proposed method as follows:
L= Lc + ALM, where À is a hyper-parameter to indicate the importance of MSE loss.
3.3 Re-ranking Processing
To aggregate the contextual information, we represent the top-K images into affinity features and refine them by the transformer encoder. Because of the assumption that the query at the first position of the ranking list, we compute its similarity with the ith image in ranking list in R using the refined affinity features follows:
(10) Finally, all the K images in the initial ranking list R are re-ranked by their new similarity score defined in Eq. (10) in descending order. The order of the remaining images which out of top- K remains unchanged.
xf sim
= 1,2,..,......
5

---

Table 1: The re-ranking latency and retrieval accuracy of our method at different re-ranking lengths K on ROxf and Rar datasets. The initial retrieval denoted as R-GeM, which the baseline in our experiments. Anchor images list length L 512.
Re-ranking lateney (ms)
Medium
Hard
Method
ROxf
RPar
R0xf
RPar
R-GeM [34]
67.3
80.6
44.3
61.5
R-GeM+OOnx(K=128)
72.7
82.0
50.0
64.3
R-GeM+OOux(K=2500
75.1
83.7
53.4
67.7
R-GiMMOOux((K551)
19
77.0
85.6
57.0
71.3
R-GeM+OOux((3766)
77.8
86.9
58.3
73.4
R-GGMM+OOnx(K==0244)
37
77.9
87.2
58.4
74.4
B-GGMM+OOnx(K11288)
78.3
87.5
59.0
75.0
R- E-GeM+OOnx(K-115500
78.2
88.2
59.1
75.3
Table 2: The retrieval accuracy of our method at different anchor images list length L are compared with the initial retrieval performance on ROxf and RPar datasets. R-GeM denotes the first-round retrieval performance, which serves the baseline in our experiments. Re-ranking length K 1024.
Medium
Hard
Method
R0xf
RPar
ROxf
Rar
R R-GeM [34]
67.3
80.6
44.3
61.5
R-GeeM+Ourx((=188)
75.9
86.9
55.9
72.3
GeM++Our(((=256)
77.5
86.8
57.4
73.0
R+GGeM+HOux(/=5122)
77.9
87.2
58.4
74.4
R+GeMHHOrs(L=768)
77.0
87.2
57.0
73.7
R-GGMM+Ouux(/=1102)
76.5
87.1
56.1
73.4
R R-GeM+Our<(==1280)
76.4
87.3
56.6
73.7
Experiment
4.1 Experiment setup
Image Representation. We extract global visual features for both the query and database images using a ResNet101 [17] backbone with GeM pooling [34]. The best-performing model fine-tuned on the GL18 [25] is used. The resulting feature is denoted as R R-GeM. To verify the robustness of our method when the training and testing sets are represented by different features, we also utilize other four models to extract testing image features: the off-the-shelf version of ResNet101 with R-MAC pooling [44], the off-the-shelf version of ResNer101 with GeM pooling, the fine-tuned version of ResNet101 with Max pooling [44], and the fine-tuned version of VGG16 [41] with GeM pooling.
Evaluation Datasets and Metrics. Four image retrieval benchmark datasets, named Revisited Oxford5k (ROxf). Revisted Parisók (RPar), ROxf+ RIM, and RPar + RIM, are used to evaluate our method. The ROxf [33] and RPar [33] datasets are the revisited version of the original Oxford5k [30] and Parisók datasets [31]. These two datasets both contain 70 query images depicting buildings, and additionally include 4,993 and 6,322 database images, respectively. ROxf + R.M and RPar R.M
are the large-scale versions of ROxf and Par which combine set of 1M distractor images with the small ones. Mean Average Precision (mAP) [30] is used to evaluate the performance. We report Medium and Hard performance of the four datasets mentioned above. The computation is performed on a single 2080Ti GPU.
Training Details. rSfM120K [34] is used to create training samples. It includes images selected from 3D reconstructions of landmarks and city scenes. In total, 91,642 images from 551 3D models are used for training. Each image in the training set is considered as query image, and the others are database images. The first-round retrieval results of each query form a training sample. For each query image, the image with the same 3D reconstruction cluster id is considered as positive sample and vice versa as a negative sample. We select the top-512 returned images for each query image to form training sample and select the top-512 images anchor images for computing the affinity features. To realize data augmentation, we extract the training image features using multiple models. Specifically, for each image in the training set, extract features using fine-tuned Resnet50,

---

Table 3: mAP performance of the proposed model with data augmentation. Re-ranking length K=1024, anchor images list length 512. rSfM120K denotes the original training set. AugrSfM1200 denotes the training set with data augmentation.
Training Dataset
Training nums
Medium
Hard
ROxf
RPar
ROxf
RPar
#SEM120K [34]
91.642
77.9
87.2
58.4
74.4
AugrSIM120K
274,926
78.5
88.1
60.0
76.3
Resnet101, and Resnet152 with GeM pooling, respectively, to construct multiple different affinity features. We denote the training set with data augmentation as AugrSfM 120k.
Our model consists of stack transformer encoder layers, each with 12 heads of 64 dimensions. The fully connected layers within the encoder layers have 4 times more dimensions than the hidden dimension. SGD used optimize the model, with an initial learning rate of 0.1, weight decay of 10-5, and a momentum of 0.9. We use cosine scheduler to gradually decay the learning rate to O. The temperature in Eq. (6) is set as 2.0. The batch size is set to 256. The model is trained for 100 epochs on four 2080Ti GPUs.
4.2 Ablation Study
Re-ranking Length. In our method, we re-rank the initial top-K images of the ranking list and keep the order of the remaining images unchanged. For training, we select the fixed-length top-512 results as training samples. Since our transformer encoder involves no position embedding and the input length of the model is variable, we can change the length of the re-ranking during testing. The ve-ranking performance of different K on ROxf and RPar and the corresponding time required are shown in Table 1. The results show that the performance of re-ranking keeps improving as K increases, and the performance starts saturate after K is larger than 1024. For practical applications. we can take a trade-off between accuracy and latency time of re-ranking.
Table 4: mAP performance of the proposed model with different feature types. Te re-ranking model trained by fine-tuned R-GeM. Re-ranking length K 1024. V: VGG16 [41]; R: ResNet101 [17]; [O]: Off-the-shell networks pretrained on ImageNet; RMAC: regional max max--ooling [44]; GeM: generalized-mett mean pooling [34]; MAC: max
max-pooling pooling [44].
Medium
Hard
Method
Training feature
ROxf
RPar
ROxf
RPar
R-RMACTO] [44] R-RMAC[O]+Ours
51.2
74.0
21.4
51.7
R-GeM
56.8
81.9
30.5
65.7
R-GeM[O] [34] R-GeM[OJ+Ours
50.3
73.0
23.0
50,9
R-GeM
55.0
81.5
30.3
65.6
R-MAC [44]
63.3
76.6
35.7
55.5
R-MAC+Ours
R-GeM
73.2
86.0
52.8
72.1
V-GeM [34]
61.6
69.3
34.3
44,9
V-GeM+Ours
R-GeM
73.3
81.5
50.0
73.0
Length of Anchor Images List. Our method updates the affinity features to capture the contextual information in the ranking list, and the choice anchor images plays a key role. We seleet the top-L images of the returned list as the anchor images for each query. Table 2 shows the effect of different anchor image lengths. The re-ranking achieves a large improvement relative to the baseline for all settings. When L is small, it limits the expressiveness of the affinity vector, and conversely when L large, the proportion of images associated with the query image is too small, introducing a lot of noisy anchor images and limiting the discriminative power of the model. When L 512, the best competitive performance achieved on all datasets.
Model Variants. In Figure 2, we show the impact of different model variants the retrieval performance. The optimal performance is achieved when the transformer depth is equal and the hidden layer dimension is 1024. As seen in Figure 2(a), when the depth is zero, the model contains
7

---

60
60
60
(1%)
-55
mAP((%)
55
mAP(%)
55
50
30
R.G:M
GeM-Uuus
R G
8
12
16
20
32
0.0
0.1
0.2
0.3
0.4
0.5
0.6
(a) Depth
(b) Dimension (x64)
(C)
Figure 2: Impact of transformer model variants and the weight the MSE loss on mAP on ROxf with Hard evaluation protocols. All model variants are evaluated with re-ranking length K = 1024 and anchor image list length
512. (a) Comparisons of different transformer depth n. The hidden size L' 768, and head number Nv. = 12. (b) Impact of the hidden size L'. Transformer depth is kept as (c) Impact of the weight of the MSE loss. The model takes the default settings.
self-attention module and is simply a stack of fully connected layers, at which point the performance lower than the baseline features, highlighting how important the self-attention is for our model. Figure 2(b) shows that the model performance rises and then falls as the model capacity increases. This can be attributed the overfitting of the model.
MSE Loss. The effect seen that as À increases, the performance gradually rises and reaches the highest when À = 0.2. An excessively large makes the model learning be dominated by the MSE loss and the performance decreases.
MSE loss on the model performance is shown in Figure 2(c). can be
Dataset augmentation. As shown Table 3, when using data augmentation, we can increase the training set to three times the original size, achieving higher performance relative to the baseline for the model trained without data augmentation. The re-ranking setting the same as the compared methods which uses ResNet101-Ger feature to perform retrieval and re-rank using our trained model.
4.3 Cross Feature Testing
Our method just requires the affinity feature generated by comparing the image with anchor images, and is not directly related the original feature. can be accomplished with different visual features. We use fine-tuned R- R-GeM feature for the first-round retrieval and compute the affinity features for top candidates train the re-ranking model. Then we test our model on various (type of features, fine-tuned or not) features. The performance of our method for different testing features is shown in Table obtains a large improvement relative to baseline several settings.
4.4 Comparison with the state-of-the-ate methods
mAP Comparison. Table reports the performance of our method and different compared methods on ROxf, R.arr ROxf RIM, RPar R.M. We compare our method with the following methods: QE and its variants as well as Database Augmentation (DBA), DSM [40], CRL [27], GSS [22], and DFS [11]. In Table 5 if the visual feature in the first-round retrieval of comparison methods in the original paper is the same with our feature, we directly use the results in the corresponding paper. For example, the results
second and the third blocks in Table are copied from LAuQE [16].
While in the fourth block in Table 5, the visual features of comparison methods in their original paper are different from our feature. Therefore, we re-test DSM [40] and DFS [11], re-train GSS [22] with feature by the released code for fair comparison. As for CRL [27], we re-implement it with our features by our own.
The proposed re-ranking method achieves higher performance than most of the compared methods in all settings. Our method obtains better performance compared with LAuQE [16], which is the state- of-the-art art method among QE-based methods. DFS [18] and GSS [22] can achieve better performance than ours some settings since they utilize database images for re-ranking but inevitably incur expensive extra time costs when the database size is large. We further combine DFS and GSS with our re-ranking approach, i.e., we apply our re-ranking approach the list of retrieved results based on DFS GSS re-rankae re-ranking and the retrieval performance is further improved. Notably, when using
8

---

Table 5: mAP comparison against existing methods on the testing datasets, with Medium and Hard evaluation protocols. The performance our method is evaluated based on the optimal settings of K and L. AugrSfM120s and rSfM120K denote the training datasets with and without data augmentation, respectively.
Medium
Hard
Method
R0xf
ROX++RIM
RPar
RPa++RIM
ROxf
ROXf+RIM
RPar
RPar+RIM
R-GeMiNo re-ranking) [34]
67.3
49.5
80.6
$7.3
44.3
25.7
61.5
29.8
Aflinity Feature
72.0
52.8
82.9
66.5
51.1
30.5
66.8
43.4
AQE [9]
72.3
57.3
82.7
62.3
49.0
30.5
65.1
36.5
AQEwD [15]
72.0
$6.9
DQE [1]
72.7
54.5
83.7
64.2
48.8
26.3
66.5
38.0
aQB [34]
69.3
52.5
86.9
66.5
44.5
26.
71.7
41.6
LAIQE [16]
73.4
58.3
86.3
67.3
49.6
70.6
42.4
ADBA AQF. [9]
71.9
55.3
83.9
65.0
53.6
32.8
68.0
39.6
ADBAwD AQEwD [15]
73.2
57.9
84.3
65.6
53.2
34,0
68.7
40.8
DQQ [1]
72.0
$6.9
aDBA + oQE [34]
71.7
56.0
87.5
70.6
50.7
31.5
73.5
48.5
LAIDBA LAIQB [16]
74.0
60.0
87.8
70.5
54.1
36.3
74.1
48.3
[40]
67.1
50.7
80.5
57.4
43.7
26.6
30.1
CRL [27]
72.0
54.4
83.3
62.6
50.0
30.7
67.4
38.4
GSS [22]
78.0
57.8
88.9
84.8
60.9
34.3
76.5
69.2
[18]
73.3
65.1
89.7
48.3
Ours(rSfM120k)
78.2
61.5
88.2
71.6
59.1
38.2
75.3
51.0
GSS-OUrs((SSM11200
79.3
62.1
90.7
85.1
62,2
42.3
80.0
70.3
DIS+OursirSMM1200,
76.3
Ours/AugrS(M1120)
80.3
62.8
90.0
73.0
62.4
78.6
53.0
GSS+Ours(Ang:S/M
AMgneAAAAASS/120... 20k)
79.0
64.1
91.4
85.3
62.0
44.0
81.0
70.2
DFS+Ours/Aug::SMM122
79.2
69.2
90.3
86.0
61.1
47.2
74.8
Table 6: Complexity comparison for different re-ranking methods. Latency is measured on an NVIDIA GTX 2080Ti GPU. number of nearest neighbors; iteration times in DFS; K: number of candidates to be re-ranked; N: number of database images; d: feature dimensionality; L:anchor image list length; Cplx: complexity.
Re-ranking latency (ms)
Memory (GB)
Method
Space Cplx.
Time Cplx.
ROxf+RIM
RPar+RIM
DFS
O(Nd NR)
O(tk
7.81
7.82
GSS [22]
O(Nd Nk)
OINd 1Pd)
317
15.37
15.39
aQE [34]
((Nd)
O(Nd+ kd)
184
7.68
7.69
Ourst 15560
((Nd)
O(KLd + K2)
58
7.68
7.69
Ours( k=0)
O(Nd)
O(KTd K
7.68
7.69
Ours(X==0244
O(Nd)
O(KLd+ K2)
7.68
7.69
data augmentation, achieve higher performance relative the baseline for the model trained without data augmentation. We show some successful cases and failed cases of our method on ROxf dataset in Figure 3.
Speed and Memory Costs. In Table 6, we present the complexity the different methods in time and space well as the measured average re-ranking latency (ms) and the total memory overhead required on the ROxf RIM, and RPar + R.M datasets. terms of spatial complexity, both our method and the QE-based method [34] only involve the top-K K initial returned images. Differently,
DFS [18] needs to encode the neighbor information between database images, resulting in an overhead to store the neighbor graph. As for GSS [34], it requires the neighbor relationship update the features the query image. Besides, since involves the GCN network, the updated features cannot be used to calculate the distance directly with the original features of the database images. So additional storage the updated features of the database images required.
As for time complexity. DFS performs iterative operations on the top- K returned candidates (generally K = 10, 000, with over 10 iterations). QE-based methods and GSS need first update the query features and then perform a second-round retrieval, which leads to larger latency when the size of the database becomes larger. Our method needs to firstly calculate the similarity between the top-K returned candidates and the L anchor images (usually L 512, K 512), and then use

---

Inital
Query
Initial
Ours
Quiry
Successes
Failures
Initial
Initil
Query
Figure 3: Selected qualitative examples of re-ranking method. We show the top- 10 results in the
figure. The figure is divided into four groups each of which consists of result of initial retrieval and result of our re-ranking method. The first two groups the successful cases and the other two groups are the failed cases. The images on the left with orange bounding boxes are the queries. The images with green bounding boxes denote the true positives and the red bounding boxes are false positives, Best viewed in color.
transformer encoder to generate the features, leading to a secondary complexity about K. Besides, the computation complexity of our method does not increase as the size of the database grows.
Conclusion
In this paper, we propose novel visual re-ranking method for instance image retrieval. We represent the top-K K images in the ranking list with affinity features according to the top-I. anchor images. In order to explore the contextual similarity information in the ranking list, we design self-attention re- ranking model, which updates the affinity features for re-ranking. Besides, our method is robust to the retrieval algorithm based on other image feature representation since the input of our re-ranking model not directly related to the original image features. We further propose a data augmentation method to improve the robustness of the re-ranking model and avoid overfitting. Extensive experiments four datasets show that our method achieves promising results compared with existing state-of-the-an methods in terms of both retrieval performance and computational time.
6 Acknowledgments
This work was supported in part by the National Key R&D Program of China under Contract 2018YFB1402605, in part by the National Natural Science Foundation of China under Contract 62102128, 61822208, and 62021001, and in part by the Youth Innovation Promotion Association CAS also supported by the GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
under Grant 2018497. It
References
[1] Relja Arandjelović and Andrew Zisserman. Three things everyone should know to improve object retrieval. In Proceedings of the IEEE International Conference Computer Vision and
10

---

Pattern Recognition, pages 2911-2915 2012. 2,9
[2] Yannis Avrithis and Giorgos Tolias. Hough pyramid matching: Speeded-up geometry re-ranking for large scale image retrieval. International Journal of Computer Vision, 107(1): 2014.
[3] Artem Babenko and Victor Lempitsky. Aggregating local deep features for image retrieval. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 1269-1277, 2015.
[4] Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. Neural codes for image retrieval. In Proceedings of the IEEE International Conference on Computer Vision and Puttern Recognition, pages 584-599, 2014. 1
[5] Song Bai, Xiang Bai, Qi Tian, and Longin Jan Latecki. Regularized diffusion process for visual retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017. 3
[6] Song Bai, Zhichao Zhou, Jingdong Wang, Xiang Bai, Longin Jan Latecki, and Qi Tian. Ensem- ble diffusion for retrieval. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2017.
[7] Song Bai, Xiang Bai, Qi Tian, and Longin Jan Latecki. Regularized diffusion process bidircctional context for object retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:1213- 1226, 2019.
[8] Song Bai, Peng Tang, Philip HS Torr, and Longin Jan Latecki. Re-ranking via metric fusion for object retrieval and person re-identification. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2019.
[9] Ondrej Chum, James Philbin, Josef Sivic, Michael Isard, and Andrew Zisserman. Total recall: Automatic query expansion with a generative feature model for object retrieval. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition, pages 1-8, 2007. 1, 3,
[10] Ondfej Chum, Andrej Mikulik, Michal Perdoch, and Jifí Matas. Total recall ii: query expansion revisited. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 889-896, 2011.
[11] Michael Donoser and Horst Bischof. Diffusion processes for retrieval revisited. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2013. 2,
8
[12] Martin Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395, 1981. 2,
[13] Zhanning Gao, Jianru Xue, Wengang Zhou, Shanmin Pang. and Qi Tian. Democratic diffusion aggregation for image retrieval. IEEE Transactions on Multimedia, 186)):66111174 2016.
[14] Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. In Proceedings of the European Conference on Computer Vision, pages 241-251 2016.
[15] Albert Gordo, Jon Almazan, Jerome Revaud, and Diane Larlus. End-to-end learning of deep visual representations for image retrieval. International Journal of Computer Vision, 124(2): 237-254, 2017. 2,9
[16] Albert Gordo, Filip Radenovic, and Tamara Berg. Attention-based query expansion learning. In Proceedings of the European Conference on Computer Vision, pages 172-188, 2020. 2, 3, 8,
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016. 6.

---

[18] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis. Teddy Furon, and Ondrej Chum. Efficient diffusion on region manifolds: Recovering small objects with compact enn representations. In Proceedings of the IEFE International Conference on Computer Vision and Pattern Recognition, pages 2077-2086, 2017. 2,3, 8, 9
[19] Herve Jegou, Hedi Harzallah, and Cordelia Schmid. A contextual dissimilarity measure for accurate and efficient image search. In Proceedings of the IEFE International Conference on Computer Vision and Pattern Recognition, pages 1-8, 2007. 1, 2,
[20] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Hamming embedding and weak geometric consistency for large scale image search. In Proceedings of the European Conference on Computer Vision, pages 304-317 2008.
[21] Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into a compact image representation. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 3304-3311, 2010.
[22] Chundi Liu, Guangwei Yu, Maksims Volkovs, Cheng Chang, Himanshu Rai, Junwei Ma, and Satya Krishna Gorti. Guided similarity separation for image retrieval. In Conference on Neural Information Processing Systems, 2019. 3, 8,
[23] David G Lowe. Distinctive image features from scale-invarianl keypoints. International Journal of Computer Vision, 60(2):91-110, 2004.
[24] Anastasiya Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. In Conference on Neural Information Processing Systems, 2017.
[25] Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large- scale image retrieval with attentive deep local features. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 3456-3465, 2017. 1,6
[26] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-net: Learning local features from images. 2018.
[27] Jianbo Ouyang, Wengang Zhou, Min Wang, Qi Tian, and Houqiang Li. Collaborative image relevance learning for visual re-ranking. IEEE Transactions on Multimedia, 2020. 3, 8,
[28] Shanmin Pang, Jin Ma, Jianru Xue, Jihua Zhu, and Vicente Ordonez. Deep feature aggregation and image re-ranking with heat diffusion for image retrieval. IEEE Transactions on Multimedia, 2 1116):5133 2018. 3
[29] Daniel Carlos Guimaraes Pedronette and Ricardo da S Torres. Distances correlation for re- ranking in content-based based image retrieval. In SIBGRAPI Conference on Graphics, Patterns and Images, pages 1-8, 2010. 2,
[30] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman, retrivval with large vocabularies and fast spatial matching. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition, pages 1-8, 2007. 1,2,3,6
[31] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Lost in quantization: Improving particular object retrieval in large scale image databases. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1-8, 2008. 6
[32] Danfeng Qin, Stephan Gammeter, Lukas Bossard, Till Quack, and Luc Van Gool. Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors. In Proceedings of the IEEE International Conference on Computer Vision and Patlern Recognition, pages 777-784, 2011.
[33] Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondřej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition, 2018. 3,
12

---

[34] Filipp Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning enn image retrieval with no human annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7): 1665-1668, 2018. 2, 7,9
[35] Jerome Revaud, Jon Almazán, Rafael S Rezende, and Cesar Roberto de Souza. Learning with average precision: Training image retrieval with listwise loss. Proceedings of the IEEE International Conference on Computer Vision and Puttern Recognition, 2019.
[36] Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. pages 14405- 2019.
[37] Torsten Sattler, Michal Havlena, Konrad Schindler, and Mare Pollefeys. Large-scale location recognition and the geometric burstiness problem. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1582-1590, 2016. 3
[38] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition Workshops, pages 806-813, 2014.
[39] Xiaohui Shen, Zhe Lin, Jonathan Brandt, Shai Avidan, and Ying Wu. Object retrieval and localization with ppatiall-conssrrimne similarity measure and k-nn re-ranking. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 3013 -0200,00,0200 2012. 1,3
[40] Oriane Siméoni, Yannis Avrithis, and Ondrej Chum. Local features and visual words emerge in activations. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition, 2019. 3, 8,
[41] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. 6,
[42] Josef Sivic and Andrew Zisserman. Video google: text retrieval approach to object matching in videos. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pages 1470-1477, 2003. 1
[43] Xinmei Tian, Yijuan Lu, Linjun Yang, and Qi Tian. Learning to judge image search results, In Proceedings of the ACM International Conference on Multimedia, pages 363-372, 2011. 4
[44] Giorgos Tolias. Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max- pooling of enn activations. In Proceedings of the International Conference on Learning Representations, 2016. 1, 6,7
[45] Giorgos Tolias, Tomas Jenicek, and Ondřej Chum. Learning and aggregating deep local descriptors for instance-level recognition. Proceedings of the European Conference on Computer Vision, pages 460-477, 2020.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Conference on Neural Information Processing Systems, 2017. 2
[47] Xingwei Yang, Lakshman Prasad, and Longin Jan Latecki. Affinity learning with diffusion on tensor product graph. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1): 28-38, 2013. 3
[48] Jing Yushi and Baluja Shumeet. Visualrank: applying pagerank large-scale image search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18877111877 2008 1, 2, 3
[49] Liang Zheng, Yi Yang, and Qi Tian. Sift meets enn: decade survey of instance retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5): 2018. 3
[50] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identification with k-reciprocal encoding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1318-1327 2017. 2,
13

---

[51] Dengyong Zhou, Jason Weston, Arthur Gretton, Olivier Bousquet, and Bernhard Schölkopf. Ranking on data manifolds. In Conference on Neural Information Processing Systems, pages 169-176, 2004. 2,3
[52] Wengang Zhou, Yijuan Lu, Houqiang Li, Yibing Song, and Tian Qi. Spatial coding for large scale partial-duplicat web image search. In Proceedings of the ACM International Conference on Multimedia, pages 131-140, 2010. 3
14


Reviews:
[{"review_id": "aOnevVzTtR1", "reviewer": "Reviewer_PEsN", "summary": "The paper proposes a re-ranking algorithm for image retrieval, capable of improving the ranking results of an off-the-shelf similarity-based retrieval system, by representing each query by an affinity vector that measures similarities to a set of anchor images, and then refine this representation by aggregating it with contextual information obtained from other top-ranked retrieval results using a transformer architecture. The refined representations of the top-K images can then be used for re-ranking. \n\nThe paper shows improvements both in precision and time complexity over previous re-ranking algorithms. The time complexity improvements are mostly due to the introduction of a fixed set of anchor images, which avoids context affinity features that scale linearly with the number of top-k retrievals considered. The precision improvements are a result of a learnable context aggregation module (in the form of a transformer) and the data augmentations used for training it.\n\nExperiments on several benchmarks show healthy improvements over prior re-ranking procedures.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces a re-ranking algorithm for content-based image retrieval. As mentioned above, the proposed procedure yields performance improvements both in terms of precision and computational complexity. These improvements were shown in several benchmark datasets. The proposed method is interesting as it seeks to *learn* how to best aggregate and re-rank the outputs of a similarity-based retrieval system.\n\nI am overall positive about the paper. Nevertheless, I hope the authors can clarify/address the two following points. I will adjust my final recommendation based on that.\n\nThe paper claims to introduce a new data augmentation technique for training the re-ranking model. However, this seems to be using different CNN backbones to generate multiple features for each image, and treat these features as augmentations. This technique does improve performance significantly. However, I'm wondering if the improvements are mostly due to the use of multiple backbones, as in the case of ensembles (often used for competitions)? Is the proposed method using all backbones at test time, or only one? if only one, which one? if not, how did you ensure a fair comparison to other methods (since it would be similar to an ensemble)?\n\nThe fixed number of anchors reduces the time complexity of the proposed procedure without affecting its precision. Table 2 also shows that the method is not too sensitive to this parameter. However, I'm concerned whether this is because evaluations are being done on datasets with a relatively low number of concepts, and so a small set of anchors can cover all the concepts. Would there be a significant performance hit if the method was applied to datasets with a large number of concepts like ImageNet? Would the method need a much large number of anchors in order to maintain high performance? And if so, wouldn't the computational gains be diminished?\n\n-----\n\nPost rebuttal:\nThanks to the authors for the rebuttal, and the other reviewers for their insightful reviews. After carefully considering all inputs on the paper, I will increase my rating to 7. The paper makes progress in content-based image retrieval, both in terms of precision and computational complexity, by improving upon prior re-ranking algorithms. The authors also addressed my main concerns about the fairness of their comparisons. I strongly suggest the authors to clearly articulate in the paper how the algorithm operates during deployment (in order to avoid such confusions in the future).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper aims to improve image retrieval (an established research area). As such, it does not introduce new negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours split over 2 days"}, "extra_text": {}}, {"review_id": "HoyoTMbtch0", "reviewer": "Reviewer_3tLh", "summary": "Paper presents a method to perform top-k reranking on a (image) retrieval system. The method uses external data to learn, in a supervised manner,  a reranking module that produces contextualized representations (by means of self attention) of the top-k results, which are then used to rerank them. \nThe method seems inspired by the recent LattQE [16], that also uses a reranking module based on self attention and trained in a supervised manner. However, I see this paper has relevant contributions / differences:\n- The proposed method uses \"affinity\" representations as the input of the reranking module, that probably capture better the similarity space.\n- The form of supervision is different, computing a contrastive loss only amongst the top-k results. \n- In addition, a \"fidelity\" loss and a different form of data augmentation is proposed.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I enjoyed reading this paper. \n\n- In terms of novelty, I think the paper is somewhat incremental from the work of [16]. However, I think enough novel technical contributions are proposed, making it, in my opinion, barely above the threshold.\n\n- In terms of clarity, the paper is generally easy to follow, although there are some technical details I could not understand completely, I'm asking some clarifying questions to the authors below.\n\n- The proposed method and decisions seem reasonable as well. There are some decisions I find somewhat surprising though, again, I will ask clarifying questions below.\n\n- In terms of significance / quantitative evaluation, the method clearly outperform existing approaches by a significant margin on two standard benchmarks (plus augmented versions). The model achieves good results both on rOxford and rParis using the same model, which is not common (typically, models that work on one dataset tend to fail in the other due to differences in statistics within the datasets, for example number of relevant images per query). On the negative side, I wish the authors could have reported results on more datasets (e.g. Instre). I've seen many papers accepted reporting results only on rOxford and rParis, so this is not grounds for rejection, but I wish the authors would consider including those results as well. Also, Pytorch code was provided, which will help other researchers build on top of this work. Thank you authors for making the code available.\n\nOn the negative side, it seems to me (and please authors correct me if I'm wrong) that some hyper parameters are validated directly on test (see, e.g. Figure 2, where Roxf hard is used). This is a barely-discussed, dark standard practice in the retrieval community. I am not going to strongly penalize the authors for this when essentially every other paper gets away with it. However, I'd urge the authors to reconsider validating the hyper parameters in rSfM120k or other dataset (and perhaps add the results validating in rOxf in the supplementary material)\n\nQuestions for authors:\n\n- It was unclear under which circumstances different features were used. Was this used only to augment the training data by generating different lists of top-k results, or was it also used to generate different affinity features (in the sense of the dot product scores being different because the features are different, not in the sense of the L anchors being different). Also, does this happen only at training time, or also at test time?\n\n- I understand that no form of database augmentation (e.g. DBA) is happening, is that correct?\n\n- Can you elaborate a bit more about the need for the MSE loss? It is unclear to me why we'd want the refined features to be similar to the original features (condition on an MLP transformation). Seems like an adhoc decision.\n\n- Did you experiment training this reranking model in GL18 (since that's where the features seem to be coming from)? Not asking the authors to do this experiment if they didn't, just wondering if they actually did it and if there's any insight (e.g. performing worse because of more overfit).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Paper describes some limitations (particularly, if the first round of results is bad, reranking will not work well).\n\n- Paper does not have a societal impact section. Although I understand this is not as obvious here as in other problems (e.g. face recognition), I think authors could have made an effort here. For example, is it possible that when trying to retrieve images with very very few relevant items (because they are not classic western-centric landmarks), the reranking actually makes the results worse?", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9-0TexqGK5I", "reviewer": "Reviewer_zg23", "summary": "The paper introduces a new re-ranking method for image retrieval. It proposes to learn an affinity vector for each of the top-ranked candidates w.r.t a set of anchor images, using a transformer encoder. These affinity vectors are then used to re-estimate the similarity scores between the top-ranked candidates and the query image. The proposed method is evaluated on the Revisited Oxford and Paris datasets and shown to be superior or complementary to other query expansion/diffusion based reranking methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "------------------------------------------------------------------------------------------------------------------\n\nStrengths\n\n[S1] The paper proposes to aggregate the contextual similarities between the query image and the top-ranked candidates for image reranking using a transformer based model. The proposed method is technically sound. Compared to other reranking methods, the proposed method is also more flexible in the sense that only the affinity vector for each image is used. Therefore, the training features and testing features can be from different feature extractors. It also enables cross-feature based data augmentation.\n\n[S2] The paper is easy to follow.\n\n[S3] The proposed method is shown to be superior or at least complementary to state-of-the-art query expansion/diffusion based reranking methods. Ablations on several components (e.g. # of images to be reranked, # of anchor images, data augmentation, etc) are also included.\n\n------------------------------------------------------------------------------------------------------------------\nConcerns\n\n[C1] For the comparisons shown in Table 5, I wonder if the scores of the baselines are from their original papers or re-training/re-testing/re-implementation? As the numbers for some baselines appear to be inconsistent with those reported in their original papers. For example, the mAP for GSS[22] on RParis Medium is {92.4}/{88.9} in {the original paper}/{this paper}.\n\n[C2] For the comparisons shown in Table 5, I wonder if the proposed method and the baselines are evaluated under similar settings. For example, i) are they using the same initial ranking lists? ii) are they using the same feature extractor, i.e. R-GeM? This is important for fair comparisons and justifying the real advantage of the proposed method.\n\nGiven the concerns listed above, I’d like to rate the paper as “marginally below the acceptance threshold” at this moment.\n\n------------------------------------------------------------------------------------------------------------------\nMinor comments\n\n[M1] Table 5, DSM [9] → DSM [40]\n\n-------------------------------------------------------------------------------------------------------------------\nPost rebuttal\n\nI have read the reviews and the authors' replies. In general, I think the idea of using only the affinity vectors for reranking is interesting. It makes it possible to reuse the pretrained reranking model for different feature extractors. The authors also addressed my concerns on the evaluation: if all the methods were tested under the same setting. Therefore, I raise my score and vote for accepting the paper.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't think there will be any potential negative societal impact of the paper. For the limitations of the paper, please refer to the \"concerns\" part of the main review.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have any ethical concerns about the paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "1gTe94iEO6a", "reviewer": "Reviewer_wuFH", "summary": "In this paper, a visual re-ranking method is proposed. This method leverages the contextual similarity of the top-K retrieved images by comparing the original features with a set of anchor images, which is further refined by a transformer encoder. The refined features are then used to re-rank the top-K ranking list.\n\nMoreover, a new data augmentation scheme is designed by employing different feature extractors, which enlarges the feature variety in the ranking list and improve the robustness of the re-ranking model.\n\nComprehensive experiments have been conducted on four benchmarks, demonstrating clear improvement and strong robustness.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality \nA new re-ranking method is proposed in this paper, the proposed method differs to previous works by extracting contextual similarity from top-K images and refining them with an attention-based model. Different kinds of re-ranking methods have been included in the related work, but the discussion about the relationship between the proposed work and previous works is missing.\n\n### Quality\nThe proposed method is clearly described, the contribution of each component is well demonstrated in the ablation studies. Comparison with state-of-the-art methods have also been given to support the effectiveness and generality of the proposed method. Time and memory complexity comparison is provided.\n\n### Clarity\nOverall, the paper is super clear and well structured. A few minor changes could be done to resolve the following issues:\n* (**L154-156**): what is the dimension of the learnable projection matrix $W_M$, and what is the size of vector $h_i$ ?\n* (**L163**): what is $S'$ and how do we initialize it ?\n* (**Table-5**): what does the entry of *Affinity Feature* means? And maybe merge the second block and the third block of QE methods by having two lines for each QE such as AQE and AQE + DBA.\n* (**Temperature parameter $\\tau$**): seeing the importance of this parameter in the contrastive loss, maybe include an ablation study of this hyper-parameter to show how robust is the method against this hyper-param.\n\n### Significance\nThe well-designed and easy-to-use re-ranking method could benefit many related applications. Besides, the usage of affinity feature and attention-based encoder could help the community understands better the effectiveness of transformer-based architecture in retrieval tasks.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential negative societal impact of this work is discussed in the supplementary materials.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new re-ranking method for image retrieval. It proposes to learn an affinity vector for each of the top-ranked candidates w.r.t a set of anchor images, using a transformer encoder. These affinity vectors are then used to re-estimate the similarity scores between the top-ranked candidates and the query image. The reviewers found the work to be technically sound, have novelty, and produce good results. There were some concerns about the fairness of the evaluations, and many other questions of detail, which the authors have addressed effectively in their rebuttal. 
Decision: Accept (Poster)