Deep Fusion of Multi-attentive Local and Global Features with Higher Efficiency for Image Retrieval
Authors: Baorong Shi
Venue: ICLR 2022 Submitted (2022)
Keywords: Image retrieval
Homography learning
Attention
Intermediate supervision

Abstract: Image retrieval is to search images similar to the given query image by extracting features. Previously, methods that firstly search by global features then re-rank images using local feature matching were proposed, which has an excellent performance on many datasets. However, their drawbacks are also obvious. For example, the local feature matching consumes time and space greatly, the re-ranking process weakens the influence of global features, and the local feature learning is not accurate enough and semantic enough because of the trivial design. In this work, we proposed a Unifying Global and Attention-based Local Features Retrieval method (referred to as UGALR), which is an end-to-end and single-stage pipeline. Particularly, UGALR benefits from two aspects: 1) it accelerates extraction speed and reduces memory consumption by removing the re-ranking process and learning local feature matching with convolutional neural networks instead of RANSAC algorithm; 2) it learns more accurate and semantic local information through combining spatial and channel attention with the aid of intermediate supervision. Experiments on Revisited Oxford and Paris datasets validate the effectiveness of our approach, and we achieved state-of-the-art performance compared to other popular methods. The codes will be available soon.

Under review as a conference paper at ICLR 2022
DEEP FUSION OF MULTI-ATTENTIVE LOCAL AND GLOBAL FEATURES WITH HIGHER EFFICIENCY FOR IMAGE RETRIEVAL
Anonymous authors Paper under double-blind review
ABSTRACT
Image retrieval is search images similar the given query image by extracting features. Previously, methods that firstly search by global features then re-rank images using local feature matching were proposed, which has an excellent per- formance many datasets. However, their drawbacks are also obvious. For ex- ample, the local feature matching consumes time and space greatly, the re-ranking process weakens the influence of global features, and the local feature learning not accurate enough and semantic enough because of the trivial design. In this work, proposed Unifying Global and Attention- based Local Features Re- trieval method (referred to as UGALR), which is an end-to-end and single-stage pipeline. Particularly, UGALR benefits from aspects: tion speed and reduces memory consumption by removing the re-ranking process and learning local feature matching with convolutional neural networks instead of RANSAC algorithm; 2) it learns more accurate and semantic local information through combining spatial and channel attention with the aid of intermediate su- pervision. Experiments on Revisited Oxford and Paris datasets validate the effec- tiveness of our approach, and we achieved state-of-the-ar performance compared to other popular methods. The codes will be available soon.
accelerates extrac-
INTRODUCTION
Image retrieval is classic task computer vision. Its primary purpose to search images that are similar to the given image through extracting features. Learning image representations well is the key factor to superior performance for image retrieval.
Global and local features are two typical image representations in image retrieval. According to (Cao et al., 2020), global features summarize image content and could learn similarity across very different poses, but lack spatial arrangement information. Local features contain geometric infor- mation and descriptors about specific regions, but lack global understanding of image content. For better performance, popular approaches first search with global features, then re-rank with local fea- ture matching. This kind method could achieve -fate-of-thh--at results according to (Cao et al.) ( al. (018)(Sari)) et al. 2019). Nevertheless, they have substantial drawbacks when put into practical use. First, the processes of extracting local features, local feature matching, and re-ranking are of low efficiency either in memory or latency. For example, extracted local fea- tures often occupy too much memory. The RANSAC algorithm used in local feature matching is extremely time-consuming and also unstable since it depends heavily on iteration times. What's more, it needs to rank twice to get the final retrieval results and re-ranking with local features would erase the influence of global features in top results; Second, the local attention module is too trivial to capture local information good enough in both accuracy and semantics. Besides, previous works mostly treat attention and features equally (Ng et al.) 2020) (Woo et al., 2018) et al. 2018), but rigit balance factor between attention and features would limit attention learning.
A question arises: facing the problem that retrieval results are correct in the category but lack at- tention to detail, is there any other way to solve it without the downsides mentioned above? Our work gives an appropriate answer. With a different view, we could interpret this problem as that the learned global features are powerful enough capture local information, while local fea-

---

Under review as a conference paper at ICLR 2022
tures perform better in distinguish details. Since local feature matching is to find the appropriate homography matrix between images, which is a process of learning linear transformation, we think it could be learned by the convolutional neural network. Based on these ideas, intend to learn a more powerful global feature that contains merits of local features with the aid of multi-attentive local attention, CNN-based homography transformation, and information fusion.
First, we focus on local feature attention. We work on enhancing it in both module design and training strategy. In terms of module design, we proposed LALM (Location Attention Learning Module) block that contains both spatial attention mechanism and channel attention mechanism to enhance semanties and accuracy. terms of training strategy, we introduced intermediate super- vision into the process of training attention modules with the aim of giving different importance to attention and features.
Second, we focus on learning CNN-based homography transformation and information fusion. We concatenate the output of LALM with the global feature branch, let subsequent blocks learn homog- raphy transformation in local feature matching, and naturally fuse it with global features. Instead of extracting global and local features respectively from the trained model, we only use the enhanced features that contain both global and local information to perform image retrieval, which reduces time and space costs caused by local features storing, matching, and re-ranking.
In general, our main contributions are summarized as follows:
We innovatively utilised CNN network learn the homography transformation in local fea- ture matching and generate a more powerful feature that contains both global information and local information.
We proposed LALM(Local Attention Learning Module) block which learns local atten- tion in both spatial and channel dimension. We introduced intermediate supervision into image retrieval to help the network treat attention and feature differently and adaptively, which effectively avoids attention degradation.
We proposed single-stage model named UGALR(Unifyy Global and Attention based Lo- cal Features Retrieval methods) that can be trained end-to-end. It achieved state-of the-ahrr art performance on two typical datasets with less memory and faster extraction speed com- pared with other popular methods.
2 RELATED WORKS
2.1 IMAGE DESCRIPTORS
Image descriptors refers to feature vectors that contain image content information, including local descriptors and global descriptors. Local descriptors generally contain both key points location and corresponding features, and some popular methods include (Noh et al., (017)) (Tolias et al., 201 6a), (Gong al. 2014) and (Arandjelovic et al., 2016). The global descriptor in the form of a compact vector, which is usually generated by a single single forward-pass of CNN. And some typical ap- proaches include(Ng et al. 2020), (Radenovic et al. 2019). 2019).(Deng et al., 20T9), (Shhn 2016). Local features tend to focus local information but have poor performance features summarize the content of an image but lose geometry information in it. Consequently, more and more cases began combine the kinds of features, such as (Sarlin al., 2019), (Cao al.) 2020). (Taira et al. 2018), and (Siméoni et al., 2019). However, this kind of approach also suffers from great time and space consumption. In this work, we propose new idea that learns homogaphy information with CNN to avoid huge time and space consumption.
the global, while global
2.2 HOMOGRAPHY ESTIMATION
Homography estimation is to find transformation matrix between images, which are taken from different perspectives of the same object. In traditional approaches, solving homography often needs to detect and match features, then using estimation algorithm, such as (Fischler & Bolles 1981).
(Chen al. (017), and (Barath et al.) 2019), to establish correspondences between images. But these approaches suffer from great time and space consumption. With the development of CNN,
more deep homography approaches were proposed, which include supervised (DeToone et al. 2016)
2

---

Under review as a conference paper at ICLR 2022
General Supervision
spatial attention
channel attention
Gem/ Whitening
Lg
CNN Blocks
Descriptor
Image
CNN Blocks
Local Attention Learning Module
Aug/ Whitening
La
Intermediate Supervision
Figure 1: Illustration of UGALR. Firstly, the images are fed into CNN blocks extract features. Then the output features are fed into the Local Attention Learning Module, which is denoted with the orange dotted box, to learn attention- based local features. There are two small branches in LALM. One branch is to learn attention score for feature maps, the other branch is the original feature maps without any operation. The LALM output is the multiplication result them, which is attention- weighted feature maps and contains local information. The weighted feature maps, which then go through avg-pooling and whitening operation, are intermediately supervised by La. On the other hand, the learned weighted feature maps goes through avg wrg-pooling in the channel dimension and then concatenated with originally extracted features to feed into the subsequent CNN blocks, Gem- pooling, and whitening FC-layers, which is supervised by Lg. "X". "C", "A", and "M" respectively denote element-wiss multiplication, concatenation, activation functions and average pooling in the channel dimension.
and unsupervised approaches (Nguyen et al. 2018). However, these methods are suitable for image retrieval.
2.3 INTERMEDIATE SUPERVISION
Intermediate Supervision is training technique that provides supervision in the intermediate part of the model. This technique is previously used in landmark detection (Newell et al.] (016)) (Weil al. (016) (Carreira et al. 2016) (Prister et al. 2015) because of the ability of emphasizing critical features and easing gradient vanishing. Nevertheless, to the best of our knowledge, there is no case that combines intermediate supervision with image retrieval yet.
3 METHODOLOGY
In this section, we first present our design motivation, then introduce model construction, followed by the retrieval process.
3.1 MOTIVATION
As illustrated in the introduction, to learn better local attention and address the time consuming, memory consuming and low efficiency problems of using local features, we intend integrate multi-attentive local information with the learning of global features and let CNN learn homography transformation in local feature matching. Our idea to learn more powerful global features that could distinguish details in images to get the same or superior retrieval results with faster extraction speed and less memory occupation compared with the global and local re-raking methods.
The implementation and theories of UGALR will be described in detail in the next section.
3.2 MODEL
Our proposed UGLAR consists of backbone network and local attention module named LALM(Local Attention Learning module), as shown in Figure 1. We first introduce the LALM block, then illustrate the backbone network from global perspective.
3

---

Under review as a conference paper at ICLR 2022
Z
softmax
flatten
flatten
flatten
#: 1x1
4: 1x1
g: 1x1
Figure 2: Illustration of spatial attention module. Here we take the relationship learning part of non-local Wang et al.) 2018 aas the spatial attention module. X is the input of the module and is the output. g, 0, Φ and denote Ixl convolution kernels.
Local Attention Learning Module As discussed above, LALM is proposed learn multi-attentive local attention and features. In this block, the input tensor firstly goes through a convolution kernel followed by ReLU to reduce dimensions, then goes through spatial attention module, which is modified non-local block block(Wang et al. 2018), then goes through channel attention module, which convolution kernel, and then activated by Softplus(Duger al. 2000).
Since the non-local block has the ability of learning relationships among features in the spatial dimension of 3d feature map tensor, we could interpret
kind of spatial attention. Considering the function of spatial weighting and the requirement of low latency. we modified the non-local block and delete the residual connection in it, as shown in Figure 2. Formally, let denote the input of the spatial attention module, g, 0, Φ, and w denote 1x convolution kernels. The whole calculation process could be expressed as the formulas below.
0, = flatten 6. = flatten(ó(X)) = flatten(g(X))
(1)
X softmar(fl, o!))
(2)
Y view(X g.)
(3)
Z wY)
(4)
where x represents matrix product, 0(-), ((-). 9(-), wx-) are convolutional calculation, and flatten, sof coftmax, view are their corresponding operations.
The channel attention module contains 1x1 convolution kernel. Since and the spatial attention module's output have the same number of channels, after training, each of these parameters in the module corresponds to weight of one channel in feature maps. Thus we could interpret it as a process learning channel attention. The reason why we do not use other types of channel attention, such SE block et al. 2020), is to reduce extraction latency.
To learn the most optimal attention for feature maps, we not only need to know which point in the spatial dimension is critical but also need to figure out which characteristics of the feature map significantly make sense. Thus, the combination of modified non-local blocks and 1x1 convolution kernel here plays important role in attention learning. Softplus function maps the output of attention modules into a positive feature map-G. And it represents the attention score. We then element-wise multiply G and X get D, which is weighted feature map that contains information of key local attention.
Intermediate supervision is added to the LALM block. As shown in Figure 1, D is then followed by avg-pooing and whitening FC-layer, which is supervised with cross-entropy loss. The cross-entropy loss can be expressed as:
W +b
La
log
(5)
Σ e
tbtb;
4

---

Under review as a conference paper at ICLR 2022
where x; E IR represents the feature of the i-th image with label ys- d is the dimension of embedding features. N denotes the batch size and n is the number of classes. by E IR" denotes the bias term and W E IRd is the j-th column of the weight W E _d
Intermediate supervision is indispensable since it differentiates the importance of attention and fea- tures. Otherwise, rigid factor between attention and features would limit attention learning, which leads to inaccuracy and fuzzy semantics caused by attention degradation. With intermediate super- vision, the network could learn attention and features differently and adaptively based on specific tasks. Furthermore, the addition of it alleviates vanishing gradient and model degradation caused by "dying ReLU", making the learned attention more accurate.
In addition, D goes through average pooling in the channel dimension produce O, and O is sub- sequently concatenated with the middle block of backbone networks and the result of concatenation put into the next block. Taking in a C x H W-shaped feature map, average pooling in the channel dimension outputs a feature map with the shape of x H x W, which is of great signif- icance. As illustrated above, the backbone network leads the training process intending learn more powerful features, while LALLM plays an auxiliary role to learn more accurate and semantical local information. If we crudely concatenate the output of L.ALM with the global branch, the back- bone network's 's dominant position would be weakened, since over-contributton of LALM leads to intermediate supervision interfering with global features training, which results in suboptimal per- formance. Average pooling in channel dimension not only reduces the dimension of feature maps to avoid this problem but also preserves key points information.
Backbone Network Given input image, we apply convolutional neural network as backbone and take the block i output X; shaped of C, x H; x W, C., H₁, W; here correspond to the number of channels, height, and width respectively. With X; being the input, we plugged LALM into this block. Then we would get a weighted and dimension-reducedd feature map named O2 that contains local information. The concatenation of O₂ and X; is then fed into the next block of backbone.
The innovative design of concatenation and feeding into next blocks has two merits. Firstly, the addition of Og provides extra attention to boost global features learning. Secondly, the subsequent blocks could utilise the key attention information to learn homography transformation in local fea- ture matching. We assume that there an identical average target available for each category of objects. Then, from each image in one category to its corresponding target image exists homogra- phy matrix. Our purpose is to let CNN learn the transformation and fuse with global features.
In general, there is a pooling operation at the end of the backbone to integrate important features in feature maps. Here we choose generalized mean pooling (oolinggGEEM)RRRRR GaM)nRadeovvie et al.) 2019), because Tollia et al., 2016b) and avg-pooling (aabenko & Lem- pitsky 2013) and its effectiveness has been widely proved. After GEM pooling, we use fully connected layer to whiten the aggregated features, as illustrated in (Gordo al. 2017b).
it combines the advantages of max-pooling poooliiggo(ooia
In terms of general supervision loss, we need to learn global features with strong discrimination ability, which are compact within classes and distant between classes, we adopt the ArcFace loss(Deng et al.] 2019) here. As an additive angular margin loss, outperforms most losses in metrie learning under the same settings, illustrated in (Srivastava al. 2019). And the Areface loss can be expressed as:
(0ocss00 +m))
Lg = Σ log ((con(8,,))- )-m) + E)=,,t
(6)
where N is the batch size, n is the number classes. 0, denotes the angle between the weight W and the feature XG, where T, E Idd is the L2-normalized feature of the i-th image with ||æ,|| = 1, W is the j-th column of the L2-normalized weight W E kdxx with |[Will = 1. d is the dimension of embedding features. yi represents the label of 2;. m is an additive angular margin penalty between I; and Wy, y. and s is a re-scale factor.
The total loss is the weighted sum of intermediate supervision loss and general supervision loss, which can be expressed as:
L. T.g + Xin where X is weight factor between T.g and Tun-
5

---

Under review as a conference paper at ICLR 2022
3.3 RETRIEVAL
There are two steps in the retrieval stage; extracting image features and measuring similarity be- Tween features. Previous works(Gordo et al., 2N17a)(Noh et al. 2017) (Radenovic et al. (Cao et al., 2020) have shown us that multi-scale information is of great significance in feature extraction. In order introduce multi-scale information, we use scales, [2 image and we could obtain five global features of different scales for one image. These features are then 1.2-normalized, summed up and 1-2-nommaiizdd again to get the final embedding F. Besides, use dot product of two features to measure similarity. Since global features are L2-normalized, the dot product denotes cosine distance between two features.
resize the original
4 EXPERIMENTS
4.1 DATASETS
Training datasets this work, the clean version of Google Landmark V2 & Yokoo 2019). which contains 1580470 images and 81313 classes, serves training dataset. We split 80% of images into the training set and 20% of images into the eval set.
Test datasets Revisited Oxford and Paris(Radenovi et al. 2018) serve test datasets, which are denoted as ROxf5k and RPa66k. ROx15k contains 4993 database images and RParók contains 6322 database images. They respectively have 70 query images. We also use RIM distractor set with 1 M images to evaluate model performance. By convention, the mean average precision(mAP) reported performance metrics.
4.2 SETTINGS
In this section, we will introduce model setup, training details and extraction settings in turn.
Model setup In this work, ResNet-50 serves as the backbone. We denote the block3 output as X, which is shaped of 1024 x H x W. LALM is fed with X and outputs feature map O shaped of x H x W. The input of block4 of the backbone network is shaped of 1025 H x W. The parameter p of GeM pooling is set as 3. The whitening FC-layer reduces the dimension of features from 2048 to 512. In Arcface loss, we set the margin m=0.15 and set the scale=30. In terms of the total loss L, the X is set 0.6.
Training details Our model is implemented using PyTorch. Data augmentation methods include random flip, random crop, and resizing images into 512 x 512. The weights of models are initialized from pre-trained ImageNet. We used batch size of 128 and applied some learning rate adjustment strategies: cosine decay and warm up. The initial learning rate 0.05. After 100 epochs using the SGD optimizer, we take the best performing model and report its results on different datasets.
Extraction settings With the extraction and retrieval process introduced in section 3.3, we illustrate some parameter settings here. The multiple scales we use to resize images include 0.3535, 0.5, 0.7071, 1.0, and 1.4142. The final extracted features F are shaped of 1 512. is directly used to perform retrieval and no more re-ranking process is needed.
4.3 COMPARSION WITH STATE-OF-ART METHODS
In this part, we compare our approach with other state-of-the-art techniques in two dimensions. One accuracy, the other is memory and extraction speed. Accuracy T. shows the comparison between our methods and the retrieval state-of-the-art in accuracy. The table includes three parts. The first part includes local features aggregation- based methods. The second part includes global naatures/globllllce features-based methods. Notice that those marked as ("global+local") denote approaches that firstly search with global features then re-rank with local feature matching, and those marked as "(Re)" are implemented under the same settings as ours. And the last part is our approach. As we can see, UGALR completely outperforms previous global features based methods, for Roxford5k, Rparisók and their R.M distractor sets. The largest two improvements we made are 7.67% in ROxf-H, and 9.31% in R.ar-H. In addition, our approach even surpasses the ofats-of-art global and local features-based method, such as DELG(Cas al. 2020), most

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison to state-of-the-art retrieval methods We present MAP(mean average pre- cision) of different approaches on ROxf5k/RParók and 1 datasets in both Medium and Hard protocols.
Medium
Hard
Methods
ROxf
+1M
RPar
+1M
ROxf
+IM
RPar
+ +1M
()) Local features
65.60
65.20
41.10
38.50
DELF-ASMK**
67.80
53.80
76.90
57.30
43.10
31.20
55.40
26.40
(II) Gocal features
Global+Local features re-ranking
AlexNet-GeM
43.30
24.20
58.00
29.90
17.70
9.40
29.70
8.40
VGG16-Gebl
61.90
42.60
69.30
45,40
33.70
19.00
44.30
19.10
RI01-R-MAC
60.90
39.30
78.90
54.80
32.40
12.50
59.40
28.00
R101-GeM
64.70
45.20
77.20
52.30
38.50
19.90
56.30
24.70
R101-SOLAR
69.90
53.50
81.60
59.20
47.90
29.90
64.50
33.40
R50-DELG
73.60
60.60
85.70
68.60
51.00
32.70
71.50
44.40
R50-DELG(Re)
76.07
61.51
85.48
68.42
49.91
33.10
70.60
43,93
350-DELGGgglobllHooca
78.30
85.70
57.90
71.00
S.0-DELLGiggobbll-loaa/k/k
79.02
88.90
58.00
76.18
(III) Ours
R50-UGALR
79.98
66.68
90.85
70.48
57.58
36.81
79.91
46.95
protocols. For example, on rparisók dataset: 1.95% improvement was made RPPa-Medium and 3.73% improvement was made in R.Par-Hard. On roxford dataset, we made an improvement of 0.96% in ROxf-Medium. Surprisingly. our approach using only global features outperforms the state-of-art global fcature+local feature re-ranking methods.
Overall, our approach achieved state-of-the-art performance in accuracy compared with the other advanced methods. These results validate the effectiveness of our ideas: combining channel mecha- nism, spatial mechanism, and based homography transformation greatly enhanced the perfor- mance of global features, which have more ability to distinguish details.
Table 2: Experiments for extraction speed and memory of different approaches. We did this experiment on the same NVIDIA Tesla P40 GPU, with images resized into 1024. UGALR" refers version of U UGALR with the same extraction settings with Group 3.
Extraction latency(ms)
Memory
Methods
ROxf+IM
RPar+1M
(1) Local features
DELF-R-ASMK*
2260
27.6
DELF(3scales)
98
434.2
434.8
DELF(( scales)
201
477.9
478.5
(2) Global+Locca features re-ranking
R50-DELGG
211
485.5
486.2
R50-DELGG scales global+local)
118
439.4
440.0
R101-DELG
383
485,9
486.6
R101-DELG(3 scales global+local)
193
437.1
437.8
(3) Global features
R50-GeM
100
7.7
7.7
R101-GeM
175
7.7
7.7
(4) Ours
R50-UGALR*
99
7.8
7.8
R50-UGALR
68.7
3.9
3,9
R101-UGALR*
177
7.8
R101-UGALR
132
3.9
3.9
Speed and Memory
Besides accuracy, speed and memory are also very important indicators to measure the model's performance. Tab.2 compares the extraction latency and feature memory of UGALR with the state-

---

Under review as a conference paper at ICLR 2022
of of-the-art methods. In extraction latency, R50-UGALR runs with 68.69(ms) and RI01-UGALR runs with (32(ms). memory, features from R50-UGALR and RI01-UGALR both occupy 3.9M, which are superior to other approaches. Methods marked with "*" used the same extraction settings as Group 3, which didn't reduce dimensions. As shown in Tab.2, when not reducing dimensions, UGALR outperforms local features based and unifying global and local features based methods in both extraction speed and memory, which performs similarly with global features based approaches. Dimensionality reduction through the FC-whitening layer further boosts its performance in its mem- ory occupation and extraction latency, which are more suitable for practical use.
4.4 ABLATION STUDY
In this section, we performed ablation study on different innovations of UGALR.
Intermediate supervision Tab.3. compares the effects of using and not using intermediate super- vision. The first row presents the result of not using intermediate supervision and the second row presents the other. Experiments show that adding supervision in the intermediate part of UGALR brings an improvement of 1.52% in ROxf-M, 3.57% ROxf-H, 1. 1.75% in R.Par-M and 3.22% in RP which is in line with our expectations, since intermediate supervision differentiates the importance of attention and features to avoid the degradation of attention learning.
Table 3: Experiments for Intermediate Supervision.
roxford5k
rparisók M
Intermediate supervision
M
H
H
no
78.46
54.01
89.1
76.69
yes
79.98
57.58
90.85
79.91
Local Attention Learning Module Tab.4. reports the effects of LALM and homography transfor- mation. The first row denotes the performance not using the LALMM block and the second row denotes the performance of using LALM. The model with LALM outperforms the model without it by 1.77% in ROxf-M, 1.27% in ROxf-H, 1.85% in RPar-M and 3.74% in RPar-H, which verified the effectiveness of the Local Attention Learning Module. As more accurate local attention and its concatenation allows the global branch to learn homography transformation and further boosts global features learning.
see, the module helps learn
Table 4: Experiments for Local Attention Learning Module.
roxford5k M
rparisók
LALM
no
78.21
56.31
89
76.17
yes
79.98
57.58
90.85
79.91
Table 5: Experiments for exploring different kinds of attentions
roxford5k M
rparis6k M
Attenion setup
H
H
no
78.53
54.31
86.99
72.29
spatial
78.6
55
89.96
78.48
channel
78.33
55.93
88.88
76.72
spatial+channel
79.98
57.58
90.85
79.91
Local attention setup Tab.5. compares the performance of no attention and three different kinds of attention in LALM. For example, the first row represents not using any attention, and the second row means using only spatial attention. Experiments show that combining spatial and channel attention mechanism outperforms other three approaches by a large margin for all four protocols. The spatial attention module helps identify the key points and the channel attention module helps identify the critical characteristics. The joint learning of them is essential to the LALMM block.
8

---

Under review as a conference paper at ICLR 2022
Table 6: Experiments for average pooling in channel dimension
roxford5k
rparisók
Avg pooling
M
H
M
H
no
76.31
50.89
87.79
74.16
yes
79.98
57.58
90.85
79.91
The most suitable block for LALM applying To explore which part of the network we should add the LALM block to, we also conducted a comparative experiment as shown in Tab.7. This experiment is based on the ResNet-50 backbone. From the table, the setup of adding LALM to the block3 has MAP of 79.98% ROxf-M, which surpasses the other three setups. Since the shallower feature maps of the network are less semantical for attention learning, plugging LALM into block2 has inferior performance. The reason why the other two setups are also suboptimal is that the deeper feature map from the block4 output is relatively small size and it is less localizable learn great local attention.
ROxf-M, 57.58% in ROxf-H, 90.85% in RPar-M and 79.91 %
Table Experiments for most suitable blocks for LALM applying
roxford5k M
rparisók M
block
H
H
2
77.4
52.17
88.55
75.81
3
79.98
57.58
90.85
79.91
4
78.1
55.43
90.32
79
3+4
78.8
54.52
89.16
76.69
Average pooling in channel dimension The significance of average pooling the channel dimen- sion is illustrated in Section 3.2, and we conduct an experiment to verify its necessity. As shown in 6,b.6, the model with Avg-pooling in the channel dimension outperforms the one without by 3.67% in ROxf-M, 6.69% in ROxf-H, 3.06% in RPar-M and 5.75% ROxf-M. By reducing di- mension an appropriate way, it effectively avoids intermediate supervision's 's over interference to global feature learning and well preserves key information at the same time.
Feature fusion To find out the best way to fuse LALM output and the block output network backbones, also carried out experiment. As shown in Tab.8, the first row represents using ad- dition to fuse features, the second row means using multiplication to fuse features, and the third row means performing concatenation. It is noticed that performance of the third group surpasses the first two groups by a large margin. This expected. Compared with simple addition and multiplication, concatenation avoids information loss and preserves complete local attention information learn homography transformation in images.
5 CONCLUSIONS
We proposed a single-stage image retrieval pipeline named UGALR. utilizes spatial and channel attention to learn key local information, then feeds it to CNN to learn homography transformation in images. Through information fusion, UGALR produces multi-attentive features that contain both global and local information. In this work, learn homography transformation with CNN then replace the re-ranking process with information fusion to obtain more powerful features and over- come the low efficiency of local features in storage and matching. We also combine spatial and channel attention to improve accuracy and semantics in local attention, apply intermediate supervi- sion to avoid the degradation of attention learning. The effectiveness of UGLAR has been validated with comprehensive experiments, which achieves state-of-the performance with lower memory occupation and faster extraction speed.
Table 8: Experiments for feature fusion
roxford5k M
rparisók M
fusion approach
H
H
add
74.41
48.87
87.82
74.21
multiply
77.87
54.07
89.53
77.4
concatenate
79.98
57.58
90.85
79.91

---

Under review as a conference paper at ICLR 2022
REFERENCES
Relja Arandjelovic, Petr Gronát, Akihiko Torii, Tomás Pajdla, and Josef Sivic. Netvlad: CNN architecture for weakly supervised place recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 5297- 5307. IEEE Computer Society, 2016. doi: N0.100//VPR.22066.772 URL https:
BTg/10.1109/CVVE 0tTp9//CPPR.226..5777 2016.572
Artem Babenko and Victor S. Lempitsky. Aggregating deep convolutional features for image trieval. CoRR, abs/1510.07493, 2015.
Daniel Barath, Jiri Matas, and Jana Noskova. MAGSAC: marginalizing sample consensus. In
IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 10197-10205, Computer Vision Foundation / IEEE, 2019. URL
doi: D0.110//CVPR.201..1100 CVPR 2019 2019/html/Barate MAGSAC Marginalizinnnnnnnaa Sample Consensus. 2PPR2019920 10019__aperrhhpre
thecvf. com content
Bingyi Cao, André Araujo, and Jack Sim. Unifying deep local and global features for image search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Com- puter Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Parl XX, volume 12365 of Lecture Notes Computer Science, pp. 726-743. Springer, 2020. doi: 9988 030-000---0 58565-5 43
URL/actas://ali http:///oo..rrr//..... otg/10.1007.
João Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In 2016 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 9733-4742, IEFE Computer htt.https://// www.oi.o.rrr...... http://////o...//000..0000000090 1109//VPR..
Society, 2016. doi: 10. 10111V. 2016.512
Kai Chen, Qi Lv, Yao Lu, and Yong Dou. Robust regularized extreme learning machine for gression using iteratively reweighted least squares. Nenrocompputing, 230:345-358, 2017. doi: 2016.
12.029. URL / 1ttp://..1111//111
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 4690-466 Computer Vision
Foundation / IREE, 2019. doi: 10. 10.1109/CVPR..2099000444 URL http::/openaccess. Aww.Aaccceaaaatiiiiveeaaeaaaaaaa Additive Anguule.Araarrrr. html
thecvf. , wttp://cccmaoooooo
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Deep image homography estimation. 03798
CoRR, 606.07996 2016. URL / http:///w.............aaa/////r/
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporat- ing second-order functional knowledge for better option pricing. In Todd K. Leen, Thomas G. Dietterich, and Volker Tresp (eds.), Advances in Neural Information Processing Systems 13, Pa- pers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA, pp. 472-478.
MIT Press, 2000. URL http::tttt:::pprpeeeiiing neurips httplps.ccpp/pe/ 44
Abst html
Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 2416) 11e./111535990 1981. doi: 1145/358669.358622 URL hwt.hhtpp/// //ooi. acm. wwt..oog/1..111///5555511 358692
Yunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik. Multi-scale orderless pooling of deep convolutional activation features. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII, volume 8695 of Lecture Notes in
10

---

Under review as a conference paper at ICLR 2022
Computer Science, pp. 392 407. Springer. 2014. doi: 01000///977883311-1-------8 URL
https://doi. http::///oi..og//1/0.. 1007 -0111999997-11111-------0-0000--
Albert Gordo, Jon Almazán, Jérôme Revaud, and Diane Larlus. End-to-end learning of deep visual
representations for image retrieval. Int. J. Comput. Vis., 124(2)237-2254, 2017a. doi: 10.1007/
1263-0077-01
Albert Gordo, Jon Almazan, Jerome Revaud, and Diane Larlus. End-to-end learning of deep visual representations for image retrieval. International Journal of Computer Vision, 124(2)23772244, 2017b.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-exciaation networks. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 7132-7141. IEEE Computer Society, 2018. doi: 101109/VPR.2011 00745. URL Eqruere-and-Exiiittiio wttpiaatiionneeeeeeeeeeeeeee CVPR 2018 paper.pp..... .tml
thecvf. com/ http//otttettttcceeeeee coportent_ccr_____ hatpeeentt__pr_______11111 2018/html /htmm//tt.H comp:ooteettcccccppcccc
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitaten networks. IEEE Trans. Pattern Anal. Mach. Intell., 2218):2011-2022 2020. doi: 10. 2011 2913372. htthttps:
org/ 1 0 1109 1w11111199MIIIIMM 201..
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose es- timation. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vi- sion ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII, volume 9912 of Lecture Notes in Computer Science, pp. 483 499. Springer, 2016. doi: 1111100/97783-3194400 (,, URL https::/ org 10. 1007/ 978-3-313 018---1--44---9
Tony Ng, Vassileios Balntas, Yurun Tian, and Krystian Mikolajczyk. SOLAR: second-order loss and attention for image retrieval. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm (eds.). Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV, volume 12370 of Lecture Notes in Computer
Science, pp. 253-270. Springer, 2020. doi: 10.
URL https:
//doi. 10. 1 0070978-0-88 --00008---0---00- 58390-5555-2211
Ty Nguyen, Steven W. Chen, Shreyas S. Shivakumar, Camillo Jose Taylor, and Vijay Kumar. Unsupervised deep homography: fast and robust homography estimation model. IEEE
Robotics Autom. Lett., 3(3):2346 2353, 2018. doi: 10. R0.10091AA220182880995 URL https:
7/doi. (idbi.orgg10.om 1109/ LRA. 2018 201.222021888888..83 2809549
Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 3476-3485. IEEE Computer Society, 2017.
doi: 10.1 10.100/CCC..2017.33... http:///////////////////////////
Kohei Ozaki and Shuhei Yokoo. Large-scale landmark retrrievallrecooniitoon under noisy and di- 04087
verse dataset. CoRR, abs//906.04087,, 2019. UR..http::/arxxiv.orr///
Tomas Pfister, James Charles, and Andrew Zisserman. Flowing convnets for human pose estimation in videos. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1913-1921. IEEE Computer Society, 2015. doi: 10.1109/ICCV. 2015.222. URL hhtph:ttpss///e / http:::///oooo.1g/.//// http:///1.111//////// http//...c.............2 .22
Filip Radenovi, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondej Chum. Revisiting ox- ford and paris: Large-scale image retrieval benchmarking. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018
Filip Radenovic, Giorgos Tolias, and Ondrej Chum. Fine-tuning CNN innaeeimgee.oo imagivel with no human annotation. CoRR, abs/1711.0255 2017. URL
02510
Filip Radenovic, Giorgos Tolias, and Ondrej Chum. Fine-tuning CNN image retrieval with no human annotation. IEEE Trans. Pattern Anal. Mach. Intell., 41(7)1655-1663 2019. doi: 10. 109/
https::::: //::://doii.
TTT.:/AM......8555622 2018.2846566
11

---

Under review as a conference paper at ICLR 2022
Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to
fine: Robust hierarchical localization at large scale. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA. June 16-20. 2019, pp. 12716
12725. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR 2019.01 URL.
http: httt:://ppeaacees:: www.theeecccoooo www.//////oooo///oocoo CVPR 20192/0119//// Sarlin, From_ Coarse to Fine Robust Hierarchicaa Local CVPR 2019 paper.html
at Large_Scale.
Oriane Siméoni, Yannis Avrithis, and Ondrej Chum. Local features and visual words emerge in activations. In IEEE Conference on Computer Vixion and Pattern Recognition, CVPR 20/9,
Long Beach, CA, USA, June 16-20, 2019, pp. 11651-11660. Computer Vision Foundation / IEEE, 2019, doi: 10. 1001/19/C 201..019991122 URL http::/ppenaceess theccf com/ content CVPR http9/hhhmmmimmmmm Emerge in Activations CVPR 2019 1ar_ppapper... html
Local Features and Visual Words
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gar- nett (eds.), Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp- 1849 1857, 2016. URL https: htt.::::pppoceeeinggg neurips wccppperr/216/haa.k 6b1 80037abbebea 991 d8b 232 232 faw.aaa8aaaaa. Abst ract. html
Yash Srivastava, Vaishnav Murali, and Shiv Ram Dubey. A performance comparison of loss func- tions for deep face recognition. CoRR, abs/1901.05993, 2019. URL /att:://axxi..oo
abs/19t 05903
Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomás Pajdla, and Akihiko Torii. Inloc: Indoor visual localization with dense matching and view synthesis. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 7199-7209 IEEE Computer Society, 2018. doi: 10.1 N0.1109/CVPR220 8 00752. URL http: /
/pwpaacccs.ccccpcppppppp/c/..... thecvf. com content. InLoc_ T Indoor Vi sua _CVPR_C___________ 2018 Bap188_pp_______ paaper.hmm
cvpr
Giorgos Tolias, Yannis Avrithis, and Hervé Jégou. Erratum to: Image search with selective match kernels: Aggregation across single and multiple images. Int. J. Comput. Vis., 116 (3):262, 201 doi: 10.10077/11233-005-000 s s11263- 015 015-037---33-66 0837-6
URL https:////// /tt.///o.../// wwt.oc.....oooooooe//eeo1 2trg/10..107,
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max- max-poooiig of CNN activations. In Yoshua Bengio and Yann LeCun (eds.). 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track
Proceedings, 2016b. http:t www.wi.ogg/aaaaaa... /1511 05879
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 7794-7803, IEEE Computer Society, 2018. doi: 10.1
CYPR.2018.00883. URL http: /ttp:///pppaaae http///ppaacccsseeeeeeeoe wtw.cccsssssste.ecoeeoo com http://cccttttttceeeeeeeeeee _tpprttttt__tt____eeeeee _vpp__01er////__0 Non-
html /Wwaaaaaanooooooooooe
Neura _tt__eeettts____ Cc______________ 2ww..o201spppprrhhttm.mm
Sh Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 4724 4732. IEEE Computer Society, 2016. doi: 10. (0.1100) 109/CPR..
2016.511. URL http::ttt///tt...t.. /tt::///oiii// org 10. 1 1109/CVPR. 2016.55 511
Sanghyun Woo, Jongchan Park, Joon-Young Young Lee, and In So Kweon. CBAM: convolutional block attention module. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, volume 11211 of Lecture Notes in Computer Science, pp. 19 URL https: //oi. 10. /ttg//11.100...
Springer, 2018 doi: 10. 00.10079978-3-000-0111 978 3 030 01234-2
12


Reviews:
[{"review_id": "tY9Buwcy00G", "reviewer": "Reviewer_MPBM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper proposes an interesting approach that obtains very good results using only global features, outperforming two-stage methods that use local features for re-ranking.\n- Experimental evaluation is also complete and thorough, with a very extensive ablative study.\n\n-----\nWeaknesses:\n- The main weakness of the paper it has to do with its presentation: it has numerous syntax and grammatical errors and typos, which makes for a challenging read sometimes. Some paragraphs are difficult to follow and in my opinion should be rewritten. The main problem with this is that it affects the motivation of the paper: I feel I couldn't fully grasp the rationale behind some design choices or the conclusions drawn from some of the experiments. It is difficult to provide a complete list of all the things that should be addressed by the authors but I'll discuss in section \"Presentation\" below what I think are the most importante ones. Nevertheless, I highly encourage the authors to do another pass on the writing.\n- Slightly related to the point above, I'm missing a deeper and more detailed explanation about the claim that the last convolutional blocks act as an homography transformation because there's no experimental or theoretical study that proves that in the text. Overall, the contributions of the paper are not clear enough.\n- Not too important: I understand that the focus of the paper is removing the re-ranking step, but it would have been interesting to see if the local features produced by the LALM module could have been used for this purpose to further improve the results in applications that can afford the extra memory and latency costs.\n- I wondered why the authors decided to use average-pooling and cross-entropy loss for the intermediate supervision instead of GeM or ArcFace as for the global features. Any insight about why this shouldn't work better?\n- Apologies if I missed this in the text, but how are R50-UGALR* and R50-UGALR in Table 2 are different? I couldn't find a reference in the text to the dimensionality reduction mentioned here.\n\nPresentation:\n- The caption of Figure 1 only half describes the diagram there so I would suggest to extend it to give a better overview of the method. It is also missing the legend/explanation of what is \"A\", \"X\", \"M\", and \"C\".\n- \"At present, there is such a problem in image retrieval: although the results of global feature-based retrieval are correct in the category, they are not sorted in detail\" -> technically, the problem in instance-retrieval datasets is that they're not correct in the category so that's why re-ranking is performed.\n- \"But if we don’t differentiate the importance of attention and features, it’s hard to make sure that the learned attention is not part of features, which is not an ideal way for attention learning.\" -> this sentence should be re-written\n- \"However, these methods have not been well modified yet to solve image retrieval problems.\" -> this sentence should be re-written\n- \"in the next small section.\" -> in the next section\n- The nomenclature used in Figure 2 does not match with the text nor with the formulas in Eq 1-4\n- In page 4 the text mentions \"D\", which I'm assuming it's the output of \"X\" in Figure 1, but it has never been defined before.\n- The previous to last paragraph in page 4 is difficult to follow and it should be re-written more clearly. This is a crucial paragraph to properly understand the rationale behind the LALM module and the intermediate supervision and in its current shapes it loses impact.\n- \"We assume that there is an identical average target available for every category of objects\" -> I couldn't understand this assumption\n- Eq 6 makes reference of La but it was never defined in the text before (only in figure 1)\n- \"Obviously, UGALR completely outperforms previous global features based methods\" -> \"As we can see, UGALR...\"\n- \"The most amazing two improvements we made are\" -> \"The largest two improvements...\"\n- \"Amazingly, our approach using only global features outperforms\" -> \"Surprisingly, our approach...\"\n- Spaces are usually missing just before parenthesis", "summary_of_the_paper": "This paper addresses the problem of image retrieval and proposes a method that tries to learn better representations through the use of local feature attention and reduces memory and latency overheads by not requiring re-ranking with local features and only using global one.\n\nFor this, the paper proposes a module called Local Attention Learning Module (LALM) that performs both spatial- and channel-wise attention on local features and that it's plugged between two consecutive blocks of the backbone architecture. The output of this module is then concatenated to the input of the next block. The rationale behind this is that the following blocks will act as an homography transformation of the local features and will produce a better global representation. The method also uses intermediate supervision by computing a cross-entropy loss on top of the output features produced by the LAL Module.", "summary_of_the_review": "Overall, even though it doesn't introduce any impactful contribution, I think that the paper proposes an interesting method that is able to obtain very good results using only global features, outperforming methods that also combine re-ranking with local features. However, I find it difficult to recommend this paper for its acceptance in its current shape since its presentation is not adequate. Introduction and Method sections are sometimes difficult to follow and I couldn't fully grasp some of the ideas and rationale behind the design choices. Another pass in the text is required, and figures and formulas should also be updated and improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eViPHAxBai4", "reviewer": "Reviewer_UMH7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* the proposed architecture is deeply described, justifying each of its parts;\n* the experiments are performed using two datasets and the proposed model is compared with a lot of other models, providing a serious and robust discussion;\n* the ablation analysis is complete and justifies the use of each component of the architecture, describing numerically why each macro-layer (e.g. LALM) helps to improve general performances;\n* all the results against the SOTA models are complete and well described;\n* one of the main (well-described) goals of the architecture is to speed up the retrieval process reducing the used space. A comparison of these aspects has been clearly done, demonstrating the great advantages of UGALR with respect to the other models.\n* the benefits in terms of time and space do not preclude an improvement in accuracy performance, which is higher than in the baseline models.\n\nWeaknesses:\n* even if complete, the description of the architecture seems a little bit confusing. For example, the use of capital letters in figure 1 (and then in the text)  is confusing;\n* the comparison in terms of time and memory seems ok but is not very clear if all the models were tested on the same GPU (using the same environment). If not, this is a real problem and the results cannot be taken into account;\n* there is no error analysis. The accuracy is higher with respect to baseline models, but not that much; in order to study why, an error analysis would be helpful, maybe providing a comparison with the common errors of the other SOTA models.\n\nMinor grammar errors:\n* in figure 2 you use $\\varphi$ (varphi) but in the text $\\phi$ (phi) is always used;\n* lack some spaces between words and brackets (but this could be a template prerogative)\n", "summary_of_the_paper": "This paper proposes a new pipeline architecture for image retrieval. The main purpose of the proposed architecture is to overcome the time and space consumption due to the images' re-ranking that exploits local features (after the search using global features). \nIn order to do that, UGALR is proposed as a single-stage pipeline able to combine global and local features to speed up the retrieval process, removing the re-ranking step. This is achieved using a CNN able to learn the homography transformation in local feature matching.\nMoreover, the architecture combines spatial and channel attention with the aid of intermediate supervision, obtaining better performances than the SOTA models presented in the analysis of the results. This is achieved with the introduction of a Location Attention Learning Module (called LALM).\n", "summary_of_the_review": "The paper describes the proposed architecture proving a complete explanation of all of its parts (with the help of a complete ablation analysis). The comparison with a lot SOTA models on two datasets confirms the robustness of the proposed model. Even if the improvements in terms of accuracy are not so high (not a negative point, there is however a clear improvement) there is a huge improvement in terms of speed and memory used. An improvement for the paper would be the addition of a complete error analysis that now is not present and a better description of the entire architecture that, at present, is not very clear. Moreover, is not clearly specified if all the models have been tested on the same GPU (by the authors) to test speed and memory usage, this is an important point to specify because is one of the main goals of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L7e_mhxsyZj", "reviewer": "Reviewer_N1bK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is feasible. However, its novelty is pretty limited. Also, many points are confusing for readers.", "summary_of_the_paper": "In this work, the authors propose a Unifying Global and Attention-based Local Features Retrieval method (referred to as UGALR). UGALR accelerates extraction speed and reduces memory consumption by removing the re-ranking process and learning local feature matching with convolutional neural networks instead of the RANSAC algorithm. In addition, UGALR learns more accurate and semantic local information by combining spatial and channel attention with intermediate supervision.", "summary_of_the_review": "Due to the limited contributions, I do not recommend this paper to publish on ICRL.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EGYBGsHJQL2", "reviewer": "Reviewer_wJ79", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nS1) Combining local and global image features is a promising research direction.\nS2) The paper provides several experiments, with many ablations.\nS3) Some performance improvements seem to be observed compared to previous work.\n\nWeaknesses:\nW1) The authors naively claim several times in the paper that their method is learning homography transformations (Introduction, Methodology, Experiments, Conclusions). However, one can clearly see that this is not the case. The authors are simply concatenating local feature maps with the ones from the global feature branch and supervising by ArcFace classification loss. The authors seem to be assuming that homography learning is implicitly happening, which is naive. Similarly, the abstract claims that the proposed method is “learning local feature matching with convolutional neural networks instead of RANSAC algorithm” – this is incorrect.\nW2) The paper is not well written, and several parts are hard to understand. For example, in Fig 1 the paper has blocks named A, X, M, C – what do they refer to? Similarly, in page 4 the text start referring to “D” (“D is then followed (...)”), but D had never been introduced before. D (and O, which is also later introduced) should be added to Fig 1, and A/X/M/C should be explained in the caption/text. It is also very difficult to understand the way that local and global feature maps are combined – the paragraph “Backbone Network” and the one before it are quite confusing.\nW3) The authors seem to have suspiciously excluded results from previous work. For example, Tab 1 does not show numbers for “R50-DELG (global+local)” in the “+1M” dataset setups, which are available in the exact same paper where the other numbers from the table were taken from. I suspect that they were not included because in half of the cases the proposed method underperforms compared to “R50-DELG (global+local)” – specifically, the ROxf+1M cases.\nW4) In Tab 2, the authors simply take latency numbers reported in previous work and compare to their own for the new method, using a different GPU. The comparison does not seem appropriate.\n", "summary_of_the_paper": "The paper proposes a new method to combine global and local image features, targeted at image retrieval applications. It designs a new local feature model branch where both spatial and channel attention are used. The local feature branch undergoes supervision directly (in the paper called “intermediate supervision”), and this branch’s output is also concatenated to the global feature branch’s output in order to produce a final image embedding at the end. ", "summary_of_the_review": "The paper addresses an important technical direction of fusing local and global image features. However, it has several major flaws, such as: claiming learnable matching/homography when none of that is happening, suspiciously excluding previous work’s performance numbers in some cases; poor writing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method to combine global and local image features, targeted at image retrieval applications. The main idea is a model branch where both spatial and channel attention are used. The local feature branch undergoes supervision directly and this branch’s output is concatenated to the global feature branch’s output in order to eventually produce the final image embedding. 

The reviewers appreciated the care in the evaluation (ablative analysis) and the promise of the approach compared to existing baselines. The reviewers also expressed concerns about several claims, for instance that the proposed approach is able to learn homography transformations, the quality of the exposition, and missing baselines. The reviewers also pointed out that several parts of the paper were hard to follow and important details were missing. 

The authors submitted responses to the reviewers' comments. After reading the response, updating the reviews, and discussion, the reviewers considered that ‘the concatenation of local features with global ones works does not mean at all that some geometric transformation is learned’ and the justification provided for omitting baselines (suggested by the reviewers) were unconvincing. The feedback provided was already fruitful, yet major issues still remain.

We encourage the paper to pursue their approach further taking into account the reviewers' comments, encouragements, and suggestions. The detailed feedback lays out a clear path to generate a stronger submission to a future venue.

Reject.
Decision: Reject