A General Rank Preserving Framework for Asymmetric Image Retrieval
Authors: Hui Wu
Min Wang
Wengang Zhou
Houqiang Li
Venue: ICLR 2023 poster (2023)
Keywords: Asymmetric image retreival

Abstract: Asymmetric image retrieval aims to deploy compatible models on platforms of different resources to achieve a balance between computational efficiency and retrieval accuracy. The most critical issue is how to align the output features of different models. Despite the great progress, existing approaches apply strong constraints so that features or neighbor structures are strictly aligned across different models. However, such a one-to-one constraint is too strict to be well preserved for the query models with low capacity. Considering that the primary concern of the users is the rank of the returned images, we propose a generic rank preserving framework, which achieves feature compatibility and the order consistency between query and gallery models simultaneously. Specifically, we propose two alternatives to instantiate the framework. One realizes straightforward rank order preservation by directly preserving the consistency of the sorting results. To make sorting process differentiable, the Heaviside step function in sorting is approximated by the sigmoid function. The other aims to preserve a learnable monotonic mapping relationship between the returned similarity scores of query and gallery models. The mapped similarity scores of gallery model are considered as pseudo-supervision to guide the query model training. Extensive experiments on various large-scale datasets demonstrate the superiority of our two proposed methods.

Published as a conference paper at ICLR 2023
A GENERAL RANK PRESERVING FRAMEWORK FOR ASYMMETRIC IMAGE RETRIEVAL
Hui Wu' Min Wang** Wengang Zhou 1.2 Houqiang Li
CAS Key Laboratory of Technology in GIPAS, University of Science and Technology of China 2 Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
wh241300emas uste. edu. wangmine iai wsw..utttcc..o.... cn, {zhwg, 1ihg) eustc. www.eeeecc.com
ABSTRACT
Asymmetric image retrieval aims to deploy compatible models on platforms of different resources to achieve a balance between computational efficiency and re- trieval aceuracy. The most critical issue is how align the output features of dif- ferent models. Despite the great progress, existing approaches apply strong con- straints so that features or neighbor structures are strictly aligned across different models. However, such one-to-one constraint is too strict to be well preserved for the query models with low capacity. Considering that the primary concern of the users is the rank of the returned images, we propose a generic rank preserv- ing framework, which achieves feature compatibility and the order consistency between query and gallery models simultaneously. Specifically, we propose two alternatives to instantiate the framework. One realizes straightforward rank order preservation by directly preserving the consistency of the sorting results. To make sorting process differentiable, the Heaviside step function in sorting is approxi- mated by the sigmoid function. The other aims preserve a learnable monotonic mapping relationship between the returned similarity scores of query and gallery models. The mapped similarity scores of gallery model are considered pseudo- supervision to guide the query model training. Extensive experiments on various large-scale datasets demonstrate the superiority of our two proposed methods.
INTRODUCTION
In recent years, deep representation learning methods (Babenko et al., 2014; Tolias et al., 2016; 2020) have achieved great progress image retrieval. Typically, most existing image retrieval tasks belong to symmetric image retrieval, in which deep representation model is deployed to map both query and gallery images into the same discriminative feature space. During online retrieval, gallery images are ranked by sorting their feature distances, e.g., cosine similarity or Euclidean distance, against query image. To achieve high retrieval accuracy, most existing methods tend to deploy a large powerful representation model. In real-world visual search system, the gallery side is usually on the cloud-based platforms, which have sufficient resources to deploy large powerful models. As for the query side, e.g., mobile phone or smart camera, its resources too constrained meet the demand of deploying large models. To strike a balance between performance and efficiency, it is better to deploy lightweight model on the query side, while large one for the gallery side. This setup is denoted as asymmetric image retrieval (Duggal et al., 2021; Budnik & Avrithis, 2021).
For asymmetric retrieval, how align the embedding spaces of the query and gallery models is the core problem. To this end, BCT (Shen et al., 2020) first introduces feature compatibility learning. Concurrent work AML (Budnik & Avrithis, 2021) learns the query model by contrastive learning with gallery model extracting features for positive and negative samples. Recently, CSD (Wu et al., 2022b) achieves promising results by considering both first-order feature imitation and second-order neighbor similarity preservation during the learning of the query model. Despite the great progress, existing methods enforce the consistency of features or neighbor structures across different models, which is too strict to be well preserved for lightweight query models with low capacity. For users, the order of the returned images plays a more important role than the image features or similarity
Corresponding authors: Min Wang and Wengang Zhou

---

Published as a conference paper at ICLR 2023
scores. Strictly enforcing the feature-level one-to-one consistency is not the best choice to achieve better asymmetrie retrieval accuracy, while rank preserving deserves more attention.
To address the above issues, propose general rank preserving framework, which directly op- timizes the consistency of rank order to realize the feature compatibility across query and gallery models implicitly. Specifically, for training image, we first extract its features with the query and gallery models, respectively. Then, the gallery feature is utilized for symmetric retrieval in a database, in which images are also embedded by the gallery model. After that, the ranking list and similarity scores are returned. We select the top K images in the ranking list and calculate their asymmetric similarity scores with the query feature. These asymmetric similarity scores may result in different rank orders from those returned by symmetric retrieval. Thus, two instantiation methods are proposed to achieve the consistency of these two rank orders.
The first aims to directly optimize the rank order consistency of the sorting results. To make the sorting process differentiable, the sigmoid function is adopted to approximate the Heaviside step function (Davies, 1978), which is typically used for numerical comparison. Secondly, we propose to maintain a learnable monotonic mapping relationship between the symmetric and asymmetric similarity scores. A learnable monotonically increasing function is applied to the similarity scores returned by the symmetric retrieval, which will serve as the pseudo-supervision of the query model. Then, we constrain the consistency between the mapped similarity scores and the asymmetric sim- ilarity scores to optimize the query model. Notably, both two instantiations preserve the rank order of the images, which are returned by symmetric retrieval.
Compared with previous methods, our framework has a unique advantage. It does not constrain the query model to mimic the features or the overall neighbor structures of the gallery model. In- stead, it expects that the query model maintains the order of the returned images from symmetric retrieval. Thus, our framework weakens the restriction on the query model with low capacity, and reduces the risk of overfitting during learning the query model. Besides, our framework utilizes no annotation or label of the training data, which makes flexible and adaptable in various real-world scenarios. To evaluate our approach, comprehensive experiments are conducted on four popular re- trieval datasets. Ablations demonstrate the effectiveness and generalizability of our framework. Our approach surpasses the existing state-of-the-art methods by a considerable margin.
2 RELATED WORK
Image Retrieval. Given a large corpus, image retrieval aims to efficiently find the images, which contain the same object or describe the same content with the queries, based on their feature sim- ilarities. Most of the traditional image retrieval systems are based on local features (Lowe, 2004; Bay et al., 2006) and bag-of-words (Sivic & Zisserman, 2003; Philbin et al., 2007) representations borrowed from text retrieval. There are also several aggregation methods including VLAD (Jégou et al., 2011), Fisher vectors (Perronnin et al., 2010) and ASMK (Tolias et al., 2013), which are used to aggregate local features into compact representations for efficient search. Recently, with the pro- posed various pooling methods (Kalantidis al., 2016; Tolias et al., 2016; Radenović al., 2018b) and loss functions (Revaud et al., 2019; Deng et al., 2019; Weinzaepfel et al., 2022), deep learning has greatly improved the performance of image retrieval. Despite the great progress, a large deep model is usually deployed for its optimal performance, which, however, is not applicable in some resource-constrained scenarios. In this work, we focus on asymmetric retrieval, where the query (user) side deploys a lightweight model while the gallery side applies a large model.
Feature Compatibility. The core of asymmetric retrieval is to align the features of the query and gallery models, which is also known as feature compatibility. BCT (Shen et al., 2020) first formu- lates the problem of feature compatible learning and reuses the old classifier for the query model training. AML (Budnik & Avrithis, 2021) achieves the feature compatibility by performing asym- metric contrastive learning between different models. After that, CSD (Wu et al., 2022b) achieves the preservation of neighbor similarities in the embedding space of the gallery model in unsu- pervised manner. As for HVS (Duggal et al., 2021), both parameter and architecture are considered simultaneously in unified framework, which gives promising performance. Other lines of research follow the model regression problem (Yan et al., 2021; Zhang et al., 2022; Duggal et al., 2022) dur- ing gallery model updating, which is also related to feature compatibility. In this work, start from
2

---

Published as a conference paper at ICLR 2023
the main concern of the users, i.e., the order of the returned images in the ranking list, and propose a general rank preserving framework, which is free of annotations from training datasets.
Lightweight Network. Thanks to the evolution of the network architecture (He et al., 2016; Tan & Le, 2021), deep convolutional neural networks (CNNs) achieve superior performance in various computer vision tasks. Usually, large powerful model leads to better performance with the con- sumption of more storage and computational resources. Real-world tasks aim to achieve the best accuracy with limited computational budget, which is determined by the target platforms, The demand of deploying nigh-perfforance deep models on resource-constrainee platforms has led to series of studies on model compression (Antonio et al., 2016; He et al., 2018b; Oktay et al., 2020) and efficient network architecture design, e.g., MobileNets (Howard et al., 2017; Sandler et al., 2018), ShuffleNets (Zhang et al., 2018; Ma et al., 2018) and EfficientNets (Tan & Le, 2019). In this work, we focus on asymmetric retrieval in resource-come cssuurce-constrained scenario. Since query features are extracted on resource-com conttraned end platforms, our approach employs the various lightweight models mentioned above as query models.
Smooth Rank Approximation. There is a long history of designing smooth surrogate for rank ap- proximation. In image retrieval, the well-known surrogate is to constrain the relative relationships between pairs (Raia et al., 2006) or triplets (Gordo et al., 2017), which implicitly leads to partial ranking. Some methods propose to utilize smooth discretization of similarity scores (He et al., Sa; Cakir et al., 2019; Ustinova & Lempitsky, 2016; Revaud et al., 2019) for the rank approx- imation. Other approaches explicitly approximate the non-differentiable rank metric with neural network (Engilberge et al., 2019) or a sum of sigmoid functions (Brown et al., 2020; Huang et al., 2022; Patel al., 2022). Recently, more accurate and robust approximation method is proposed in ROADMAP (Elias et al., 2021). However, all the methods mentioned above are designed for symmetric retrieval, where only a single model exists and no cross- croes- feature compatibility has been considered. Thus, they cannot be directly applied for asymmetric retrieval. In our approach, asymmetric and symmetric retrieval are performed with the same query. Then, the order of two returned ranking lists are constrained to be consistent, which also ensures feature compatibility.
BACKGROUND: ASYMMETRIC IMAGE RETRIEVAL
Given images of interest (referred to as query set Q), image retrieval targets at correctly finding images of the same content or object from large-scale gallery set G. An image encoder o(-) is deployed to map the images into Lz-normalized feature vectors. Then, the cosine similarity of two normalized vectors is used for measuring the similarity between query and gallery images. Usually, some metrics, e.g,, mean Average Precision (mAP), are adopted to evaluate retrieval system, which conditioned on o6-), Q and G. For convenience, we ignore query and gallery sets and denote the metric as M( (.)))))))))) (.)-)) where (.)() and (-(-) are the image encoders deployed for query and gallery feature extraction, respectively.
In a conventional symmetric retrieval system, the same encoder, i.e., (g(+) = (g(+), is used for both gallery and query sides. Typically, deploying a powerful model results in better retrieval accuracy. However, it is not applicable in some resource-conssrrineed platforms, egg., mobile devices or smart cameras. Assuming (.).)) is different from and significantly smaller than (g).), asymmetric image retrieval leverages (.) to embed query images and (-)+) to embed the gallery images. Thus, we should ensure that the lightweight query model (-)-) maps images into the same embedding space of the large gallery model (-))) Besides, an asymmetric retrieval system is expected to achieve a retrieval accuracy comparable to that of a symmetric retrieval system (Duggal al., 2021),
a MM(,((). M(6g().0g())
M(),(),,()).
4 RANK PERSEVERING FRAMEWORK
An overview of our framework is shown in Fig. 1. Given a well-trained gallery model (,(-), we aim to learn lightweight query model (,(+) to be compatible with it. Assume there exits a training gallery set Gr we first embed it into F [f] f2. ...., J Nxd with (g(+):
f/ (g( RRd 1.2 ..... N,
3

---

Published as a conference paper at ICLR 2023
Rank weight W
Pg())
9
Features F extracted by dg(+)
10 0
H00
Gallery St
Gallery Model
D
Wiightel MSE
Symmetric simalarities Sy
Symmetric retrieval
OOO
S
Top-K returned results
Rank onder preservation
Rank Order Preservation
D2
IDs
Images
Menetomic similarity preservation
f(S,)
softmax
Features
Fi
$
OOO
Pg
f(x
KL Loss
(g(C)
1.0
soltmax
Query
Cosine
O 0.00
Model
similarity
Asymmetric similaritiss
S
$4
-1.0
Monotonic Similarity Preservation
Figure I: An overview of Rank preserving framework, Given a training image I, gallery model (-) and query model (.)-)) encode it into features g and q. respectively. g is treated as the query to search in a training gallery set G., of which the images are also embedded by dg(:). We fetch the features FK of the top-K images in the ranking list and calculate the asymmetric similarity scores S with q. Two instantiations of our framework named Ranking Order Preservation (Sec. 4.1) and Monotonic Similarity Preservation (Sec. 4.2) are proposed to ensure the consistency of rank orders when the query is embedded by (.).) and (.),), respectively.
where Ji is the i-th image in the training gallery set G.. During the learning of <g(+). gallery model fixed. For each training image x, we extract its features q and g with (,).) and (.). respectively:
= 0,(a) E RRd g o Rdd.
(3)
Then, we perform symmetric retrieval in G, with g as the query. After that, we obtain the ranking list R [2,, .r......K rx] e R-K and similarity scores Sg [g fo gg" fga..... frx] ] R&K of lop-K images, where ri denotes the ID of the i-th image in G. Notably, the values in S satisfy the monotonically decreasing property, i.e., g" 1 > g" f ... =g" fgx. With the ranking list R, the corresponding feature embeddings Fx [f f ........ f= for the top-K images are taken from F. Then, we calculate the asymmetric similarity scores between the query feature q and Fx:
S, = lg" fg q" fg ........ {
(4)
In CSD (Wu et al., 2022b), the consistency of S. and Sy directly constrained to optimize the query model. However, we argue that it is too strict to be preserved well for the lightweight model with low capacity. Besides, the user experience mainly influenced by the rank order of the returned images rather than the specific similarity scores. Thus, it is better to directly impose the constraint on the rank order of the returned images rather than the specific similarity scores. Specifically, we need to ensure the values in Sg are also monotonically decreasing:
qT fg JT .......... q J
To this end, we propose two methods instantiate the rank preserving framework. One aims to achieve straightforward Rank Order Preservation (ROP) by constraining the consistency of the sorting results, which are formulated as the indicator matrices in Sec. 4.1. Typically, is not feasible to optimize the sorting results directly due to the non-differentiable Heaviside step function (Davies, 1978), which used for numerical comparison in sorting. Thus, sigmoid function is introduced as the smooth approximation, which enables the optimization of sorting process with back-propagatte methods. The other is Monotonic Similarity Preservation (MSP), which aims to preserve a learn- able monotonic mapping relationship between the returned similarity scores S, and Sy, Specifically, we learn a monotonically increasing function, which predicts S, given Sy- Then, the consistency between the mapped Sg and S, is restricted to train the query model.
4.1 RANK ORDER PRESERVATION
The motivation of our method is directly preserving the rank order of the returned images between asymmetric and symmetric retrieval. Thus, the critical problem is select suitable evaluation
4

---

Published as a conference paper at ICLR 2023
(x) and ((),))
Derivatives of (z) and a(x,
R0sf+ RIM
RDd RIM
Biz
0.8
p Reg
0.5
MMrnnoue Stndamy Pruaration
9.3
0.8
0.6
0.66
992
0.5
0.4
0.2
11.2
0.2
Figure 2: (Left) Heaviside step function and two sig- moid functions with different tempurature ap- proximations. (Right) Corresponding derivatives.
Figure 3: Visualization of the similarity score dis-
tribution (symmetric vs. asymmetric). syt symmetric similarity score; syi asymmetric similarity score.
metric as the constraint. In this section, we dive into the sorting process and constrain the sorting results of symmetric and asymmetric retrieval be consistent.
Sorting process. Sorting the similarity scores between the query and database images is an essential operation to get the final ranking list image retrieval. Numerical comparison is the most funda- mental operation in various sorting algorithms. In this work, we take the comparison sorting as the sorting algorithm. Assume there exists list of similarity scores S [s1, .2,..., *N] E IR needed to be sorted. We first compute the difference matrix D RNN* between any elements in S. Then, the binary indicator matrix I N is calculated by applying the Heaviside step function (x), which is equal to 0 for negative values, otherwise equal to I, the difference matrix D:
8N
81
81
H(s1 - 81)
H(SN 81)
D=
(6)
SN
SN
SN
H(s1 1 SN) H(SN SN)
Each element I.j in the indicator matrix I denotes the relative relationship between the similarity larger than or equal 8;, otherwise, It, 0.
scores 8; and sy. It, =1, if
Indicator matrix consistency. Since the indicator matrix demonstrates the relative relationships of similarity scores, we take as the constraint for optimizing the query model. If the indicator matrix is well preserved, the asymmetric retrieval will result in the same ranking list as the symmetric retrieval. Specifically, we take S, and S, into Eq. (6) get the corresponding indicator matrices 1 R K and I IR KxK;
H(A KJ)
)(
9
1=
H(A)K)
H(A)")
where Am STD ST. 1 E {g.9). Generally, if one gallery image has higher similarity score against query image, is more possible to be the true positive, which deserves more attention during the training of query model. To this end, we design a ranking weight W; exp(St/T,///8
ΣK exp( (S TT)) = 0, 1, K, where т, is the temperature. Finally, the weighted indicator matrix consistency loss is minimized as the final objective function to train the query model:
K K
LROP = W. 19 12.
E 1-1--
-
(8)
Heaviside step function approximation. Unfortunately, the derivative of the Heaviside step func- tion Hxx) is Dirac delta function 5(x), which is either flat everywhere, with zero gradient, dis- continuous, and hence cannot be optimized with gradient- based method. Inspired by (Brown et al., 2020), the sigmoid function a T) - where denotes the temperature adjusting the sharp- ness, is used to approximate the Heaviside step function smoothly. As shown in Fig. 2, the temper- ature governs the approximation tightness and the -radient-effective interval. Substituting a axx 7) into Eq. (8), the weighted indicator matrix consistency loss is approximated as:
K K
LROP
Wi(H(A)) - (AA 7)
(9)
5

---

Published as a conference paper at ICLR 2023
4.2 MONOTONIC SIMILARITY PRESERVATION
In this section, we introduce another method to preserve the rank order. In Fig. 3, we visualize the distributions of the similarity score pairs 8, and 8g in existing methods. is observed that same sg may correspond to a wide range of values for 8q- In other words, the similarity score pairs locate on wide strips, which means that existing methods do not well preserve the order of the images returned by symmetric retrieval. We think it is due to the fact that existing methods all impose a strict one-to-one constraint, which may cause overfitting for the query model with low capacity.
Therefore, we introduce a learnable monotonic mapping function f(x) applied to Sg to form the pseudo-supervision of S.. Formally, M = f(S)), i = 1, 1.2...., K. This avoids the strict neigh- bor structure alignment between query and gallery models. After that. the Kulbback-Leiber (KL) divergence, which shows excellent performance in CSD (Wu et al., 2022b), between M, and Sy is adopted as the final objective function to optimize the query model. Specifically, we first convert Mg g and y into the form of probability distributions:
exp (Ma/Ta) Σ exp (M
exp (5
P9
P4
i 1,2,.... K,
(10)
E exp (Tg
where Ty and Ty are temperature coefficients. Both To and Ty are set less than I to keep (-(-) focus on the top images of the ranking list. Then, the monotonic similarity preservation loss is defined as
K
LMSP = KL(Pollpg) = Σ
P log
(11)
P
To preserve the order of each elements in Sy. we should ensure that the function f(x) is monotoni- cally increasing. In this work, we consider three common families of the monotonically increasing functions, which are discussed in the following.
Logarithmic function. Considering that the definition domain of the logarithmic functions is (0, +20 and the cosine similarity lies between -1 and 1, we define the mapping function as f(z) (x + 1), (-1.0 < < 1.0, > 1), where is learnable parameter.
Exponential function. Another common monotonically increasing function is the exponential func- tion. To make each Mj exist in the range less than 1.0, fzz is defined as f(x) a=-1, (-1.0- < 1.0, 1), where a is also learnable parameter.
Polynomial function. There are a wide variety of polynomial combinations. In this work, we consider simple case. We first choose set of basis functions X' = {Ir 220 ..... No then the mapping function is defined as linear combinations of those basis functions. Formally, f(x) = N.N
-1 a,2² (-1.0- < < 1.0, a > 0), where (a1, a2...., ..v) is a set of learnable parameters. To ensure that f(x) monotonically increasing, each a; should be greater than O. Besides, we also make the sum of a; equal to I to control the range of (z))
Relation with contextual similarity distillation. In CSD (Wu et al., 2022b), KL loss strictly re- stricts the consistency between S, and Sg. When the loss minimum is achieved, linear relation- ship Tq S, = S, between S, and S, is presented. In MSP, when the mapping function is set to f(x) x (diagonal line in Fig. 3), My is equal to S.. the loss function LMsP degrades the form in CSD. Thus, CSD is a special case of our monotonic similarity preservation. As shown in Fig. 3, MSP method realizes narrower striped similarity distribution, since it introduces learnable monotonically increasing mapping between symmetric and asymmetric similarities. As a result, our MSP method could preserve the order of the returned images better than CSD.
EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
Training datasets. Two datasets are used for training. One is 11M-120 (Radenović et al., 2018b), of which 551 3D models are taken for training while the other 162 3D models for validation. The other is GLDv2 (Weyand et al., 2020), which consists of 1, 580, 470 images with 81, 311 classes. We randomly sample 80% images from GL for training and leave the rest 20% for validation.

---

Published as a conference paper at ICLR 2023
ROxf RIM (avg)
Rar RIM (avg)
INSTRE -2
GLDv2-Test
14
14 34,24
61 61
60.81
13.33
31:06
mAPE 100K
mAP(S)
30
28,56
mAP((16)
-3
32
32,23 CSD
mAP(SE)
58
CSD
12
71.81+ CSD
28
58
10
Ours" Ouurs
31
Ours Ours
Ours' Ours
Ours
Ours
26
5122,018
4,095
8,192
5122.18
4,066
5122,005
8,152
5122.015
1,096
8,192
K
K
Figure 4: Analysis of the length of the ranking list K in our methods. SIM-120K, GeM and MobileNetV2 are adopted as training set, gallery and query models, respectively.
SIM-120K INSTRE ROxf + RIM Par + R.M
SfM-120K INSTRER
RMRPaar RIM
Wi
val
Ø
Med
Hard
Med
Hard
val
Ø
Med
Hard
Med
Hard
0.001
67.33
34.13
31.59
12.46
37.11
12.23
1
50.99
25.12
32.73
14.81
34.00
16.85
0.01
71.16
41.30
34.87
13.96
40.47
14.56
1/8
75.99
52.57
42,52
20.17
44.76
18.03
0.05
76.33
59.59
42.08
19.28
46.63
19.36
S
59.43
59.59
36.55
16.27
36.27
16.11
0.1
77.06
60.75
42.33
19.78
47.72
20.34
P.
76.91
58.63
42.73
20.07
47.92
20.11
0.5
75.46
57.19
41.68
18.83
45.64
18.64
P
77.06
60.75
42.33
19.78
47.72
20.34
1.0
74.86
55.78
40.94
18.15
44.36
16.99
1/K
69.57
31.08
34,95
15.89
44.36 16.99
Table 1: Analysis of temperature in Eq. (9). controls the smoothness of sigmoid function used Eq. (9). K: length of the ranking list; P. denotes to approximate the Hcaviside step function.
Table 2:
Analysis of ranking weight W: in
exp(Si //)//// Ex= exp(S)/TT).
Evaluation datasets and metrics. We evaluate the trained query models on four datasets under the setting of asymmetric retrieval, including GLDv2-Test (Weyand et al., 2020), Revisited Oxford with RIM (ROxf + R IM), Revisited Paris with RIM (RPar + RIMM (Radenović et al., 201 8a) and INSTRE (Wang Jiang, 2015). The evaluation metric for GLDv2- is mAP@100, while all other datasets are mAP. See App. A.1 for the more detailed descriptions of the testing sets.
Architectures.
Following the settings in CSD (Wu et al., 2022b), we choose ResNet101 (He et al., 2016) trained by GeM (Radenović et al., 2018b) and DELG (Cao et al., 2020) as the gallery models. For the query model, common lightweight models, e.g., ShuffleNets (Ma et al., 2018), MobileNets (Sandler et al., 2018) and EfficientNets (Tan & Le, 2019), are chosen. To adapt the lightweight model for the image retrieval task, we adjust the model architecture slightly. The details are present in App. A. with computation and parameter complexity statistics.
5.2 ANALYSIS AND ABLATIONS
In this section, we analyze the proposed rank preserving framework and perform exhaustive ab- lations. R101 and Mv2 denote ResNet101 and MobileNetV2, respectively. "Ours" and "Ours"" denote that we train lightweight query model (.(.) with ROP and MSP constraints, respectively.
Length K of the ranking list. In Fig. 4, we compare our proposed methods against CSD when ranking list has different length K E [64, 256, 512, 2048, 4096, 8192). As the length increases, the performance increases but saturates when K" = 4096, after which, the performance decreases. When K is small, the query model only needs focus on the elements in front of the ranking list, without taking full advantage of the order information in the ranking list. Thus, the performance unsatisfactory. On the contrary, when K is particularly large, the query model is concerned with very wide range of elements in the ranking list. Images at the bottom of the ranking list are far away from the query, their relative orders have almost no effect on the overall retrieval accuracy. Constraining the query model to preserve the order of this part leads to a decreased performance.
Heaviside step function approximation. In Tab. 1, we investigate the effect of the temperature т, which governs the smoothness of the sigmoid function. Results show that 7 = 0.1 leads to the best performance over different datasets. As explained in Sec. 4. a smaller value of T leads to a nar- rower gradient-effective interval (Fig. 2) and a tighter approximation to the Heaviside step function. The strong acceleration of the gradient around zero encourages moving instances in the embedding space, leading to change of rank. However, excessively small 7 causes gradient vanishing, which is harmful to the optimization of the query model. In contrast, a large value of T provides a wide gradient-effective interval at the cost of looser approximation the true order.
7

---

Published as a conference paper at ICLR 2023
R0xf RIM (avg)
R.Par + RIM (avg)
R0xf RIM (avg)
R. RIM (mvg)
70
40
OOurs"
BCSD BOuns BOuns
B CSD Ours B Ouns'
CCD Ours Ours
65
35
mAP(%)
mAPY (5)
55
60
30
30
50
25
25
45
DOLG CVNo
Token
50
DOLG CVNNe
Token
20
Sv2
SV2
E60
Eb1
20
SV2
5v2
E800
Ehl
Figure 5: Analysis of different model architectures. Two on the left: Three lypical deep representation mod- cls are chosen as gallery models. (1) DOLG (Yang al., 2021) based on global and local feature aggregation. (2) CVNet (Lee al., 2022) based on global pooling. (3) Token (Wu al., 2022a) based on local feature aggregation. MobileNetV2 is adopted as the query model for all settings. Two on the right: GeM serves as the gallery model, and four lightweight models, including ShuffleNet2 (0.5x) (Sv20.5x), ShuffleNetv2 (Sv2), EfficientNetBO (EbO) and EfficientNe(BI (Eb1), are chosen query models, respectively.
Method
dataset T
Training
GLDv2
-Test
INSTRE ROxf RIM RP +
Ø +DIM Med
Hard
Med
RIM Hard
Method
Query Gallery Net du( Net Ne
GLDv2
-Test
INSTRE
2 +DIM
Reg CSD
12.11
60.88 51.62
40.34
18.15
46.91
19.77
GeM
R101
R101
14.47 71.23 64.10
GLDv2
14.55
64.51
55.77
46.02
21.85
52.05
24.13
Contr"
9.44
28.05
15.93
Ours Ours
-index
15.01
65.70
56.74
47.57
23.33
53.44
25.01
Reg
10.58
34.96 23.06
15.30
65.36
57.77
47.76
23.12
53.12
25.43
CSD
Mv2
R101
11.81
58.22 46.84
Ours
12.35
60.81 50.96
Reg
11.22
69.34
59.44
38.76
16.72
44.24
17.29
Ours
13,33
60.86
49.72
CSD
INSTRE!
13.00
71.86
62.83
44.95
21.09
49.07
21.94
DELG
RIOI
R101
26.77
37.54 31.66
Ours
DIM
14.27
75.30
64.50
46.75
22.74
51.59
23.87
Ours
14,06
73.89
64.18
46.08
22.12
51.10
23.13
Reg
21.59
21.84
13.88
HVS
21.99
22.80
14.71
Reg
ROxt
12.54
61.41
51.67
42.19
19.23
47.31
19.46
LCE
Mv2
R101
22.42
23.85
15.91
Ours
CSD
+ RPar
14.85
14.19
67.39 60.78
65.65
57.81
49.83
46.81
23.97
21.70
53.06
54.46
27.29
25.88
CSD
Ours
23.86
23,45
22.34
29.93
21.76
15.44
Ours"
+ RIM
14.31
68.05 61.42 5U.77
24.96
55.03
27.56
Ours
25.44
34.34
26.58
Table 4: Comparison of different unsupervised meth- ods trained on the deployed gallery set. i denotes the gallery set of that testing dataset. MobileNetV2 and GeM are adopted as query and gallery models, respectively. Reg: (Budnik& & Avrithis, 30 CSD: (Wu et al., 2022b).
Table 5: Comparison to the state-of-the-
art methods on INSTRE and GLDv2- Test. Contr (Budnik & Avrithis,
2021); HVS: (Duggal et al., 2021);
LCE: (Meng et al., 2021).
Rank weight W in Eq. (9). Tab. 2 shows the impact of different ranking weight W. When no weight is used, ie., W, 1, it leads to unsatisfactory results, and the best results are obtained when both similarity score S, and ranking position i are considered, simultancously. Since the images at bottom of the ranking list are more likely dissimilar from the queries, keeping the order between them wastes the representation capability of the query model. It should focus more the order of samples at the top of the ranking list.
Various query and gallery models. In Fig. 5, study the adaptation to different model architec- tures. Specifically, models with different architectures are adopted as gallery and query models. See App. A. for the number of parameter and computation complexity of different models in details. Our methods outperform CSD in all settings, demonstrating the superiority of rank preserving to the strict neighbor structure alignment.
Impact of the mapping function f(x). In Tab. 3, we study three different types of mapping functions introduced in Sec. 4.2. The specific parameters of each function are present in App. C.2. We further visual- ize the distribution of the similarity scores obtained from three mapping functions in App. D.3. The logarithmic function fits the distribution best and thus leads to the opti- mal asymmetric retrieval accuracy.
SIM-120k
ROxf R.MRPPr RIM Hard
f(x) val
INSTRE
Med
Hard
Med
a T-1
77.83
62.69
43.32
20.51
47.94 20.66
log, (ar + 1)
78.94
60.61
44.24
20.93
48.13 21.45
3--1 f/22
78.49
60.86
43.47
20.20
47.56 20.33
11=1 1/3
78.07
59.17
42,20 19.45 47.84 20.34
Table 3: Analysis of different mapping functions. SIM- 120k, MobileNetV2 and GeM are adopted as training set, query and gallery models, respectively.
Training with unlabeled data. Since our methods require no annotations from training set, various unlabeled data can be utilized. In practical scenarios, the deployed gallery images can be utilized to train lightweight query models. Notably, query images, which are unknown in advance, cannot
8

---

Published as a conference paper at ICLR 2023
Query Net (.)-)
Gallery Net (.())
ROxf
ROxf RIM
RPar
RPar RIM
Method
Medium Hard
Medium Hard
Medium Hard
Medium Hard
Performances the gallery models
GeM
(Radenovid ot al. 20158)
Mv2
Mv2
58,81
33.41
40.02
17.71
67.87
40.97
42.25
16.59
GeM
(Radenovi et al, 20156)
R101
R101
65.43
40.13
45.23
19.92
76.75
55.24
52.34
24.77
DELG
(Clo et al,, 20120)
Mv2
Mv2
62.42
36.56
42.21
18.64
77.91
57.96
55.09
28.81
DELG (Cao et. 2020)
R101
R101
78.55
60.89
66.02
41.75
88.58
76.05
73.65
51.46
Truining with GeM as gallery model and SJM-120k as training set
Contr (Rudeik & Avritiis, 2021)
47.10
21.80
18.00
6.20
61.50
37.70
28.80
8.80
Reg (Budnik & Avrithis, 2021)
49.20
23.20
26.50
7.80
65.00
40.70
34.60
12.70
CSD (Wu al. 2022b1
Mv2
R101
64.12
37.53
39.38
17.73
76.16
54.29
44.40
18.08
Ours
64.33
39.65
42.33
19.72
76.33
54.65
47.78
20.69
Ours
65.22
39.50
43.47
20.20
76.31
54.82
47.56
20.33
Training with DELG as gallery model and GLDv2 training
Reg (Budnik & Avrithis, 2021)
72.75
53.07
56.03
32.21
85.81
69.96
65.23
39.29
HVS (Duggal
74.39
54.68
58.24
34.77
86.86
72.42
67.44
43.39
LCE (Meng al. ol. 2021)
75.45
54.95
58.03
33.88
87.24
73.03
67.30
43.01
Mv2
RI01
CSD (Wa et 202261
76.01
57.61
58.42
36.59
87.55
74.82
69.24
45.68
Ours
79.11
59.44
64.34
39.19
89.08
76.78
72.00
48.47
Ours
77.55
58.54
64.18
40.19
88.74
76.04
71.04
48.65
Table 6: comparison (asymmetric retrieval) to the state-of-the-art methods. 4: the same gallery models as comparison methods; R101: ResNet101; Mv2: MobileNetV2; Black bold: the best performance. See App. B for more comparisons.
participate in the training. In Tab. 4, compared with two other unsupervised algorithms Reg and CSD, our methods achieve the optimal performance. Besides, it shows that when evaluating on test set, the model trained with the corresponding gallery images leads better performance. Except for the experiments present Tab. 4, there is no experiment in the paper, which uses the gallery images from the testing sets to participate in the training.
5.3 COMPARISON TO THE STATE-OF-THE-AN METHODS
We compare our method to the state-of-the-arr methods in Tab. 5 and Tab. 6. First, we observe that methods based on feature imitation, e.g., Reg, give inferior asymmetric retrieval performance. CSD achieves better results, owing to taking feature preservation and neighbor structure alignment into consideration, simultaneously. Our methods maintain the order of the returned images in ranking experience. It achieves the best performance
list, which is directly related datasets under the asymmetric setting. When trained with GeM as gallery model and SfM-120k as training set, our framework outperforms the best previous method CSD in mAp by 4.09%, 2.47% on ROxf + R.M R.M and 3. 3.16% 2.25% on RPar + R IM, with Medium and Hard protocols, respectively. The results on INSTRE and GLDv2-Test also confirm the superiority of our methods.
all evaluation
CONCLUSIONS
In this paper, we present a general rank preserving framework for asymmetric image retrieval. Dif- ferent from strictly feature imitation and neighbor structure alignment, we focus on preserving the order of the returned images in the ranking list when the same query is utilized for symmetric and asymmetric retrieval, respectively. To this end, we devise two instantiations. One is directly con- straining the consistency of the sorting results. To make sorting differentiable, sigmoid function is introduced as the smooth approximation for the non-differentiable Heaviside step function used in sorting. The other aims to preserve monotonic relationship between the returned similarity scores of symmetric and asymmetric retrieval. It introduces a learnable monotonically increasing function to the similarity scores of the symmetric retrieval, which is considered the target of the asymmetric similarity scores. The proposed framework requires no annotation or labels during training, which shows broad applicability and great generalizability in extensive experiments.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Con- tract 62102128 and 6202202000 1001, and in part by the Fundamental Research Funds for the Central Uni- versities under contract WK3490000007. It was also supported by the GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
REFERENCES
Polino Antonio, Razvan Pascanu, and Alistarh Dan. Model compression via distillation and quantization. In International Conference on Learning Representations (ICLR), 2016.
Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. Neural codes for image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 584-599, 2014.
Herbert Bay. Tinne Tuytelaars, and Luc Van Gool. SURF: Speeded up robust features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 404-417. 2006.
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-ap: Smoothing the path towards large-scale image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Matcusz Budnik and Yannis Avrithis. Asymmetric metric learning for knowledge transfer. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 8228-8228 2021.
Fatih Cakir, Kun He, Xide Xia, Brian Kulis, and Stan Sclaroff. Deep metric learning rank. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Bingyi Cao, André Araujo, and Jack Sim. Unifying deep local and global features for image search. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 726-743, 2020.
Brian Davies. Integral transforms and their applications /B, B. Davies. Applicd mathematical sciences (Springer- Verlag New York Inc.); 25. 1978.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei. ImageNet: large-scale hierarchi- cal image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248 255, 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos 7afeiriou. ArcFace: Additive angular margin loss for deep face recognition. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Rabul Duggal, Hao Zhou, Shuo Yang, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Compatibility-ave aware heterogencous visual search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10723- 2021.
Rahul Duggal, Hao Zhou, Shuo Yang, Jun Fang, Yuanjun Xiong, and Wei Xia. Towards regression-free neural networks for diverse compute platforms. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Ramzi Elias, THOME Nicolas, Rambour Clément, Audebert Nicolas, and Bitot Xavier. Robust and decompos- able average precision for image retrieval. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Martin Engilberge, Louis Chevallier, Patrick Pérez, and Matthieu Cord. Sodeep: A sorting deep net to leam ranking loss surrogates. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Albert Gordo, Jon Almazan, Jerome Revaud, and Diane Larlus. End-to-end learning of deep visual representa- tions for image retrieval. International Journal of Computer Vision (IJCV), pp. 237-254, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kun He, Fatih Cakir, Sarah Adel Bargal, and Stan Selaroff. Hashing tic-aware learning to rank. In Proceed- ings of the IEFE Conference on Computer Vision and Patter Recognition (CVPR), 2018a.
10

---

Published as a conference paper at ICLR 2023
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. AMC Automl for model compres- sion and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 834-800,220 2018b.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.08661 2017.
Yuge Huang, Jiaxiang Wu, Xingkun Xu, and Shouhong Ding. Evaluation-orieeneè knowledge distillation for deep face recognition. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15740-18749, 2022.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Teddy Furon, and Ondrej Chum. Efficient diffusion region manifolds: Recovering small ohjects with compact cmn representations. In Proceddings of the IEEF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 2077-2086, 2017.
Hervé Jégou, Matthijs Douze, Jorge Sánchez, Patrick Pérez, and Cordelia Schmid. Aggregating local image descriptors into compact codes. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). pp. 1704-1716, 2011.
Yannis Kalantidis, Clayton Mcllina, and Simon Osindero. Cross-dimensional weighting for aggregated deep convolutional features. Proceedings of the European Conference on Computer Vision (ECCV), pp. 685- 701, 2016.
Scongwon Lec, Hongje Scong, Suhyeon Lee, and Euntai Kim. Correlation verification for image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5374-5384, 2022.
David G Lowe. Distinctive image features from scalc-invariant keypoints. International Journal of Computer Vision (ICV),, pp. 91-110, 2004.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116- 131, 2018.
Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou. Learning compatible embeddings. In Proceed- ings of the IEEF International Conference on Computer Vision (ICCV), pp. 9939-9948, 2021.
Deniz Oktay, Johannes Ballé, Saurabh Singh, and Abhinav Shrivastava. Scalable model compression by entropy penalized reparameterization. In International Conference on Learning Representations (ICLR), 2020.
Yash Patel, Giorgos Tolias, and Jifí Matas. Recall@k k surrogate loss with large batches and similarity mixup. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7502-7511, 2022.
Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. Large-scale image retrieval with compressed fisher vectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3384-3391, 2010.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivie, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-8, 2007.
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondiej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5706-5715, 2018a
Filip Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning enn image retrieval with no human annotation.
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 1655-1668, 2018b.
Hadsell Raia, Chopra Sumit, and LeCun Yann. Dimensionality reduction by learning an invariant mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2006.
Jerome Revaud, Jon Almazán, Rafacl S Rezende, and Cesar Roherto de Souza. Learning with average precision: Training image retrieval with listwise loss. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). pp. 5107-5116, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFE Conference on Computer Vision and Pottern Recognition (CVPR), 2018
11

---

Published as a conference paper at ICLR 2023
Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compatibee representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Josef Sivie and Andrew Zisserman. Video google: A text retrieval approach to object matching videos. In Proceedings of the IEFE International Conference on Computer Vision (ICCV), pp. 1470-1470, 2003.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), pp. 6105 6114, 2019.
Mingxing Tan and Quoe Le. Efficientnetv2:: Smaller models faster training. In International Conference on Machine Learning (ICML), pp. 10096-10106, 2021.
Giorgos Tolias, Yannis Avrithis, and Hervé Jégou. To aggregate or not to aggregate: Selective match kernels for image search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), pp. 1401-144000 2013.
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular Object Retrieval With Integral Max- Max-Pooling of CNN Activations. In International Conference on Learning Representations (ICLR), 2016.
Giorgos Tolias, Tomas Jenicek, and Ondiej Chum. Learning and aggregating deep local descriptors for instance- level recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 460-477, 2020.
Evgeniya Ustinova and Victor Lempitsky. Learning deep embeddings with histogram loss. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Shuang Wang and Shuqiang Jiang. INSTRE: new benchmark for instance-level object retrieval and recog- nition. ACM Transactions Multimedia Computing, Communications, and Applications (TOMM), 11(3): 21 2015.
Philippe Weinzaepfel. Thomas Lucas, Diane Larlus, and Yannis Kalantidis. Learning super-features for image retrieval. In International Conference Learning Representations (ICLR), 2022.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2 large-scale bench- mark for instance-leve recognition and retrieval. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2575-2584, 2020.
Hui Wu, Min Wang, Wengang Zhou, Yang Hu, and Houqiang Li. Learning token-based representation for image retrieval. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 2703-2711, 2022a.
Hui Wu, Min Wang, Wengang Zhou, Houqiang Li, and Qi Tian, Contextual similarity distillation for asym- metrie image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9489-99 949999998,4 2022b.
Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang. Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto. Positive-congruent training: Towards regression-free model updates. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14299-14308, 2021.
Min Yang, Dongliang He, Miao Fan, Baorong Shi, Xuetong Xue, Fu Li, Errui Ding, and Jizhou Huang. Dolg: Single-stage image retrieval with deep orthogonal fusion of local and global features. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 11772-11781, 2021.
Binjie Zhang, Yixiao Ge, Yantao Shen, Yu Li, Chun Yuan, Xuyuan Xu, Yexin Wang, and Ying Shan. Hot- refresh model upgrades with rngression-alleviainn compatible training in image retrieval. In International Conference on Learning Representations (ICLR), 2022.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEF Conference on Computer Vision and Patter Recognition (CVPR). pp.
2018
12

---

Published as a conference paper at ICLR 2023
Appendix
In this appendix, we firstly present the pseudo-code our methods, more details about training, testing and model architectures (App. A). Then, conduct more comparisons (App. B) demon- strate the superiority of our methods various settings. Finally, more extended ablations (App. C) conducted for in-depth understanding of our framework (App. D).
Algorithm Pseedo-code of Rank Preserving Framework in PyTorch-like style.
ga leryy_addda: we well-trainen and fixed encoder for gallery set, gradient query_model: lightweight query encoder, with gradient topk: the length of ranking list, & in Equ. qallery_feats: features of the training gallery set, enbedded by qallery model
for in tradei:g load mini-batch with N samples from training set with torca.no_grad()= Symmetric retrieval, Sac. qallery_feat gallery_model fory_modl.. forward tensor shape: NxD gallery_sim, einsum (ataummmb,,,eaeeee ko->bk''''', gallery_feat, gallery_feats) gallery_sim, tonk_index (gallgaalery__ii tepk, dim--1)
Asymmetric retrieval, Sec.
quezy_feat feat meeey_model..oraaa query_sim einsum('be, kc->bk', query_feat, falleryffetts query_sim (queryyyary_sim, dim=-1, topk_index) \bm($)__ shape: Nxtopk
instantiationa the rank preserving framework
loss = taak_order_preservttte (gallerr_ssgle query_sim)
or loss non.meer.sinsllettyy brr.yyy.rrrrrrrtoo..l.ll.......
(gallery_eing quezy_aim)
SGD update: query model
loss baskwaac update
def sigmoid taul:
- 1.0 11.0 exp(-x )
return
def wtt__oorrrrreeeeetttaeeee (sim sim_q): Rank Order Preservation Indicatoz matrix \bm(z) _9, Equ. (8)
sim_g_repeat sim unsqueeze (1) repeat (1, topk, 1) ram_ggrrpet..perruu (0, 2, 1)
sim_g_diff gimeg
pass through the signoid
L_g = sigmoid 0.000001) # tesnor shape: Bxtopkxtopk
rank weights \bm(W]
sim sim_weight (sinna tau_r, dimm-1) tensor shape Axtopk position_weighhe arange (start=0, and=topk) 1.0 4 tensor shape : Bxtopk rack_ (sim_weight, position_weightt unsqueeze(-1)
Indicatoz matrix \bm(I)_q, Equ (8)
sim sim_q repeat sim unsquuezee (1) repeat (1, tepk, sim_g sim_a_repeat rispatt wep.rrr.peerueeeeeeeeeeeeee permute (0, 2, 1)
pass through the sigmoid
T_q = sigmot (sim_a_diif tua) tua: Atua in Equ. (10)
(pow Lg,2) rank_weight, dim= (-1,-2)) .mean
def monetonic_simiiee.... a
sim_q): Monotonic similarity
Preservation sim_g_mapped f(sim_g) f(x) is mapping function. Sea. 4.2
p_g softmax (sip_g_mapeedd tau_g, dim=-1) tensor shape: Bxtopk P_q = softmax (sim_s tau_g, dim=-1), sim=-11,1op (lo()) # tensor shapet 3xtopk
kl_div(p_9- P_g, detuction='mean"
cinstein summation convention; guther values along concatenation; softmax function.
13

---

Published as a conference paper at ICLR 2023
Query Net (-)
Gallery
FLOPs (G)
Param. (M)
Net ox(-)
ABS
%
ABS
Feature extractor
Feature extractor
ResNec101
ResNet101 42.85
100.0
42.50
100.0
H w X
H W C
ShuffleNetV2 (0.5x)
0.84
1.96
2.44
5.74
ShuflleNetV2
1.44
3.36
3.35
7.88
Global pooling
Convolutional layer
MobileNetV2
2.50
5.83
4.85
11.41
C
H W x C,
EfficientNetBO
ResNet101
2.86
6.67
6.63
15.60
EfficientNetBI
3,92
9.15
9.13 21.49
Several FC layers
GeM pooling
EfficientNerB2
4.50
10.51
10.58 24.90
EfficientNe:B3
6.24 14.57 13.84 32.56
1 x C2
1 x C
Classificution layer
Whitening layer
Table 7: The number of parameters and the compu- tational complexity (in FLOPs) of query models used in this work. To adapt for image retrieval, all layers except feature extractor are removed. A convolutional layer is added the feature extractor for matching the Figure 6: Model architecture comparison
1 x C
1 x C
(a) Classification
(b) Image retrieval
output dimension of the gallery model. (0.5x) denotes of classification and image retrieval task.
a model with 0.5x width.
A IMPLEMENTATION DETAILS
In Alg. 1, we demonstrate the pseudo-code of our methods. All the training and evaluation datasets publicly available, and the splits have been carefully described in Sec. 5. Training and testing details are present at App. A.1.
A.1 EXPERIMENT SETTINGS
Training Details. When SfM-120K is adopted for training, we follow the common settings in AML (Budnik & Avrithis, 2021). Training images are resized with the maximum side equal to 512, keeping the aspect ratio. No data augmentation is adopted. Training epochs and batch size are set as 10 and 64, respectively. The query model is trained on one NVIDIA RTX 3090 GPU. When using GLDv2 as the training set, 512 512 pixels are center cropped from the randomly resized image. Random color jittering is adopted as the data augmentation. We train the query model on NVIDIA RTX 3090 GPUs for 10 epochs with batch size of 256. All models are optimized us- ing Adam with an initial learning rate of 10-3 and weight decay of 10-6. A linearly decaying scheduler is adopted to gradually decay the learning rate to when the desired number of steps is reached. When query model is trained with Rank Order Preservation (Sec. 4.1), the length K of ranking list is set to 4096, and the temperature coefficient 5, in ranking weight W; is set as 0.2. As for Monotonic Similarity Preservation (Sec. 4.2), K is also set 4096, and both Ty and Ty are set to 0.1. The gallery set G., used during training, is the same as training dataset in all cases.
Details about Testing Datasets. GLDv2-Test contains 761, 757 gallery images and 750 queries, with evaluation metric of mAP@100. As for ROxf and Rarr there are 70 queries for both of them, with 4, 993 and 5, 007 gallery images, respectively. mAP on the Medium (Med) and Hard settings are reported. Large-scale results are reported with the RIM (Radenović et al., 2018a) 8a) dataset added, which contains IM distractor images. The last corpus INSTRE (Wang & Jiang, 2015), which contains various everyday 3D or planar objects from buildings to book covers in natural scenes. We follow the data partitioning proposed by Iscen al. (2017) with 250 queries and 27, 293 gallery images. mAP is reported with another 1M distractor images D1M added into the gallery or not.
Model Architecture Details. In Fig. 6, we show the typical differences in the model architecture for the classification task and image retrieval task. To obtain better transfer performance, most of the models are pre-trained on ImageNet (Deng et al., 2009) with category labels. These models usually consist of a feature extractor, global mean pooling, several fully-connected layers, and classification layer. To adapt these models for the image retrieval task, we keep only the feature ex- tractor and discard the other layers. After that, GeM pooling (Radenović et al., 2018 (b))) is adopted for aggregating the feature map output by the feature extractor. Finally, a whitening layer, implemented by a fully-connected layer, adopted obtain the final global feature. Notably, the whitening layer
14

---

Published as a conference paper at ICLR 2023
Query NET (,(+)
Gallery NET (,(+)
ROxf
ROxf RIM Medium
Rarr
R
R 1M
Method
Medium
Hard
Hard
Medium
Hard
Medium
Hard
Performances of the gallery models
GeM (Radetovi et al, 20186)
Eb3
Eb3
54.22
27.53
37.10
17.49
71.21
48.00
44.67
18.45
GeM (Radenosit et al. 20180
R101
R101
65.43
40.13
45.23
19.92
76.75
55.24
52.34
24.77
DELG (Caoettl al, 2020)
Eb3
Eb3
66.64
43.82
49.67
24.89
81.78
63.90
61.10
32.34
DELG (Caoett al, 20200
R101
R101
78.55
60.89
66.02
41.75
88.58
76.05
73.65
51.46
Training with GeM as gallery model and SJM- as training sel
Contr (Rodsik & Acrihis, 20213
45,20
19.60
24.70
12.20
63.70
40.90
32.80
12.50
Reg Avritis, 2021)
52.90
27.80
29.70
10.40
65.20
42.40
39.00
16.00
CSD (Wi 21022))
65.16
38.62
43.05
18.81
75.94
53.05
46.76
19.43
Ours
Eb3
R101
65.44
39.36
45.33
21.50
76.35
54.77
52.07
24.10
(mAP gains over CSD)
(1 0.28)
(1 0.74)
(1 2.28)
(1 2.69)
(1 0,41)
(1 1.72)
(1 5,31)
(1 4.67)
Ours
66.26
39.83
46,20
21.39
76.33
53.68
50.65
22.97
(mAP gains over CSD)
(1 1.10) (1 1.21) (1 3.15)
(1 2,58)
(1 0.39)
(1 0.53)
(1 3.89)
(1 3.54)
Training with DELG gallery model and GLDv2 training
Reg (Budnik Avritiis, 2021)
74.60
53.41
59.88
33.31
83.81
68.15
59.36
35.24
HVS (Drggal 2001)
76.41
56.13
62.72
36.86
87.07
74.53
71.54
49.09
LCE (Meng 2021)
75.89
55.21
61.90
36.53
86.63
73.62
70.98
48.94
CSD (Wu 20222)
77.44
58.97
63.21
38.20
87.94
75.68
73.37
50.09
Eb3
RI01
Ours
80.23
62.60
65.40
40.93
89.35
77.35
73.45
50.33
(mAP gains over CSD)
(1 2.79)
(1 3.63)
(1 2.19)
(1 2.73)
(1 1.41)
(1 1.67)
(T 0.08)
(1 0.24)
Ours
80.60
62,45
66.21
40.86
89.36
76.78
74.24
$1.05
(mAP gains over CSD)
(1 3.16) (1 3.48) (1 3.00)
(1 2.66)
(1 1.42)
(1 1.10) (1 0.87) (1 0.94)
Table 8: mAP (asymmetric retrieval) comparison to the state-of-the-an methods. DELG and GeM are trained with SIM-120K and GLDv2, respectively. t: gallery models are the same as compar- ison methods; R101: ResNet101; Eb3: EfficientNetB3.. Black bold denotes the best performance.
pre-trained in the embedding space of the gallery model and remains fixed during the training of the query model. In Tab. 7, we show the number of parameters and computational complexity (in FLOPS) of different networks, all modified for the retrieval task, when the size of the input image is 362 x 362. Compared with large models, lightweight models significantly reduce the computation during inference phase. Thus, they can be used in various resource-constrained scenarios.
Testing Details. During testing, as for ROxf and RPar datasets, we resize images so that the larger dimension is equal to 1024 pixels and preserve the aspect ratio. Besides, the image features are extracted at three scales, i.e., (1/v/2, 1, /2). We perform L2 normalization for each scale indepen- dently and the features of three scales are averaged, followed by another L2 normalization. Under the asymmetric retrieval setting. queries are embedded with the lightweight query model dy(-), while the gallery images are embedded by large model (g(-).
B EXTENDED COMPARISONS
B. DIFFERENT QUERY MODEL
In this section, we perform more extensive comparisons with existing methods. Specifically, in Tab. 8, we take EfficientNet33 as the lightweight query model. The performance of asymmetric retrieval improves as the capacity of the query model becomes larger (compared with the accuracy of MobileNetV2 in Tab. 6). Our proposed methods achieve optimal performance in various settings, with the performance of asymmetric retrieval being almost comparable to that of symmetric retrieval.
B.2 DIFFERENT GALLERY MODEL
In Tab. 9, we further extend the experiments in Fig. 5 in the main paper. Three recent deep repre- sentation models are adopted to embed gallery images. (1) Token (Wu et al., 2022a) based on local feature aggregation. (2) CVNet (Lee et al., 2022) based on global pooling, (3) DOLG (Yang et al., 2021) based on local and global feature fusion. Our methods achieve the best performance across various settings, which demonstrates the generalization of them.
15

---

Published as a conference paper at ICLR 2023
Query NET 2g(-)
Gallery NET (x()
ROxf
ROxf RIM
RPar
RPar R.M
Method
Medium Hard
Medium Hard
Medium Hard Medium Hard
Performances of the gallery models
Token* (Wi 2022g
82.16
65.75
70.58
47.46
89.40
78.44
77.24
56.81
CVNet
RIO1
R1O1
80.01
62.83
74.25
54.56
90.18
79.01
80.82
62.74
DOLG (ring atal. 2021)
82.37
64.94
75.19
53.55
90.97
81.71
82.28
66.45
Traing with Token gallery model
Reg' (Budbik Avriltis, 2023)
71.67
50.92
59.55
35.41
83.26
69.48
63.98
42.93
HVS (Duggal
73.16
53.19
57.68
35.52
84,27
71.43
62.90
41.00
LCB (Meng et,l. 2021)
Mv2
R101
74.82
56.40
61.57
39.43
84.09
71.88
65.70
43.86
CSD (Wu et 20226)
75.52
56.83
63.46
39.01
84.50
70.73
65.93
43.77
Ours
77.22
57.33
65.87
40.97
85.51
71.95
67.88
44.87
Ours
76.99
57.35
66.00
41.24
86.35
73.69
70.37
48.89
Training with CVNet as gallery model
Reg (Boddik Aorthis, 2021
74.60
53.41
59.88
33.31
83.81
68.15 15
59.36
35.24
HVS (Dgggal et st. 2021)
74.90
55.17
62.32
42.66
84.92
71.62
67.13
46.80
LCE
(Meng 2021)
Mv2
R101
75.95
57.87
63.77
43.37
83.66
69.71
66.44
46.72
CSD
20226)
76.44
58.41
64.42
43.90
85.32
71.61
68.32
47.76
Ours
76.62
59.48
66.49
46.26
86.48
73.58
72.89
53.11
Ours
77.92
60.83
68.68
47.04
87.81
75.62
75.16
55.97
Training with DOLG gallery model
Reg!
Avcibis 2023)
69.04
48.00
56.81
35.51
79.13
63.13
60.22
42.59
HVS (Dugasl et 2021)
72.79
54.20
63.29
41.74
85.18
70.72
68.13
48.25
LCB (Meng 2001)
Mv2
R101
72.84
53.70
61.90
40.84
85.77
69.54
67.65
48.53
CSD (Woun ol., 20226)
75.53
56.23
64.02
42.79
86.34
72.84
69.29
49.47
Ours Ours
77.67
59.87
67.40
44.93
87.19
73.64
70.91
52.38
78.46
60.74
66.98
43.84
87.81
74.72
72.18
53.63
Table 9: Extend mAP (asymmetric retrieval) comparison the state-of-the-art methods with
different gallery models. All the models are trained with GLDv2. f; re-evaluate the official public weights; 1; our re-implementation. RI01: ResNet101; Mv2: MobileNetV2. Black bold denotes the best performance.
C ADDITIONAL ABLATIONS
C.1 IMPACT OF DISTANCE TYPE
In Rank Order Preservation (Sec. 4.1), the weighted mean square error is token as the final objec- tive function train the query model end to end. the section, we explore an alternative measure the distance between the Indicator matrices 1 and I" Specifically, we convert each row of I and 1a into the form of probability distribution:
exp (IZ,/Tg)
cxp (Ii i 1,2, K,
P
(12)
Σ exp (1./TT)
Σ exp [1//Tgg
where Ty and Ty are the temperatures. Then, the distance between indicator matrices is defined as the KI. divergence of two distributions:
K K
P pd )
CHOP
Σ
W.P.
log(
(13)
As for Monotonic Similarity Preservation (Sec. 4.2), we try to use the Lg distance to measure the inconsistency between the mapped similarity scores M, and the asymmetric similarity scores Sg. Then, Eq. (11) is defined as
K LMPP MM, - --1 Σ (M - S.
(14)
1-1
16

---

Published as a conference paper at ICLR 2023
Gallery Net (.)))
INSTRE
ROxf
RIM
RPar
R1M
Loss function GLDv2-Test
+ 2
+ DIM
Med
Hard
Med
Hard
Eq. (12)
13.33
59.83
50.98
42.49
19.99
48.05
20.15
GeM
Eq. (9)
12.35
60.81
50.96
42.33
19.78
47.72
20.34
Eq. (12) Eq. (9)
24.92
35.67
25.05
64.23
39.65
70.90
48.18
DELG
23.86
29.93
21.76
64.34
39.19
72.00
48.47
(a) Query model (.()) is trained by Rank Order Preservation (Sec. 4.1).
Gallery
INSTRE
ROxf + RIM RPar + RIM
Loss funetion GLDv2-Test
Net (-)
+ Ø + DIM Med
Hard
Med
Hard
Eq. (14)
12.79
60.95
52.48
43.00
20.83
48.46
20.63
GcM
Eq. (11)
13.33
60.86
49.72
43.47
20.20
47.56
20.33
Eq. (14)
25.02
35.21
27.50
64.75
42.18
71.13
48.74
DELG
Eq. (11)
25.44
34.34
26.58
64.18
40.19
71.04
48.65
(b) Query model (.(+) is trained by Monotonic Similarity Preservation (Sec. 4.2).
Table 10: mAP (asymmetric retrieval) comparison of different distance types. S1M-1206 and GLDv2 are adopted for training the query model when GeM and DELG serve as the gallery model, respectively. MobileNetV2 is adopted as the query model.
Map func.
SIM-120K
INSTRE
ROxf + RiMRPar RIM Hard
Med
Hard
Med
Map fune.
SfM-120K
INSTRE
ROxf
RIMPPar RIM Med
val
Med
Hard
Hard
Logarithmic function
log (2z
46.13
Polynomial function
10 +
78.12
60.19
42.84
19.59
47.83
19.89
75.65
54.55
39.88
18.08
44.87
18.34
log,(1 +-1)
78.94
60.61
44.24
20.93
48.13
21.45
78.05
58.21
42.13
19.76
46.74
19.56
log,(a
77.98
60.05
42.60
19.61
48.49
20.38
78.49
60.86
43.47
20.20
47.56
20.33
1log 1)
78.24
60.49
43.01
20.51
48.52
20.69
Expomential funetton
76.19
56.39
38.27
17.40
43.92
17.76
6F-1
76.59
61.17
41.79
19.32
46.29
19.39
77.47
58.05
42.41
19.31
46.16
19.85
77.55
62.21
43.08
19.77
47.78
19.89
78.07
59.17
42.20
19.45
47.84
20.34
10--1
77.83
62.69
43.32
20.51
47.94
20.66
76.63
53.99
40.25
17.76
44.23
17.62
12
77.61
62.63
43.20
20. 14
47.82
20.48
110.69
77.47
62.15
43.13
19.72
47.76 20.20
77.22
58.33
41.02
18.29
45.93 18.81
77.94
59.02
42.18
19.84 46.43 19.32
Table 11: Ablation on different mapping functions. denotes that the base is learned from the data dis- tribution, otherwise is determined artificially in ad- vance. MobileNetV2 and GeM are adopted as query and gallery models, respectively.
Table 12: Ablation on different mapping functions. Polynomial function is adopted as the mapping func- tion. MobileNetV2 and GeM are adopted as query and gallery models, respectively.
The comparison of different distance type are summarized in Tab. 10, the performance of various loss types are similar, which shows that rank preserving is the key to achieve superior performance rather than a specific consistency loss.
C.2 IMPACT OF THE UPDATE FUNCTION
In Tab. 11 and Tab. 12, show extend version of Tab. in the main paper, where the specific parameter values are given. As for the logarithmic and exponential functions, we also consider manually defining the bases of the mapping functions. As shown in Tab. 11, both learnable bases and manual defined bases lead to similar results. For the polynomial functions, we consider various settings of the basis functions, i.e., [/21/2 - 1tal///99____//////// 1/4 [ Ja-1- Note that the parameters (a1, a2 ... av } in this are all learned from the data distribution. It can be seen that superior performance is achieved with various settings of the basis functions when the order of the basis functions is properly chosen.
C.3 EXTENDED ABLATION ON TRAINING DATASETS
In this section, we study the scalability of our framework. we randomly sample different number of images from GLDv2 dataset for training. As shown in Tab. 13, more training data leads to better
17

---

Published as a conference paper at ICLR 2023
Gallery Net (.(.)
Training
Image
INSTRE
ROxf +
RAM
Ra
RIM
GLDv2-Test Numbers
Set T
Ø + DIM Med
Hard
Med
Hard
SfM-120k
91, 642
12.35
60.81
50.96
39.65
19.78
47.72
20.34
GeM
GLDv2 (x0.1) 128, 078
14.16
59.29
48.75
44.24
20.18
49.64
21.93
GLDv2 (x0.2) 256, 156
14.62
61.06
51.57
45.95
22.34
51.33
23.57
GL.vv2 (x0.4) 512, 312
14.77
62.90
54.00
47.01
23.61
53.16
25.47
StM-120k 91, 642
18.62
23,89
14.49
52.08
30.54
59.63
33.86
GLDv2 (x0.1) 128, 078
23.39
28.64
22.19
61.99
38.87
67.83
46.25
DELG
GL.Dv2 (x0.2) 256, 156
24.14
29.28
22.53
62.31
39.40
68.14
46.80
GI.Dv2 (x0.4) 512, 312
24.79
32.48
25.70
63.23
40.23
69.40
47.21
(a) Query model Φy
) is trained by Rank Order Preservation (Sec. 4.1).
Gallery
Training Set T
Image Numbers
INSTRE
ROxf + R1M Rarr RIM
GLDv2-Test
Net (.)
Ø + DIM Med
Hard
Med
Hard
SfM-1206 120k 91, 642
13.33
60.86
49.72
43.47
20.20
47.56
20.33
GeM
GLDv2 (x0.1)
128, 078
14.13
62.41
53.42
46,88
22.53
51.41
23.48
GL.v2 (x0.2) 256, 156
14,40
63.82
55.32
47.15
23.16
52.96
25.35
GL.vv2 (x0.4) 512, 312
14.86
65.26
57.48
48.90
23.97
53.17
25.91
SfM-120k 91. 642
19.07
25.12
13.35
52.19
28.35
56.19
30.03
GL.v2 (x0.1) 128, 078
24.12
32.42
24.16
60.20
37.72
67.84
45.69
DELG
GL.Dv2 (x0.2) 256, 156
24.27
33.02
25.34
62.31
38.66
68.23
46.41
GLDv2 (x0.4) 512, 312
24.63
33.76
26.33
63.45
39.52
69.47
47.67
(b) Query model (.(+) ) is trained by Monotonic Similarity Preservation (Sec. 4.2).
Table 13: mAP (asymmetric retrieval) comparison of different training dataset size. We randomly sample some images from the original training datasets form the training ,where (xx) denotes the proportion. MobileNetV2 is adopted as the query model (,).)
Gallery Net (-)
Training Set T
ROxf Med
RIM
RPar
RIM
GLDv2-Test
Hard
Med
Hard
GL.vv2 ROxt + RRar' + RIM
27.12
65.87
40.97
67.88
44.87
Token
26.33
66.95
42.26
74.63
53.16
GLDv2 ROxt" RRPr' RIM
28.68
66.49
46.26
72.89
53.11
CVNet
26.39
69.58
48.39
78.32
59.22
GLDv2
25.16
67.40
44.93
70.91
52.38
DOLG ROxt + RPar" + RIM
24.73
68.83
46.41
75.21
55.2
(a) Query model (.(-) is trained by Rank Order Preservation (Sec. 4.1).
Gallery
Training Set T
ROxf + R1M
RPar R.M
GLDv2-Test
Net (-)
Med
Hard
Med
Hard
GLDv2 ROxt'
27.55
66.00
41.24
70.37
48.89
Token
RPar' RIM
26.17
66.37
41.89
73.85
52.28
GLDv2 ROxf
28.57
68.68
47.04
75.16
55.97
CVNet
R.Pr' RIM
26.05
68.51
47.08
78.50
59.26
GLDv2
26.25
66.98
43.84
72.18
53.63
DOLG
ROxt"
RPar RIM
24.96
69.11
44.69
77.34
59.14
(b) Query model (-) is trained by Monotonic Similarity Sreservation (Sec. 4.2).
Table 14: Extended experiments about training on the deployed gallery set. The query sets for all test sets are not involved training. denotes the gallery set of that testing dataset. MobileNetV22 is adopted query models.
performance across all settings. In Tab. 13, conduct more experiments with unlabeled gallery set for training. When evaluating on a deployed database, training on it is always better than on collected training dataset, which is also confirmed by Tab. 14 in the main paper.
18

---

Published as a conference paper at ICLR 2023
R0xf R.IM (Med)
ROxf+ + R.M (Hard)
R.Par R.IM (Med)
RPar R.IM (Hard)
50
50
40
20
20
mAP(%)
30
mAP(%
15
mAP(%)
40
mAP(
Ouns
-- Ours
30
Ours
Ours!
Ours
Our
Ours
Oars
10
20
30
10
20
30
10
20
30
10
20
FLOPs (G)
FLOPs
(G)
OLOPs (G)
FLOPs (G)
Figure 7: Performance versus inference computation complexity when varying the size of input image. MohileNetV2 and GeM adopted as query and gallery models, respectively. Query features are extracted as
the original single scale. The x-axis represents the average FLOPs (G) for five inferences, which proportional
to the size of the image. We resize the queries to (0.2, 0.4, 0.6, 1/ 2/ 0.8, 1.0, v2) of the original size (1024 x
768).
D ANALYSIS AND DISCUSSIONS
D.1 INFERENCE COMPUTATION
During testing, we follow the common settings to use multi-scale feature extraction, which greatly aggravates the inference computation of the query model. The inference computation is mainly re- lated to the complexity of the model and the size of the test image. In this section, MobileNetV2 adopted as query model. We use single-scale feature extraction and vary the image size to study the relationship between inference computation and asymmetric retrieval accuracy. The results shown in Fig. 7, the asymmetric retrieval accuracy increases and saturates as the inference computa- tion increases. In practical application scenario, should choose the appropriate test image size to achieve a balance between efficiency and accuracy.
Gallery Net (.).)
INSTRE + DØ + DIM Med
ROxf + RIM RPar +
R1M Hard
Loss function
GLDv2-Test
Hard
Med
Eq.(
13.33
60.86
49.72
43.47
20.20
47.56
20.33
GeM
(9)
12.35
60.81
50.96
42.33
19.78
47.72
20.34
A (9) ((1)
12.71
61.56
53.36
43.27
20.55
48.86
20.91
(1.(
25.44
34.34
26.58
64.18
40.19
71.04
48.65
DELG
(q)
23.86
29.93
21.76
64.34
39.19
72.00
48.47
A (9) (11)
24.84
34.84
26.56
64.56
40.79
71.64
48.02
Ablation on the combination of different training losses. SfM-120k and GLDv2 are adopted for training the query model when GeM and DELG serve as the gallery model, respectively. MobileNetV2 is adopted as the query model.
Table 15:
D.2 COMBINE TWO INSTANTIATIONS
In this section, we try to combine the two instantiation methods proposed in the main paper. As shown in Tab. 15, the simple combination fails to bring further performance improvement. We believe this is due to the fact that the optimization goals of both methods are to maintain the order of the images in the returned ranking list. Thus, the final results obtained with methods not complementary to each other, which also confirmed by the distribution of the similarity scores present in Fig. and Fig. 8.
D.3 SIMILARITY SCORE DISTRIBUTION
In this section, we visualize the similarity score distributions in Fig. 8, when the query model is trained by different methods. The first row shows the training process of Rank Order Preserva- tion. The following three rows correspond to the training process of Monotonic Similarity Preser- vation, when three different mapping functions are chosen, respectively. As the training proceeds, the region of the similarity distributions gradually becomes slender, which indicates that the orders of the images in the returned ranking list are better maintained. It can be found that under rank
19

---

Published as a conference paper at ICLR 2023
preserving framework, the different instantiations yield similar results, which is also confirmed by the final retrieval accuracy (Tab. 6).
Epoch
Epoch 2
Epoch 4
Epoch 8
1
ROxf RIM
+Oxt+ + RIM RPPr RIM
+ ROOf + RIM R.Par RIM
+ROO + RIM -RPar RIM
0.8
RParr + R.IM
01.
0.8
0.8
0.6
01.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
80
0.6
(a) (-(-) is trained for ROP Eq. (9).
Epoch
Epoch 2
Epoch 4
Epoch 8
#20ff + RAI RRPar RIM
ROxT RIM RPar RIM
RROf + RAM RRPar RIM
.ROf RIM
0.8
0.8
0.8
0.8
RRPar RRM
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
20
80.
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(b) (-) is trained for MSP Eq. (11) with learnable polynomial mapping function.
Epoch
Epoch 2
Epoch 4
Epoch
1
RROf&& 4 RIM RPar RIM
ROaT RIM R.ur RIM
#ROf + RIM R.Par RIM
ROxf RIM RPar
0.8
0.8
0.8
0.8
RIM
0.6
8
0.6
9.82
0.6
10. 10.38
0.6
10.69%
0.4
0.4
0.4
0.2
0.2
0.2
0.2
80
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(c) (-) is trained for MSP Eq. (11) with learnable exponential mapping function.
Epech
Epoch
Epoch
Epoch
19
ROOf+ + RIM
RIM
0.8
0.
0.6
8g
log 5:1
0.6
8g
log, o92
log, (8,)
0.4
loga
0.4
0.2
20
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(d) (x(-) is trained for MSP Eq. (11) with learnable logarithmic mapping function.
Figure 8: Visualization of the similarity score distributions. SfM-120k, MobileNetV2 and GeM are adopted the training dataset, query and gallery models, respectively. 8y. symmetric similarity score; syi asymmetric similarity score.
20


Reviews:
[{"review_id": "qKNdkD1YDCR", "reviewer": "Reviewer_BuFD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about asymmetric image retrieval, where the query features and the gallery features are extracted with different feature extractors. Gallery features typically use a more expensive feature extractor, whereas the query features are extracted with a more lightweight feature extractor to increase the speed during inference. The authors first compute the ranked list using the gallery features, and  add a consistency loss term such that the same ordering will be preserved when using the query features. Two solutions are proposed for preserving the ranked list ordering between different sets of features. The experiments are thorough and show that the proposed method brings improvements.", "summary_of_the_review": "This paper proposes a novel method for preserving the ranked list for asymmetric image retrieval. The paper is easy to read, the proposed method looks correct, and the experiments confirm the claims made in the paper. ", "strength_and_weaknesses": "Strenghts:\n- The paper is easy to read.\n- The proposed method is well motivated, and makes sense for this problem.\n- The authors justify their claims with thorough experiments and analysis.\n\nWeaknesses:\n- There are grammatical mistakes/typos in the paper that should be fixed, e.g. \"we aims\" in Page 3, \"the top-K images are token\nfrom F\" (should be taken) etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aK-q5F9rbx", "reviewer": "Reviewer_BchT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The method addresses the problem of asymmetric image retrieval, in which different models are deployed for query and gallery - usally for computation efficiency reasons - and that consists on aligning their embedding spaces. The authors argue that current approaches, which try to enforce the consitency of features between models, are too limiting since this is too strict for lightweight query models with low capacity. To address this issue, instead of forcing the query model to produce the same representations as the gallery, the authors relax this problem and propose to optimize instead the consitency of the rank order. For this, they propose two methods that aim to achieve this rank consistency between two rank orders.", "summary_of_the_review": "Even though I think that a deeper analysis comparing the methods proposed would have made the paper more well-rounded, it is already a very strong paper with a very strong contribution presented with a very clear motivation. Other than this weakness just mentioned, the experimental analysis is also quite extensive. Taking all this into account, I would recommend this for its acceptance.", "strength_and_weaknesses": "The paper is well written and the method very well motivated by presenting a possible limitation of current methods (enforcing consistency of feature spaces) and proposing a direct solution to this problem by optimizing the consistency of the rank order instead, which by itself I think it is a very good idea. Then, figures and the explanation of their framework and methods are also very clear. Finally, experiments are extensive and includes a very useful ablation analysis.\n\nThere is however something that I would have liked to see in the paper, which is a deeper insight of the differences between the two methods for rank preserving proposed. For example, both methods are on par when using Sfm120K as training set but when using GLv2, MSP significantly outperforms ROP: why does this happen? Is there any scenario where one would choose ROP over MSP? It would have been interesting to see a deeper comparison between both methods.\n\nA couple of typos I spotted:\n- Page 3, last paragraph: we aims to learn -> we aim to learn\n- Page 4, first paragraph: images are token -> images are taken\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PktuMXwMIS", "reviewer": "Reviewer_6hUg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for asymmetric image retrieval. The main goal is to utilize a lightweight CNN without affecting the retrieval performance. A sizeable pretrained model is employed to extract features from the dataset, and a ranking list is assembled by retrieving the most similar images to the query image. Then, the asymmetric similarity scores are calculated using the features of the images from the ranking list and the features of the query image from the lightweight model. Finally, to ensure that the original ranking is preserved, two different techniques are proposed: Rank Order Preservation and Monotonic Similarity Preservation. The first technique optimizes the consistency of rank, while the second uses a learnable function to map the similarity scores of both models.", "summary_of_the_review": "Pros:\nThe empirical findings support the efficacy of the proposed solution. The ablation study and the extra experiments depicted in the appendix aid in understanding the core concepts of the algorithm.\nThe paper is well-written, and the premise is original and novel.\n\nMinor remarks:\nThe metric used for ranking the gallery images is not denoted.\nProof that the similarity scores of the gallery satisfy the monotonically decreasing property could be useful.\nMinor typos, especially in section 4.", "strength_and_weaknesses": "Pros:\nThe empirical findings support the efficacy of the proposed solution. The ablation study and the extra experiments depicted in the appendix aid in understanding the core concepts of the algorithm.\nThe paper is well-written, and the premise is original and novel.\n\nMinor remarks:\nThe metric used for ranking the gallery images is not denoted.\nProof that the similarity scores of the gallery satisfy the monotonically decreasing property could be useful.\nMinor typos, especially in section 4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DQDR9y9imvw", "reviewer": "Reviewer_vzrF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a generic rank preserving framework for asymmetric image retrieval. The rank preserving framework allows feature compatibility and order consistency between a query model and a gallery model simultaneously. Two methods are proposed as part of the rank preserving framework: i) the first realizes rank order preservation by directly preserving the consistency of the sorting results using a sigmoid function approximation, and ii) the second learns a monotonic mapping function between the returned similarity scores of query and gallery models. An evaluation of the proposed framework on 4 public datasets showed that that this framework outperforms significantly some state of the art methods.\n", "summary_of_the_review": "This paper proposed a new and generic rank preserving framework for asymmetric image retrieval. The proposed framework doesn't require a labeled dataset and optimizes directly the consistency of rank order between the query model and the gallery model. Experimental results on 4 public datasets showed that the proposed framework outperforms state of the art methods. Therefore, the proposed framework looks solid and is novel enough.\n\n", "strength_and_weaknesses": "Strengths of this paper:\n- The proposed framework is more flexible than existing methods for asymmetric image retrieval, and doesn't require the query model to mimic the feature embeddings or the overall neighbor structures of the gallery model.\n- This framework utilizes no annotation or labels during the training process, instead it directly optimizes the consistency of rank order between the query model and the gallery model.\n- The paper proposes a comprehensive evaluation on 4 public datasets and the proposed framework showed to outperform state of the art methods by a significant margin.\n\nAreas where the paper could improve on:\n- The query model can only be as good as the gallery model is since no labels are used (depends on the quality of the gallery model). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster