Robust and Decomposable Average Precision for Image Retrieval

1 Introduction . The task of ‘ query by example ’ is a major prediction problem , which consists in learning a similarity function able to properly rank all the instances in a retrieval set according to their relevance to the query , such that relevant items have the largest similarity . In computer vision , it drives several major applications , e.g . content-based image retrieval , face recognition or person re-identification . Such tasks are usually evaluated with rank-based metrics , e.g . Recall @ k , Normalized Discounted Cumulative Gain ( NDCG ) , and Average Precision ( AP ) . AP is also the de facto metric used in several vision tasks implying a large imbalance between positive and negative samples , e.g . object detection . In this paper , we address the problem of direct AP training with stochastic gradient-based optimization , e.g . using deep neural networks , which poses two major challenges . Firstly , the AP loss LAP = 1− AP is not differentiable and is thus not directly amenable to gradientbased optimization . There has been a rich literature for providing smooth and upper bound surrogate 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . losses for LAP [ 43 , 21 , 22 , 6 , 25 ] . More recently , smooth differentiable rank approximations have been proposed [ 35 , 14 , 15 , 3 , 27 , 8 , 2 ] , but generally lose the important LAP upper bound property . The second important issue of AP optimization relates to its non-decomposability : LBAP averaged over batches underestimates LAP on the whole training dataset , which we refer as the decomposability gap . In image retrieval , the attempts to circumvent the problem involve ad hoc methods based on batch sampling strategies [ 10 , 32 , 20 , 32 , 30 ] , or storing all training representations/scores [ 39 , 3 , 27 , 25 ] , leading to complex models with a large computation and memory overhead . In this paper , we introduce a method for RObust And DecoMposable Average Precision ( ROADMAP ) , which explicitly addresses the aforementioned challenges of AP optimization . Our first contribution is to propose a new surrogate loss LSupAP for LAP . In particular , we introduce a smooth approximation of the rank function , with a different behaviour for positive and negative examples . By this design , LSupAP provides an upper bound of LAP , and always back-propagates gradients when the correct ranking is not satisfied . These two features illustrated in the the toy example on Figure 1a are not fulfilled by binning approaches [ 3 , 27 ] or by SmoothAP [ 2 ] . As a second contribution , we propose to improve the non-decomposability in AP training . To this end , we introduce a simple yet effective training objective Lcalibr. , which calibrates the scores among different batches by controlling the absolute value of positive and negative samples . We provide a theoretical analysis showing that Lcalibr . decreases the decomposability gap . Figure 1 illustrates how Lcalibr . can be leveraged to improve the overall ranking . We provide a thorough experimental validation including three standard image retrieval datasets and show that ROADMAP outperforms state-of-the-art methods . We also report the large and consistent gain compared to rank/AP approximation baselines , and we highlight in the ablation studies the importance of our two contributions . Finally , ROADMAP does not entail any memory or computation overhead and remains competitive even with small batches . 2 Related work . We discuss here the literature in image retrieval dedicated to AP optimization , and compare to other approaches based on optimizing representations [ 23 , 1 , 44 , 46 , 33 ] in the experiments . Smooth AP approximations Studying smooth surrogate losses for AP has a long history . The widely used surrogate for retrieval is to consider constraints based on pairs [ 41 , 12 , 26 ] , triplets [ 11 ] , quadruplets [ 18 ] or n-uplets [ 30 ] to enforce partial ranking . These metric learning methods optimize a very coarse upper bound on AP and need complex post-processing and tricks to be effective . One option for training with AP is to design smooth upper bounds on the AP loss . Seminal works are based on structural SVMs [ 43 , 21 ] , with extensions to speed-up the `` loss-augmented inference '' [ 22 ] or to adapt to weak supervision [ 6 ] . Recently , a generic blackbox combinatorial solver has been introduced [ 25 ] and applied to AP optimization [ 28 ] . To overcome the brittleness of AP with respect to small score variations , an ad hoc perturbation is applied to positive and negative scores during training . These methods provide elegant AP upper bounds , but generally are coarse AP approximations . Other approaches rely on designing smooth approximations of the the rank function . This is done in soft-binning techniques [ 14 , 15 , 35 , 3 , 27 ] by using a smoothed discretization of similarity scores . Other approaches rely on explicitly approximating the non-differentiable rank functions using neural networks [ 8 ] , or with a sum of sigmoid functions in the recent SmoothAP approach [ 2 ] . These approaches enable accurate AP approximations by providing tight and smooth approximations of the rank function . However , they do not guarantee that the resulting loss is an AP loss upper bound . The LSupAP introduced in this work is based on a smooth approximation of the rank function leading to an upper bound on the AP loss , making our approach both accurate and robust . Decomposability in AP optimization Batch training is mandatory in deep learning . However , the non-decomposability of AP is a severe issue , since it yields an inconsistent AP gradient estimator . Non-decomposability is related to sampling informative constraints in simple AP surrogates , e.g . triplet losses , since the constraints ’ cardinality on the whole training set is prohibitive . This has been addressed by efficient batch sampling [ 13 , 10 , 32 ] or selecting informative constraints within mini-batches [ 30 , 9 , 4 , 32 ] . In cross-batch memory technique [ 39 ] , the authors assume a slow drift in learned representations to store them and compute global mining in pair-based deep metric learning . In AP optimization , the non-decomposability has essentially been addressed by a brute force increase of the batch size [ 3 , 27 , 25 ] . This includes an important overhead in computation and memory , generally involving a two-step approach for first computing the AP loss and subsequently re-computing activations and back-propagating gradients . In contrast , our loss Lcalibr . does not add any overhead and enables good performances for AP optimization even with small batches . 3 Robust and decomposable AP training . We present here our method for RObust And DecoMposable AP ( ROADMAP ) dedicated to direct optimization of a smooth surrogate of AP with stochastic gradient descent ( SGD ) , see Fig . 2 . Training context Let us consider a retrieval set Ω = { xj } j∈J1 ; NK composed of N elements , and a set of M queries included in Ω , i.e . Q = { qi } i∈J1 ; MK ⊆ Ω . For each query qi , each element in Ω is assigned a label y ( xj , qi ) ∈ { +1 ; −1 } , such that y ( xj , qi ) = 1 ( resp . y ( xj , qi ) = −1 ) if xj is relevant ( resp . irrelevant ) with respect to qi . This defines a query-dependent partitioning of Ω such that Ω = Pi ∪Ni , where Pi : = { xj ∈ Ω|y ( xj , qi ) = +1 } and Ni : = { xj ∈ Ω|y ( xj , qi ) = −1 } . For each xj ∈ Ω , we define a prediction model parametrized by parameters θ , e.g . a deep neural network , which provides a vectorial embedding vqi ∈ Rd of each element , i.e . : vqi : = fθ ( qi ) . In the embedded space Rd , we compute a similarity score between each query qi and each element in Ω , e.g . by using the cosine similarity : s ( qi , xj ) = vqi Tvj ||vqi ||2||vj||2 . During training , our goal is to optimize , for each query qi , the model parameters θ such that positive elements are ranked before negatives . More precisely , we aim at minimizing the AP loss LAPi for each query qi in the retrieval set Ω . Our overall AP loss LAP is averaged over all queries : LAP ( θ ) = 1− 1 M M∑ i=1 APi ( θ ) , APi ( θ ) = 1 |Pi| ∑ k∈Pi Pre ( k , θ ) = 1 |Pi| ∑ k∈Pi rank+ ( k , θ ) rank ( k , θ ) ( 1 ) where Pre ( k , θ ) is the precision for the kth positive example xk , rank+ ( k , θ ) its rank among positives Pi , and the rank ( k , θ ) its rank over Ω = Pi ∪Ni . As previously mentioned , there are two main challenges with SGD optimization of AP in Eq . ( 1 ) : i ) AP ( θ ) is not differentiable with respect to θ , and ii ) AP does not linearly decompose into batches . ROADMAP addresses both issues : we introduce the robust differentiable LSupAP surrogate ( Section 3.1 ) , and add the Lcalibr . loss ( Section 3.2 ) to improve AP decomposability . Our final loss LROADMAP is a linear combination of LSupAP and Lcalibr. , weighted by the hyperparameter λ : LROADMAP ( θ ) = ( 1− λ ) · LSupAP ( θ ) + λ · Lcalibr . ( θ ) ( 2 ) 3.1 Robustness in smooth rank approximation . The non-differentiablity in Eq ( 1 ) comes from the ranking operator , which can be viewed as counting the number of instances that have a similarity score greater than the considered instance , i.e.1 : rank+ ( k ) = 1 + ∑ j∈Pi\ { k } H ( sj − sk ) , where H ( t ) = { 1 if t ≥ 0 0 otherwise rank ( k ) = rank+ ( k ) + ∑ j∈Ni H ( sj − sk ) = rank+ ( k ) + rank− ( k ) ( 3 ) From Eq . ( 3 ) it becomes clear that the non-differentiablity is due to the Heaviside ( step ) function H , whose derivative is either zero or undefined . Note that the computation of rank+ ( k ) and rank− ( k ) in Eq . ( 3 ) relates to the rank of positive instances xk ∈ Pi : the score sk in Eq . ( 3 ) is always the score of a positive , whereas sj can either be a negative ’ s or positive ’ s score . Smooth loss LSupAP To provide a smooth approximation of LAP in Eq . ( 1 ) , we introduce a smooth approximation of the rank function . In particular , we propose a different behaviour between rank+ ( k ) and rank− ( k ) in Eq . ( 3 ) by defining two functions H+ and H− . For rank+ ( k ) , we choose to keep the Heaviside ( step ) function , i.e . H+ = H ( see Fig . 3a ) , which consists in ignoring rank+ ( k ) in gradient-based AP optimization . This is done on purpose since ∂AP ∂ rank+ ( k ) = rank − ( k ) ( rank+ ( k ) +rank− ( k ) ) 2 ≥ 0 : the gradient would tend to increase rank+ ( k ) and to decrease the score of sk . Reminding xk is always a positive instance , this behaviour is undesirable . For rank− ( k ) , we define the following smooth surrogate H− for H , shown in Fig 3b : H− ( t ) = σ ( tτ ) if t ≤ 0 , where σ is the sigmoid function ( Fig . 3c ) σ ( tτ ) + 0.5 if t ∈ [ 0 ; δ ] with δ ≥ 0 ρ · ( t− δ ) + σ ( δτ ) + 0.5 if t > δ ( 4 ) where τ and ρ are hyperparameters , and δ is defined such that the sigmoidal part of H− reaches the saturation regime and is fixed for the rest of the paper ( see supplementary Sec . A ) . From 1For the sake of readability we drop in the following the dependence on θ for the rank , i.e . rank ( k ) : = rank ( k , θ ) and on the query for the similarity , i.e . sj : = s ( qi , xj ) . the H− smooth approximation defined in Eq . ( 4 ) , we obtain the following smooth approximation rank−s ( k ) = ∑ j∈Ni H− ( sj − sk ) , leading to the following smooth AP loss approximation : LSupAP ( θ ) = 1− 1 M M∑ i=1 1 |Pi| ∑ k∈Pi rank+ ( k ) rank+ ( k ) + rank−s ( k ) ( 5 ) LSupAP in Eq . ( 5 ) fulfills two main features for AP optimization : I 1 LSupAP is an upper bound of LAP in Eq . ( 1 ) . Since H− in Eq . ( 4 ) is an upper bound of a step function ( Fig 3b ) , it is easy to see that LSupAP ≥ LAP . This is a very important property , since it ensures that the model keeps training until the correct ranking is obtained . It is worth noting that existing smooth rank approximations in the literature [ 35 , 3 , 27 , 2 ] do not fulfill this property . I 2 LSupAP brings training gradients until the correct ranking plus a margin is fulfilled . When the ranking is incorrect , the negative xj is ranked before the positive xk , thus sj > sk and H− ( sj − sk ) in Eq . ( 4 ) has a non-null derivative . We use a sigmoid to have a large gradient when sj − sk is small . To overcome vanishing gradients of the sigmoid for large values sj − sk , we use a linear function ensuring constant ρ derivative . When the ranking is correct ( sj < sk ) , we enforce robustness by imposing a margin parametrized by τ ( sigmoid in Eq . ( 4 ) ) . This margin overcomes the brittleness of rank losses , which vanish as soon as the ranking is correct [ 14 , 3 , 25 ] . Comparison to SmoothAP [ 2 ] LSupAP differs from LSmoothAP in [ 2 ] by i ) providing an upper bound on LAP , ii ) improving the gradient flow ( Fig . 3b vs Fig . 3c ) , and iii ) overcoming adverse effects of the sigmoid for rank+ , as shown in Fig . 1a ( and in supplementary sec . A ) . We experimentally verify the consistent gain brought out by LSupAP over LSmoothAP .

Summary: This work tackles the problem of large-scale fine-grained image retrieval by proposing ROADMAP, short for ROBust And DecoMposable Average Precision which is a loss function that consists of two components 1) A smooth approximation to the non-differentiable average precision (AP) loss which provides an upper bound and better gradient flow compared to existing variants of the loss. 2) A calibration component which tackles the problem of mismatch between in-batch and true AP values commonly seen in previous methods.  The authors provided thorough theoretical proofs to support their claims on the proposed loss. Experimental results show consistent and significant improvements over existing methods in various categories, and gives state-of-the-art (SotA) performance among AP-loss based methods across three large-scale image (object) retrieval benchmarks.