Robust and Decomposable Average Precision for Image Retrieval

1 Introduction . The task of ‘ query by example ’ is a major prediction problem , which consists in learning a similarity function able to properly rank all the instances in a retrieval set according to their relevance to the query , such that relevant items have the largest similarity . In computer vision , it drives several major applications , e.g . content-based image retrieval , face recognition or person re-identification . Such tasks are usually evaluated with rank-based metrics , e.g . Recall @ k , Normalized Discounted Cumulative Gain ( NDCG ) , and Average Precision ( AP ) . AP is also the de facto metric used in several vision tasks implying a large imbalance between positive and negative samples , e.g . object detection . In this paper , we address the problem of direct AP training with stochastic gradient-based optimization , e.g . using deep neural networks , which poses two major challenges . Firstly , the AP loss LAP = 1− AP is not differentiable and is thus not directly amenable to gradientbased optimization . There has been a rich literature for providing smooth and upper bound surrogate 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . losses for LAP [ 43 , 21 , 22 , 6 , 25 ] . More recently , smooth differentiable rank approximations have been proposed [ 35 , 14 , 15 , 3 , 27 , 8 , 2 ] , but generally lose the important LAP upper bound property . The second important issue of AP optimization relates to its non-decomposability : LBAP averaged over batches underestimates LAP on the whole training dataset , which we refer as the decomposability gap . In image retrieval , the attempts to circumvent the problem involve ad hoc methods based on batch sampling strategies [ 10 , 32 , 20 , 32 , 30 ] , or storing all training representations/scores [ 39 , 3 , 27 , 25 ] , leading to complex models with a large computation and memory overhead . In this paper , we introduce a method for RObust And DecoMposable Average Precision ( ROADMAP ) , which explicitly addresses the aforementioned challenges of AP optimization . Our first contribution is to propose a new surrogate loss LSupAP for LAP . In particular , we introduce a smooth approximation of the rank function , with a different behaviour for positive and negative examples . By this design , LSupAP provides an upper bound of LAP , and always back-propagates gradients when the correct ranking is not satisfied . These two features illustrated in the the toy example on Figure 1a are not fulfilled by binning approaches [ 3 , 27 ] or by SmoothAP [ 2 ] . As a second contribution , we propose to improve the non-decomposability in AP training . To this end , we introduce a simple yet effective training objective Lcalibr. , which calibrates the scores among different batches by controlling the absolute value of positive and negative samples . We provide a theoretical analysis showing that Lcalibr . decreases the decomposability gap . Figure 1 illustrates how Lcalibr . can be leveraged to improve the overall ranking . We provide a thorough experimental validation including three standard image retrieval datasets and show that ROADMAP outperforms state-of-the-art methods . We also report the large and consistent gain compared to rank/AP approximation baselines , and we highlight in the ablation studies the importance of our two contributions . Finally , ROADMAP does not entail any memory or computation overhead and remains competitive even with small batches . 2 Related work . We discuss here the literature in image retrieval dedicated to AP optimization , and compare to other approaches based on optimizing representations [ 23 , 1 , 44 , 46 , 33 ] in the experiments . Smooth AP approximations Studying smooth surrogate losses for AP has a long history . The widely used surrogate for retrieval is to consider constraints based on pairs [ 41 , 12 , 26 ] , triplets [ 11 ] , quadruplets [ 18 ] or n-uplets [ 30 ] to enforce partial ranking . These metric learning methods optimize a very coarse upper bound on AP and need complex post-processing and tricks to be effective . One option for training with AP is to design smooth upper bounds on the AP loss . Seminal works are based on structural SVMs [ 43 , 21 ] , with extensions to speed-up the `` loss-augmented inference '' [ 22 ] or to adapt to weak supervision [ 6 ] . Recently , a generic blackbox combinatorial solver has been introduced [ 25 ] and applied to AP optimization [ 28 ] . To overcome the brittleness of AP with respect to small score variations , an ad hoc perturbation is applied to positive and negative scores during training . These methods provide elegant AP upper bounds , but generally are coarse AP approximations . Other approaches rely on designing smooth approximations of the the rank function . This is done in soft-binning techniques [ 14 , 15 , 35 , 3 , 27 ] by using a smoothed discretization of similarity scores . Other approaches rely on explicitly approximating the non-differentiable rank functions using neural networks [ 8 ] , or with a sum of sigmoid functions in the recent SmoothAP approach [ 2 ] . These approaches enable accurate AP approximations by providing tight and smooth approximations of the rank function . However , they do not guarantee that the resulting loss is an AP loss upper bound . The LSupAP introduced in this work is based on a smooth approximation of the rank function leading to an upper bound on the AP loss , making our approach both accurate and robust . Decomposability in AP optimization Batch training is mandatory in deep learning . However , the non-decomposability of AP is a severe issue , since it yields an inconsistent AP gradient estimator . Non-decomposability is related to sampling informative constraints in simple AP surrogates , e.g . triplet losses , since the constraints ’ cardinality on the whole training set is prohibitive . This has been addressed by efficient batch sampling [ 13 , 10 , 32 ] or selecting informative constraints within mini-batches [ 30 , 9 , 4 , 32 ] . In cross-batch memory technique [ 39 ] , the authors assume a slow drift in learned representations to store them and compute global mining in pair-based deep metric learning . In AP optimization , the non-decomposability has essentially been addressed by a brute force increase of the batch size [ 3 , 27 , 25 ] . This includes an important overhead in computation and memory , generally involving a two-step approach for first computing the AP loss and subsequently re-computing activations and back-propagating gradients . In contrast , our loss Lcalibr . does not add any overhead and enables good performances for AP optimization even with small batches . 3 Robust and decomposable AP training . We present here our method for RObust And DecoMposable AP ( ROADMAP ) dedicated to direct optimization of a smooth surrogate of AP with stochastic gradient descent ( SGD ) , see Fig . 2 . Training context Let us consider a retrieval set Ω = { xj } j∈J1 ; NK composed of N elements , and a set of M queries included in Ω , i.e . Q = { qi } i∈J1 ; MK ⊆ Ω . For each query qi , each element in Ω is assigned a label y ( xj , qi ) ∈ { +1 ; −1 } , such that y ( xj , qi ) = 1 ( resp . y ( xj , qi ) = −1 ) if xj is relevant ( resp . irrelevant ) with respect to qi . This defines a query-dependent partitioning of Ω such that Ω = Pi ∪Ni , where Pi : = { xj ∈ Ω|y ( xj , qi ) = +1 } and Ni : = { xj ∈ Ω|y ( xj , qi ) = −1 } . For each xj ∈ Ω , we define a prediction model parametrized by parameters θ , e.g . a deep neural network , which provides a vectorial embedding vqi ∈ Rd of each element , i.e . : vqi : = fθ ( qi ) . In the embedded space Rd , we compute a similarity score between each query qi and each element in Ω , e.g . by using the cosine similarity : s ( qi , xj ) = vqi Tvj ||vqi ||2||vj||2 . During training , our goal is to optimize , for each query qi , the model parameters θ such that positive elements are ranked before negatives . More precisely , we aim at minimizing the AP loss LAPi for each query qi in the retrieval set Ω . Our overall AP loss LAP is averaged over all queries : LAP ( θ ) = 1− 1 M M∑ i=1 APi ( θ ) , APi ( θ ) = 1 |Pi| ∑ k∈Pi Pre ( k , θ ) = 1 |Pi| ∑ k∈Pi rank+ ( k , θ ) rank ( k , θ ) ( 1 ) where Pre ( k , θ ) is the precision for the kth positive example xk , rank+ ( k , θ ) its rank among positives Pi , and the rank ( k , θ ) its rank over Ω = Pi ∪Ni . As previously mentioned , there are two main challenges with SGD optimization of AP in Eq . ( 1 ) : i ) AP ( θ ) is not differentiable with respect to θ , and ii ) AP does not linearly decompose into batches . ROADMAP addresses both issues : we introduce the robust differentiable LSupAP surrogate ( Section 3.1 ) , and add the Lcalibr . loss ( Section 3.2 ) to improve AP decomposability . Our final loss LROADMAP is a linear combination of LSupAP and Lcalibr. , weighted by the hyperparameter λ : LROADMAP ( θ ) = ( 1− λ ) · LSupAP ( θ ) + λ · Lcalibr . ( θ ) ( 2 ) 3.1 Robustness in smooth rank approximation . The non-differentiablity in Eq ( 1 ) comes from the ranking operator , which can be viewed as counting the number of instances that have a similarity score greater than the considered instance , i.e.1 : rank+ ( k ) = 1 + ∑ j∈Pi\ { k } H ( sj − sk ) , where H ( t ) = { 1 if t ≥ 0 0 otherwise rank ( k ) = rank+ ( k ) + ∑ j∈Ni H ( sj − sk ) = rank+ ( k ) + rank− ( k ) ( 3 ) From Eq . ( 3 ) it becomes clear that the non-differentiablity is due to the Heaviside ( step ) function H , whose derivative is either zero or undefined . Note that the computation of rank+ ( k ) and rank− ( k ) in Eq . ( 3 ) relates to the rank of positive instances xk ∈ Pi : the score sk in Eq . ( 3 ) is always the score of a positive , whereas sj can either be a negative ’ s or positive ’ s score . Smooth loss LSupAP To provide a smooth approximation of LAP in Eq . ( 1 ) , we introduce a smooth approximation of the rank function . In particular , we propose a different behaviour between rank+ ( k ) and rank− ( k ) in Eq . ( 3 ) by defining two functions H+ and H− . For rank+ ( k ) , we choose to keep the Heaviside ( step ) function , i.e . H+ = H ( see Fig . 3a ) , which consists in ignoring rank+ ( k ) in gradient-based AP optimization . This is done on purpose since ∂AP ∂ rank+ ( k ) = rank − ( k ) ( rank+ ( k ) +rank− ( k ) ) 2 ≥ 0 : the gradient would tend to increase rank+ ( k ) and to decrease the score of sk . Reminding xk is always a positive instance , this behaviour is undesirable . For rank− ( k ) , we define the following smooth surrogate H− for H , shown in Fig 3b : H− ( t ) = σ ( tτ ) if t ≤ 0 , where σ is the sigmoid function ( Fig . 3c ) σ ( tτ ) + 0.5 if t ∈ [ 0 ; δ ] with δ ≥ 0 ρ · ( t− δ ) + σ ( δτ ) + 0.5 if t > δ ( 4 ) where τ and ρ are hyperparameters , and δ is defined such that the sigmoidal part of H− reaches the saturation regime and is fixed for the rest of the paper ( see supplementary Sec . A ) . From 1For the sake of readability we drop in the following the dependence on θ for the rank , i.e . rank ( k ) : = rank ( k , θ ) and on the query for the similarity , i.e . sj : = s ( qi , xj ) . the H− smooth approximation defined in Eq . ( 4 ) , we obtain the following smooth approximation rank−s ( k ) = ∑ j∈Ni H− ( sj − sk ) , leading to the following smooth AP loss approximation : LSupAP ( θ ) = 1− 1 M M∑ i=1 1 |Pi| ∑ k∈Pi rank+ ( k ) rank+ ( k ) + rank−s ( k ) ( 5 ) LSupAP in Eq . ( 5 ) fulfills two main features for AP optimization : I 1 LSupAP is an upper bound of LAP in Eq . ( 1 ) . Since H− in Eq . ( 4 ) is an upper bound of a step function ( Fig 3b ) , it is easy to see that LSupAP ≥ LAP . This is a very important property , since it ensures that the model keeps training until the correct ranking is obtained . It is worth noting that existing smooth rank approximations in the literature [ 35 , 3 , 27 , 2 ] do not fulfill this property . I 2 LSupAP brings training gradients until the correct ranking plus a margin is fulfilled . When the ranking is incorrect , the negative xj is ranked before the positive xk , thus sj > sk and H− ( sj − sk ) in Eq . ( 4 ) has a non-null derivative . We use a sigmoid to have a large gradient when sj − sk is small . To overcome vanishing gradients of the sigmoid for large values sj − sk , we use a linear function ensuring constant ρ derivative . When the ranking is correct ( sj < sk ) , we enforce robustness by imposing a margin parametrized by τ ( sigmoid in Eq . ( 4 ) ) . This margin overcomes the brittleness of rank losses , which vanish as soon as the ranking is correct [ 14 , 3 , 25 ] . Comparison to SmoothAP [ 2 ] LSupAP differs from LSmoothAP in [ 2 ] by i ) providing an upper bound on LAP , ii ) improving the gradient flow ( Fig . 3b vs Fig . 3c ) , and iii ) overcoming adverse effects of the sigmoid for rank+ , as shown in Fig . 1a ( and in supplementary sec . A ) . We experimentally verify the consistent gain brought out by LSupAP over LSmoothAP .

Summary: This paper proposes a new method for deep image retrieval. The main contribution is a differentiable and decomposable loss function based on average precision loss. The method shows superior performance and the theoretical seems correct.