Learning Super-Features for Image Retrieval

1 INTRODUCTION . Image retrieval is a task that models exemplar-based recognition , i.e . a class-agnostic , fine-grained understanding task which requires to retrieve all images matching a query image over an ( often very large ) image collection . It requires learning features that are discriminative enough for a highly detailed visual understanding but also robust enough to extreme viewpoint/pose or illumination changes . A popular image retrieval task is landmark retrieval , whose goal is to single out pictures of the exact same landmark out of millions of images , possibly containing a different landmark from the exact same fine-grained class ( e.g . ‘ gothic-era churches with twin bell towers ’ ) . While early approaches relied on handcrafted local descriptors , recent methods use image-level ( global ) or local Convolutional Neural Networks ( CNN ) features , see Csurka & Humenberger ( 2018 ) for a review . The current state of the art performs matching or re-ranking using CNN-based local features ( Noh et al. , 2017 ; Cao et al. , 2020 ; Tolias et al. , 2020 ) and only learns with global ( i.e . image-level ) annotations and losses . This is done by aggregating all local features into a global representation on which the loss is applied , creating a discrepancy between training and inference . Attention maps from modules like the ones proposed by Vaswani et al . ( 2017 ) are able to capture intermediate scene-level information , which makes them fundamentally similar to mid-level features ( Xiao et al. , 2015 ; Chen et al. , 2019 ) . Unlike individual neurons in CNNs which are highly localized , attention maps may span the full input tensor and focus on more global or semantic patterns . Yet , the applicability of mid-level features for instance-level recognition and image retrieval is currently underwhelming ; we argue that this is due to the following reasons : generic attention maps are not localized and may fire on multiple unrelated locations ; at the same time , object-centric attentions such as the one proposed by Locatello et al . ( 2020 ) produce too few attentional features and there is no mechanism to supervise them individually . In both cases , methods apply supervision at the global level , and the produced attentional features are simply not discriminative enough . In this paper , we present a novel image representation and training framework based solely on attentional features we call Super-features . We introduce an iterative Local feature Integration Trans- former ( LIT ) , which tailors existing attention modules to the task of image retrieval . Compared to the slot attention ( Locatello et al. , 2020 ) for example , it is able to output an ordered and much larger set of features , as it is based on learned templates , and has a simplified recurrence mechanism . For learning , we devise a loss that is applied directly to Super-features , yet it only requires imagelevel annotations . It pairs a contrastive loss on a set of matching Super-features across matching images , with a decorrelation loss on the attention maps of each image , to encourage Super-feature diversity . In the end , our network extracts for each image a fixed-size set of Super-features that are semantically ordered , i.e. , each firing on different types of patterns ; see Figure 1 for some examples . At test time , we follow the protocol of the best performing recent retrieval methods and use ASMK ( Tolias et al. , 2013 ) , except that we aggregate and match Super-features instead of local features . Our experiments show that the proposed method significantly outperforms the state of the art on common benchmarks like ROxford and RParis ( Radenović et al. , 2018a ) , while requiring less memory . We further show that performance gains persist in the larger scale , i.e . after adding 1M distractor images . Exhaustive ablations suggest that Super-features are less redundant and more discriminative than local features . Contributions . Our contribution is threefold : ( a ) an image representation based on Super-features and an iterative module to extract them ; ( b ) a framework to learn such representations , based on a loss applied directly on Super-features yet only requiring image-level labels ; ( c ) extensive evaluations that show significant performance gains over the state of the art for landmark image retrieval . We call our method Feature Integration-based Retrieval or FIRe for short . 2 BACKGROUND : LEARNING LOCAL FEATURES WITH A GLOBAL LOSS . Let function f : I Ñ RWˆHˆD denote a convolutional neural network ( CNN ) backbone that encodes an input image x P I into a pW ˆH ˆDq-sized tensor of D-dimensional local activations over a pW ˆ Hq spatial grid . After flattening the spatial dimensions , the output of f can also be seen as set of L “ W ¨ H feature vectors denoted by U “ tul P RD : l P 1 .. Lu ; note that the size of this set varies with the resolution of the input image . These local features are then typically whitened and their dimension reduced , a process that we represent by function op¨q in this paper . Global representations , i.e . image-level feature vectors , are commonly produced by averaging all local features , e.g . via global average or max pooling ( Babenko & Lempitsky , 2015 ; Tolias et al. , 2016 ; Gordo et al. , 2016 ) . A global contrastive loss for training . Tolias et al . ( 2020 ) argue that optimizing global representations is a good surrogate for learning local features to be used together with efficient match kernels for image retrieval . When building their global representation gpUq , they weight the contribution of each local feature to the aggregated vector using its l2 norm : gpUq “ ĝpUq } ĝpUq } 2 , ĝpUq “ L ÿ l “ 1 } ul } 2 ¨ opulq , ( 1 ) where } ¨ } 2 denotes the l2 norm . Given a database where each image pair is annotated as matching with each other or not , they minimize a contrastive loss over tuples of global representations . Intuitively , this loss encourages the global representations of matching images to be similar and those of non-matching images to be dissimilar . Let tuple pU , U ` , V´1 , . . . , V´n q represent the sets of local features of images px , x ` , y´1 , . . . , y´n q , where x and x ` are matching images ( i.e . a positive pair ) and none of the images y´1 , . . . , y ´ n is matching with image x ( i.e . they are negatives ) . Let r¨s ` denote the positive part and µ a margin hyper-parameter . They define a contrastive loss over global representations as : Lglobal “ › ›gpUq ´ gpU ` q › › 2 2 ` n ÿ j “ 1 “ µ´ › ›gpUq ´ gpV´j q › › 2 2 ‰ ` . ( 2 ) In HOW , Tolias et al . ( 2020 ) employ the global contrastive loss of Eq . ( 2 ) to learn a model whose local features are then used with match kernels such as ASMK ( Tolias et al. , 2013 ) for image retrieval . ASMK is a matching process defined over selective matching kernels of local features ; it is a much stricter and more precise matching function than comparing global representations , and is crucial for achieving good performance . By learning solely using a loss defined over global representations and directly using ASMK over local features U , HOW achieves excellent image retrieval performance . 3 LEARNING WITH SUPER-FEATURES . Methods using a global loss for training but local features for matching have a number of disadvantages . First , using local activations as local features leads to high redundancy , as they exhaustively cover highly overlapping patches of the input image . Second , using ASMK on local features from a model trained with a global loss introduces a mismatch between training and testing : the local features used for ASMK are only trained implicitly , but are expected to individually match in the matching kernel . To obtain less redundant feature sets , we propose to learn Super-features using a loss function that operates directly on those features , and to also use the latter during retrieval ; the train/testing discrepancy of the pipeline presented in Section 2 is thus eliminated . In this section , we first introduce the Local feature Integration Transformer ( LIT ) , an iterative attention module which produces an ordered set of Super-features ( Section 3.1 ) . We then present a framework for effectively learning such features ( Section 3.2 ) that consists of two losses : A contrastive loss that matches individual Super-features across positive image pairs , and a decorrelation loss on the Super-feature attention maps that encourages them to be diverse . An overview of the pipeline is depicted in Figure 2 . We refer to our approach as Feature Integration-based Retrieval or FIRe for short , an homage to the feature integration theory of Treisman & Gelade ( 1980 ) . 3.1 LOCAL FEATURE INTEGRATION TRANSFORMER ( LIT ) . Inspired by the recent success of attention mechanisms for encoding semantics from global context in sequences ( Vaswani et al. , 2017 ) or images ( Caron et al. , 2021 ) , we rely on attention to design our Local feature Integration Transformer ( LIT ) , a module that outputs an ordered set of Super-features . Let LIT be represented by function ΦpUq : RLˆD Ñ RNˆd that takes as input the set of local features U and outputs N Super-features . We define LIT as an iterative module : ΦpUq “ QT , Qt “ φpU ; Qt´1q , ( 3 ) where φ denotes the core function of the module applied T times , and Q0 P RNˆd denotes a set of learnable templates , i.e . a matrix of learnable parameters . Super-features are progressively formed by iterative refinement of the templates , conditioned on the local features from the CNN . The architecture of the core function φ is inspired by the Transformer architecture ( Vaswani et al. , 2017 ) and is composed of a dot-product attention function ψ , followed by a multi-layer perceptron ( MLP ) . The dot-product attention function ψ receives three inputs , the key , the value and the query1 which are passed through layer normalization and fed to linear projection functionsK , V andQ that project them to dimensions dk , dv and dq , respectively . In practice , we set dk “ dv “ dq “ d “ 1024 . The key and value inputs are set as the local features ul P U across all iterations . The query input is the set of templates Qt “ tqtn P Rd , n “ 1 .. Nu . It is initialized as the learnable templates Q0 for iteration 0 , and is set as the previous output of function φ for the following iterations . After projecting with the corresponding linear projection functions , the key and the query are multiplied to construct a set ofN attention maps over the local features , i.e. , the columns of matrixα P RLˆN , while the L rows αl of that matrix can be seen as the responsibility that each of the N templates has for each local feature l , and is given by2 : α “ » — – α1 ... αL fi ffi fl P RLˆN , αl “ α̂l řL i “ 1 α̂i , α̂l “ eMl řN n “ 1 e Mln , Mln “ Kpulq ¨ Qpqtnq ? d . ( 4 ) The dot product between keys and templates can be interpreted as a tensor of compatibility scores between local features and templates . These scores are normalized across templates via a softmax function , and are further turned into attention maps by l1 normalization across all L spatial locations . This is a common way of approximating a joint normalization function across rows and columns , also used by Locatello et al . ( 2020 ) . The input value is first projected with V , re-weighted with the attention maps and then residually fed to a MLP3 to produce the output of function φ : Qt “ φpU ; Qt´1q , φpU ; Qq “ MLPpψpU ; Qqq ` ψpU ; Qq , ψpU ; Qq “ V pUq ¨α ` Q . ( 5 ) Following standard image retrieval practice , we further whiten and l2-normalize the output of ΦpUq to get the final set of Super-features . Specifically , let ΦpUq “ rŝ1 ; . . . ; ŝN s P RNˆd be the raw output of our iterative attention module , we define the ordered set of Super-features as : S “ ! sn : sn “ opŝnq } opŝnq } 2 , n “ 1 , .. , N ) , ( 6 ) where , as in Section 2 , op¨q denotes dimensionality reduction and whitening . Figure 2 ( right ) illustrates the architecture of LIT . Note that all learnable parameters of φ are shared across all iterations . What do Super-features attend to ? Each Super-feature in S is a function of all local features in U , invariant to permutation of its elements , and can thus attend arbitrary regions of the image . 1The term ‘ query ’ has a precise meaning for retrieval ; yet , for this subsection only , we overload the term to refer to one of the inputs of the dot-product attention , consistently with the terminology from seminal works on attention by Vaswani et al . ( 2017 ) . 2The attention maps presented in Eq . ( 4 ) are technically taken at iteration t , but we omit iteration superscripts for clarity . For the rest of the paper and visualizations , we use attention maps to refer to the attention maps of Eq . ( 4 ) after the final ( T -th ) iteration of the iterative module . 3The MLP function consists of a layer-norm , a fully-connected layer with half the dimensions of the features , a ReLU activation and a fully-connected layer that projects features back to their initial dimension . To visualize the patterns captured by Super-features , Figure 1 shows the attention maps of the same five Super-features for three images , including two of the same landmark ( first two rows ) . The type of patterns depends on the learned initialization templates Q0 ; this explains why the Super-features form an ordered set , a property which allows to directly compare Super-features with the same ID . We observe the attention maps to be similar across images of the same landmark and to contain some mid-level patterns ( such as a half-circle on the second column , or windows on the third one ) .

Summary: This paper introduces the concept of super features into image retrieval. The whole framework is named FIRe. The idea is to aggregate a set of features on the feature maps at different scales according to a set of query vectors representing different latent concepts. Hence, the proposed model somehow implements a transformer, which is called LIT in this paper.  To train this model, the authors define two losses. The first one is a triplet-like one executed on the super-feature granular. Some empirical criteria are established to select positive and negative super features. The second one decorrelates the super features from a single image.  ASMK becomes the measurement to compare two images when testing, which is a conventional option in comparing two groups of local features.  We see experiments on ROxford and RParis. The results are quite promising.