Learning Super-Features for Image Retrieval

1 INTRODUCTION . Image retrieval is a task that models exemplar-based recognition , i.e . a class-agnostic , fine-grained understanding task which requires to retrieve all images matching a query image over an ( often very large ) image collection . It requires learning features that are discriminative enough for a highly detailed visual understanding but also robust enough to extreme viewpoint/pose or illumination changes . A popular image retrieval task is landmark retrieval , whose goal is to single out pictures of the exact same landmark out of millions of images , possibly containing a different landmark from the exact same fine-grained class ( e.g . ‘ gothic-era churches with twin bell towers ’ ) . While early approaches relied on handcrafted local descriptors , recent methods use image-level ( global ) or local Convolutional Neural Networks ( CNN ) features , see Csurka & Humenberger ( 2018 ) for a review . The current state of the art performs matching or re-ranking using CNN-based local features ( Noh et al. , 2017 ; Cao et al. , 2020 ; Tolias et al. , 2020 ) and only learns with global ( i.e . image-level ) annotations and losses . This is done by aggregating all local features into a global representation on which the loss is applied , creating a discrepancy between training and inference . Attention maps from modules like the ones proposed by Vaswani et al . ( 2017 ) are able to capture intermediate scene-level information , which makes them fundamentally similar to mid-level features ( Xiao et al. , 2015 ; Chen et al. , 2019 ) . Unlike individual neurons in CNNs which are highly localized , attention maps may span the full input tensor and focus on more global or semantic patterns . Yet , the applicability of mid-level features for instance-level recognition and image retrieval is currently underwhelming ; we argue that this is due to the following reasons : generic attention maps are not localized and may fire on multiple unrelated locations ; at the same time , object-centric attentions such as the one proposed by Locatello et al . ( 2020 ) produce too few attentional features and there is no mechanism to supervise them individually . In both cases , methods apply supervision at the global level , and the produced attentional features are simply not discriminative enough . In this paper , we present a novel image representation and training framework based solely on attentional features we call Super-features . We introduce an iterative Local feature Integration Trans- former ( LIT ) , which tailors existing attention modules to the task of image retrieval . Compared to the slot attention ( Locatello et al. , 2020 ) for example , it is able to output an ordered and much larger set of features , as it is based on learned templates , and has a simplified recurrence mechanism . For learning , we devise a loss that is applied directly to Super-features , yet it only requires imagelevel annotations . It pairs a contrastive loss on a set of matching Super-features across matching images , with a decorrelation loss on the attention maps of each image , to encourage Super-feature diversity . In the end , our network extracts for each image a fixed-size set of Super-features that are semantically ordered , i.e. , each firing on different types of patterns ; see Figure 1 for some examples . At test time , we follow the protocol of the best performing recent retrieval methods and use ASMK ( Tolias et al. , 2013 ) , except that we aggregate and match Super-features instead of local features . Our experiments show that the proposed method significantly outperforms the state of the art on common benchmarks like ROxford and RParis ( Radenović et al. , 2018a ) , while requiring less memory . We further show that performance gains persist in the larger scale , i.e . after adding 1M distractor images . Exhaustive ablations suggest that Super-features are less redundant and more discriminative than local features . Contributions . Our contribution is threefold : ( a ) an image representation based on Super-features and an iterative module to extract them ; ( b ) a framework to learn such representations , based on a loss applied directly on Super-features yet only requiring image-level labels ; ( c ) extensive evaluations that show significant performance gains over the state of the art for landmark image retrieval . We call our method Feature Integration-based Retrieval or FIRe for short . 2 BACKGROUND : LEARNING LOCAL FEATURES WITH A GLOBAL LOSS . Let function f : I Ñ RWˆHˆD denote a convolutional neural network ( CNN ) backbone that encodes an input image x P I into a pW ˆH ˆDq-sized tensor of D-dimensional local activations over a pW ˆ Hq spatial grid . After flattening the spatial dimensions , the output of f can also be seen as set of L “ W ¨ H feature vectors denoted by U “ tul P RD : l P 1 .. Lu ; note that the size of this set varies with the resolution of the input image . These local features are then typically whitened and their dimension reduced , a process that we represent by function op¨q in this paper . Global representations , i.e . image-level feature vectors , are commonly produced by averaging all local features , e.g . via global average or max pooling ( Babenko & Lempitsky , 2015 ; Tolias et al. , 2016 ; Gordo et al. , 2016 ) . A global contrastive loss for training . Tolias et al . ( 2020 ) argue that optimizing global representations is a good surrogate for learning local features to be used together with efficient match kernels for image retrieval . When building their global representation gpUq , they weight the contribution of each local feature to the aggregated vector using its l2 norm : gpUq “ ĝpUq } ĝpUq } 2 , ĝpUq “ L ÿ l “ 1 } ul } 2 ¨ opulq , ( 1 ) where } ¨ } 2 denotes the l2 norm . Given a database where each image pair is annotated as matching with each other or not , they minimize a contrastive loss over tuples of global representations . Intuitively , this loss encourages the global representations of matching images to be similar and those of non-matching images to be dissimilar . Let tuple pU , U ` , V´1 , . . . , V´n q represent the sets of local features of images px , x ` , y´1 , . . . , y´n q , where x and x ` are matching images ( i.e . a positive pair ) and none of the images y´1 , . . . , y ´ n is matching with image x ( i.e . they are negatives ) . Let r¨s ` denote the positive part and µ a margin hyper-parameter . They define a contrastive loss over global representations as : Lglobal “ › ›gpUq ´ gpU ` q › › 2 2 ` n ÿ j “ 1 “ µ´ › ›gpUq ´ gpV´j q › › 2 2 ‰ ` . ( 2 ) In HOW , Tolias et al . ( 2020 ) employ the global contrastive loss of Eq . ( 2 ) to learn a model whose local features are then used with match kernels such as ASMK ( Tolias et al. , 2013 ) for image retrieval . ASMK is a matching process defined over selective matching kernels of local features ; it is a much stricter and more precise matching function than comparing global representations , and is crucial for achieving good performance . By learning solely using a loss defined over global representations and directly using ASMK over local features U , HOW achieves excellent image retrieval performance . 3 LEARNING WITH SUPER-FEATURES . Methods using a global loss for training but local features for matching have a number of disadvantages . First , using local activations as local features leads to high redundancy , as they exhaustively cover highly overlapping patches of the input image . Second , using ASMK on local features from a model trained with a global loss introduces a mismatch between training and testing : the local features used for ASMK are only trained implicitly , but are expected to individually match in the matching kernel . To obtain less redundant feature sets , we propose to learn Super-features using a loss function that operates directly on those features , and to also use the latter during retrieval ; the train/testing discrepancy of the pipeline presented in Section 2 is thus eliminated . In this section , we first introduce the Local feature Integration Transformer ( LIT ) , an iterative attention module which produces an ordered set of Super-features ( Section 3.1 ) . We then present a framework for effectively learning such features ( Section 3.2 ) that consists of two losses : A contrastive loss that matches individual Super-features across positive image pairs , and a decorrelation loss on the Super-feature attention maps that encourages them to be diverse . An overview of the pipeline is depicted in Figure 2 . We refer to our approach as Feature Integration-based Retrieval or FIRe for short , an homage to the feature integration theory of Treisman & Gelade ( 1980 ) . 3.1 LOCAL FEATURE INTEGRATION TRANSFORMER ( LIT ) . Inspired by the recent success of attention mechanisms for encoding semantics from global context in sequences ( Vaswani et al. , 2017 ) or images ( Caron et al. , 2021 ) , we rely on attention to design our Local feature Integration Transformer ( LIT ) , a module that outputs an ordered set of Super-features . Let LIT be represented by function ΦpUq : RLˆD Ñ RNˆd that takes as input the set of local features U and outputs N Super-features . We define LIT as an iterative module : ΦpUq “ QT , Qt “ φpU ; Qt´1q , ( 3 ) where φ denotes the core function of the module applied T times , and Q0 P RNˆd denotes a set of learnable templates , i.e . a matrix of learnable parameters . Super-features are progressively formed by iterative refinement of the templates , conditioned on the local features from the CNN . The architecture of the core function φ is inspired by the Transformer architecture ( Vaswani et al. , 2017 ) and is composed of a dot-product attention function ψ , followed by a multi-layer perceptron ( MLP ) . The dot-product attention function ψ receives three inputs , the key , the value and the query1 which are passed through layer normalization and fed to linear projection functionsK , V andQ that project them to dimensions dk , dv and dq , respectively . In practice , we set dk “ dv “ dq “ d “ 1024 . The key and value inputs are set as the local features ul P U across all iterations . The query input is the set of templates Qt “ tqtn P Rd , n “ 1 .. Nu . It is initialized as the learnable templates Q0 for iteration 0 , and is set as the previous output of function φ for the following iterations . After projecting with the corresponding linear projection functions , the key and the query are multiplied to construct a set ofN attention maps over the local features , i.e. , the columns of matrixα P RLˆN , while the L rows αl of that matrix can be seen as the responsibility that each of the N templates has for each local feature l , and is given by2 : α “ » — – α1 ... αL fi ffi fl P RLˆN , αl “ α̂l řL i “ 1 α̂i , α̂l “ eMl řN n “ 1 e Mln , Mln “ Kpulq ¨ Qpqtnq ? d . ( 4 ) The dot product between keys and templates can be interpreted as a tensor of compatibility scores between local features and templates . These scores are normalized across templates via a softmax function , and are further turned into attention maps by l1 normalization across all L spatial locations . This is a common way of approximating a joint normalization function across rows and columns , also used by Locatello et al . ( 2020 ) . The input value is first projected with V , re-weighted with the attention maps and then residually fed to a MLP3 to produce the output of function φ : Qt “ φpU ; Qt´1q , φpU ; Qq “ MLPpψpU ; Qqq ` ψpU ; Qq , ψpU ; Qq “ V pUq ¨α ` Q . ( 5 ) Following standard image retrieval practice , we further whiten and l2-normalize the output of ΦpUq to get the final set of Super-features . Specifically , let ΦpUq “ rŝ1 ; . . . ; ŝN s P RNˆd be the raw output of our iterative attention module , we define the ordered set of Super-features as : S “ ! sn : sn “ opŝnq } opŝnq } 2 , n “ 1 , .. , N ) , ( 6 ) where , as in Section 2 , op¨q denotes dimensionality reduction and whitening . Figure 2 ( right ) illustrates the architecture of LIT . Note that all learnable parameters of φ are shared across all iterations . What do Super-features attend to ? Each Super-feature in S is a function of all local features in U , invariant to permutation of its elements , and can thus attend arbitrary regions of the image . 1The term ‘ query ’ has a precise meaning for retrieval ; yet , for this subsection only , we overload the term to refer to one of the inputs of the dot-product attention , consistently with the terminology from seminal works on attention by Vaswani et al . ( 2017 ) . 2The attention maps presented in Eq . ( 4 ) are technically taken at iteration t , but we omit iteration superscripts for clarity . For the rest of the paper and visualizations , we use attention maps to refer to the attention maps of Eq . ( 4 ) after the final ( T -th ) iteration of the iterative module . 3The MLP function consists of a layer-norm , a fully-connected layer with half the dimensions of the features , a ReLU activation and a fully-connected layer that projects features back to their initial dimension . To visualize the patterns captured by Super-features , Figure 1 shows the attention maps of the same five Super-features for three images , including two of the same landmark ( first two rows ) . The type of patterns depends on the learned initialization templates Q0 ; this explains why the Super-features form an ordered set , a property which allows to directly compare Super-features with the same ID . We observe the attention maps to be similar across images of the same landmark and to contain some mid-level patterns ( such as a half-circle on the second column , or windows on the third one ) .

Summary: The paper proposes Local Feature Integration Transformer (LIT), a new architecture to extract mid-level local features from images that are used for image retrieval. The resulting features are called SuperFeatures and are shown to yield state-of-the-art performance on two standard image retrieval benchmarks. The paper further proposes a learning framework, called Feature Integration-based Retrieval (FIRe), to train LIT using a contrastive loss with only image-level supervision. In contrast to other deep learning based image features that apply the loss on the image embedding or aggregated local features, FIRe applies the loss on the local features directly. An interesting property of SuperFeatures is that they are ordered, meaning a SuperFeature with the same index i in two matching images will be localized on similar structures.