Re-ranking for image retrieval and transductive few-shot classification

1 Introduction . Learning deep image features that generalize beyond the training classes they have been trained on has been a clear success [ 63 ] . Using these strong features , recent works have shown that high performances can be obtained simply by computing nearest neighbors , in particular for image retrieval [ 1 , 16 , 50 , 11 , 30 , 63 ] and few-shot image classification [ 71 , 6 ] . In this paper we highlight that , even with these strong features , results can be further improved by a large margin through re-ranking images based on the similarity graph between neighbors . To this end , we present a graph based deep architecture for re-ranking neighborhood images via a learning approach . Our method can be seen as revisiting and complementing classical approaches to re-ranking with deep learning techniques . These approaches can be broadly classified into query expansion that compute a new query feature based on the top retrieved neighbors [ 9 , 8 , 2 , 16 , 50 ] , and k-reciprocal re-ranking that compute a new distance between images based on the Jaccard distance between confident neighbors [ 48 , 75 ] . The approach we propose is related to both approaches . Since on one side , we focus successively on the neighbors of each sample , and on the other side , we update features to compute new similarities . This is achieved by designing a neural network architecture that can update the similarity graph by successively focusing on different subgraphs ( as visualized in Figure 1 ) and updating features based on synthetic gradients [ 24 , 21 ] . Our architecture for updating the similarity graph , referred to as Subgraph Similarity Refiner ( SSR ) , operates on the adjacency matrices 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of the subgraphs . We focus successively on each image ( as a query image ) , extract a subgraph , where our key technical insight is to sort all other images according to the similarity to the query image . We show that our approach can improve different image features for both image retrieval and transductive few-shot classification . We relate both tasks by casting them as a similarity graph refinement problem , where the refined similarity graph is used for task-specific predictions . In image retrieval , based on classical re-ranking techniques query expansion [ 9 , 8 , 16 , 50 ] and k-reciprocal [ 75 ] , our method consistently improves the image retrieval performance ( mean average precision , mAP ) achieved by recent state-of-the-art features [ 11 , 30 , 63 ] . For example , applying our method with the recent ProxyNCA++ [ 63 ] features improves the mAP @ R from 55.4 % to 60.6 % on Stanford Online Product dataset [ 62 ] and combining our approach with k-reciprocal [ 75 ] re-ranking further boost them to 62.3 % . In transductive few-shot classification , we show that k-reciprocal [ 75 ] based re-ranking yields a simple but surprisingly good baseline , while by learning to re-rank our method consistently improves over several competitive transductive approaches , e.g. , the synthetic information bottleneck ( SIB ) [ 21 ] . 2 Approach . Motivation Consider the similarity graph among N images , each node of the graph corresponds to one image and each weighted edge represents the similarity between two images . The similarity graph plays an important role in computer vision . As an example , the k-nearest-neighbor classifier remains a competitive method in image classification if the similarity graph is sufficiently informative . Note that , during training , we do have access to the target similarity graph , which has similarity 1 if two images belong to the same class and 0 otherwise , can we learn to improve an initial similarity graph of test images given that we have seen the target similarity matrix for training images ? Indeed , in the context of image retrievel , this idea of exploring and improving the similarity graph is called re-ranking , which refers to the case where an initial set of images have been retrieved , we then re-rank their relevance to the query image by examining again their similarities . Certainly , this idea can be hardly scaled to a large set of images as the complexity is quadratic , but it is quite interesting in the case of few-shot learning for unseen categories / domains . In this section , we first present an overview of our approach as a generalization to re-ranking ( Section 2.1 ) ; we then introduce our model architecture , called Subgraph Similarity Refiner ( SSR ) ( Section 2.2 ) and how to combine our approach with the classical k-reciprocal [ 75 ] re-ranking approach ( Section 2.3 ) ; finally , in Section 2.4 we explain how we train our approach and give implementation details . 2.1 Learning to improve a similarity graph . We assume that we are given a set of N images and for each image an initial feature . During training , we also assume that we are given a label for each image . Our goal is to predict an improved similarity , where images with the same label have a higher similarity than images with different labels . We will represent the similarity graph by its adjacency matrix , which we will refer to as similarity matrix . Note that the similarity matrix allows to explore neighbors of the nearest neighbors ( high order nearest neighbors ) , which is a classical and effective way to tackle the re-ranking problem [ 48 , 75 ] . More formally , for i ∈ { 1 , . . . , N } we denote by f0i the initial feature of ith image and yi its label . The initial similarity matrix is given by : S0 = [ s0ij ] i , j∈ { 1 , ... , N } with s 0 ij = 〈f0i , f0j 〉 ‖f0i ‖‖f0j ‖ . ( 1 ) Our goal is to predict the target similarity matrix Ŝ = [ 1yi=yj ] i , j∈ { 1 , ... , N } where 1yi=yj is the indicator function of yi = yj . We refine the similarity matrix iteratively . First , consider the following update given a similarity matrix St−1 : S̃t : = [ s̃tij ] i , j∈ { 0,1 , ... , N } = S t−1 +G ( St−1 ) , ( 2 ) where G is the Subgraph Similarity Refiner ( SSR ) that we will describe in the next section . The issue with such an update is the flexibility . We therefore would like to constraint the updated similarity matrix to have the form St = [ stij ] i , j∈ { 1 , ... , N } with s t ij = 〈f ti , f tj 〉 ‖f ti ‖‖f tj‖ . ( 3 ) This amounts to construct the update in features rather than in the similarity matrix directly . As such , St always remain positive semidefinite . Now , the question is how we make use of S̃t to obtain f t. Given f t−1 , in fact , we only need to move a single gradient step along the projection direction according to the Euclidean distance Lf ( f ) : = 12 ∑ i , j ( s̃ t i , j − 〈fi , fj〉 ‖fi‖‖fj‖ ) 2 . Specifically , we obtain f t by considering S̃t as an intermediate result : ∀i : f ti = f t−1i − λ ∂Lf ∂fi ( f t−1 ) with ∂Lf ∂fi ( f t−1 ) = − ∑ j ( s̃ti , j − st−1i , j ) ∂st−1i , j ∂f t−1i ( f t−1 ) ( 4 ) where λ is the step size . Indeed , this update can be seen as applying the synthetic gradient descent [ 21 , 24 ] on the features . Here , we motivate it from a different perspective . Note that the network G defines a meta-model , as the update is supposed to generalize to unseen image categories . We learn G by minimizing a task-specific loss between the target similarity matrix Ŝ and the updated similarity matrix ST . The key for this approach to work is the architecture design of the network G , which we detail in the next section . 2.2 Subgraph Similarity Refiner ( SSR ) . The adjacency matrix S is a natural way to encode a similarity graph . However , it is defined up to the order of the nodes , i.e . permutations of its column and rows . For this reason , learning directly to make a prediction from the similarity matrix is challenging . One can simply order the nodes with respect to their distance to a specific sample , such as the query in the case of image retrieval , but this gives one of the nodes a specific role and it might be hard for the network to use the similarity structure among the other nodes , especially since S is very high dimensional and overfitting might be an issue . Thus , we propose a permutation-invariant architecture illustrated in Figure 2a . Our key idea is to make N update predictions by considering only subgraphs centered on each node i and then to aggregate them to obtain an update prediction on the full graph . We can associate a similarity matrix Mi to the ith subgraph , where we order nodes according to their similarity to the node i . Note that the order will be different for each subgraph . We then predict updates g ( Mi ) for each of the subgraphs using a simple network g . Finally , we aggregate the predictions on all the subgraphs by summing the updates predicted for each subgraph at the relevant position in S : G ( S ) = GraphSum ( { g ( Mi ) } Ni=1 ) ( 5 ) where g is a multi-layer perceptron ( MLP ) and GraphSum is an aggregation operator to account for the summation with respect to graph structure since each node is contained in multiple subgraphs . We now discuss the exact structure of the subgraph and associated similarity matrix Mi that we use in the case of image retrieval or transductive few-shot classification . The choice of structures in both cases are validated by the ablation studies presented in Section 3.3 . Image Retrieval . In image retrieval , we search for images similar to a query image in a pool of test images . We assume that a first algorithm already selected the N − 1 test images most similar to the query and we focus on improving the similarities between the resulting set of N images ( the query and its N − 1 nearest neighbors ) . By increasing the similarities between the query and the positive images against those of the negatives , we aim to obtain a better retrieval result where positives should be ranked before negatives for the N − 1 test images selected for the query . During training , we optimize the InfoNCE loss [ 44 ] with a learnable temperature parameter τ : ∑ i , j : yi=yj=yqry − log ( exp ( τsTij ) exp ( τsTij ) + ∑ k : yk 6=yi exp ( τs T ik ) ) ( 6 ) where sTij signifies the similarity between the i th sample and jth sample after T similarity updates . As visualized in Figure 2b , we build the ith subgraph by considering only edges that connect nodes to either the ith sample or the query . Rather than considering Mi as the adjacency matrix of this graph , we define it as a 2 ×N matrix defined as follows . Each column corresponds to a different sample and the samples are ordered with respect to their similarity of the ith sample . The values in the first row are the similarities between the ith sample and the samples associated to each column , it is thus decreasing . The values in the second row are the similarity between the query and the samples associated with each column . Note that for the subgraph corresponding to the query , the two lines of the matrix are the same and actually correspond to the same edges in the graph . Transductive few-shot classification . In few-shot classification , the set of N images to consider can be divided in two : a support set S with known labels and an unlabelled query setQ for which we want to predict labels . Our approach assumes that Q is accessible during the inference . This setting is known as transductive few-shot classification . For each sample in the query set , we aim to increase the similarities between itself and the support images from the same class comparing to those from different classes . To this end , we minimize the Cross Entropy loss : ∑ i∈Q , j∈S : yqi =y s j − log ( exp ( τsTij ) exp ( τsTij ) + ∑ k∈S , yqi 6=ysk exp ( τsTik ) ) ( 7 ) where τ is a learnable temperature parameter , yq· and ys· are labels of query and support samples . As visualized in Figure 2c , we build the ith subgraph by considering only the edges that connect nodes either to the ith sample or to the support set . Similar to the case of retrieval , we represent this graph using a structured matrix , but we keep the nodes corresponding to the support and query set separated . We define Mi as a ( |S|+ 1 ) × ( |S|+ |Q| ) matrix with |·| the number of elements in a set . Its first |S| columns and last |Q| columns correspond to the samples in support set and query set , respectively . They are both sorted by decreasing similarity with respect to the ith sample . While the values in the first row of Mi represent the similarities between the ith sample and the samples in different columns , the last |S| rows represent similarities between support samples and different columns . Note that if the ith sample comes from the support set , one row will be repeated in Mi .

Summary: The paper deals with image retrieval, i.e. the problem of finding images in a database which are the most similar to a given query image. More specifically, the paper describes a method for re-ranking retrieval (and few-shot classification) results in order to improve performance. The technique consists in learning how to refine the retrieval results by first representing them as a similarity matrix, then learning a deep network G that learns how to produce a step that refines this matrix.  The contributions of the paper could be summarized as follows:  1) A novel method for re-ranking that uses a neural network to enhance the similarity matrix directly (which the authors call a ubgraph Similarity Refiner); 2) An approach for combining the method above with k-reciprocal re-ranking; 3) Experiments showing the method above leads to consistent improvements (although with varying levels of significance)