Re-ranking for image retrieval and transductive few-shot classification

1 Introduction . Learning deep image features that generalize beyond the training classes they have been trained on has been a clear success [ 63 ] . Using these strong features , recent works have shown that high performances can be obtained simply by computing nearest neighbors , in particular for image retrieval [ 1 , 16 , 50 , 11 , 30 , 63 ] and few-shot image classification [ 71 , 6 ] . In this paper we highlight that , even with these strong features , results can be further improved by a large margin through re-ranking images based on the similarity graph between neighbors . To this end , we present a graph based deep architecture for re-ranking neighborhood images via a learning approach . Our method can be seen as revisiting and complementing classical approaches to re-ranking with deep learning techniques . These approaches can be broadly classified into query expansion that compute a new query feature based on the top retrieved neighbors [ 9 , 8 , 2 , 16 , 50 ] , and k-reciprocal re-ranking that compute a new distance between images based on the Jaccard distance between confident neighbors [ 48 , 75 ] . The approach we propose is related to both approaches . Since on one side , we focus successively on the neighbors of each sample , and on the other side , we update features to compute new similarities . This is achieved by designing a neural network architecture that can update the similarity graph by successively focusing on different subgraphs ( as visualized in Figure 1 ) and updating features based on synthetic gradients [ 24 , 21 ] . Our architecture for updating the similarity graph , referred to as Subgraph Similarity Refiner ( SSR ) , operates on the adjacency matrices 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of the subgraphs . We focus successively on each image ( as a query image ) , extract a subgraph , where our key technical insight is to sort all other images according to the similarity to the query image . We show that our approach can improve different image features for both image retrieval and transductive few-shot classification . We relate both tasks by casting them as a similarity graph refinement problem , where the refined similarity graph is used for task-specific predictions . In image retrieval , based on classical re-ranking techniques query expansion [ 9 , 8 , 16 , 50 ] and k-reciprocal [ 75 ] , our method consistently improves the image retrieval performance ( mean average precision , mAP ) achieved by recent state-of-the-art features [ 11 , 30 , 63 ] . For example , applying our method with the recent ProxyNCA++ [ 63 ] features improves the mAP @ R from 55.4 % to 60.6 % on Stanford Online Product dataset [ 62 ] and combining our approach with k-reciprocal [ 75 ] re-ranking further boost them to 62.3 % . In transductive few-shot classification , we show that k-reciprocal [ 75 ] based re-ranking yields a simple but surprisingly good baseline , while by learning to re-rank our method consistently improves over several competitive transductive approaches , e.g. , the synthetic information bottleneck ( SIB ) [ 21 ] . 2 Approach . Motivation Consider the similarity graph among N images , each node of the graph corresponds to one image and each weighted edge represents the similarity between two images . The similarity graph plays an important role in computer vision . As an example , the k-nearest-neighbor classifier remains a competitive method in image classification if the similarity graph is sufficiently informative . Note that , during training , we do have access to the target similarity graph , which has similarity 1 if two images belong to the same class and 0 otherwise , can we learn to improve an initial similarity graph of test images given that we have seen the target similarity matrix for training images ? Indeed , in the context of image retrievel , this idea of exploring and improving the similarity graph is called re-ranking , which refers to the case where an initial set of images have been retrieved , we then re-rank their relevance to the query image by examining again their similarities . Certainly , this idea can be hardly scaled to a large set of images as the complexity is quadratic , but it is quite interesting in the case of few-shot learning for unseen categories / domains . In this section , we first present an overview of our approach as a generalization to re-ranking ( Section 2.1 ) ; we then introduce our model architecture , called Subgraph Similarity Refiner ( SSR ) ( Section 2.2 ) and how to combine our approach with the classical k-reciprocal [ 75 ] re-ranking approach ( Section 2.3 ) ; finally , in Section 2.4 we explain how we train our approach and give implementation details . 2.1 Learning to improve a similarity graph . We assume that we are given a set of N images and for each image an initial feature . During training , we also assume that we are given a label for each image . Our goal is to predict an improved similarity , where images with the same label have a higher similarity than images with different labels . We will represent the similarity graph by its adjacency matrix , which we will refer to as similarity matrix . Note that the similarity matrix allows to explore neighbors of the nearest neighbors ( high order nearest neighbors ) , which is a classical and effective way to tackle the re-ranking problem [ 48 , 75 ] . More formally , for i ∈ { 1 , . . . , N } we denote by f0i the initial feature of ith image and yi its label . The initial similarity matrix is given by : S0 = [ s0ij ] i , j∈ { 1 , ... , N } with s 0 ij = 〈f0i , f0j 〉 ‖f0i ‖‖f0j ‖ . ( 1 ) Our goal is to predict the target similarity matrix Ŝ = [ 1yi=yj ] i , j∈ { 1 , ... , N } where 1yi=yj is the indicator function of yi = yj . We refine the similarity matrix iteratively . First , consider the following update given a similarity matrix St−1 : S̃t : = [ s̃tij ] i , j∈ { 0,1 , ... , N } = S t−1 +G ( St−1 ) , ( 2 ) where G is the Subgraph Similarity Refiner ( SSR ) that we will describe in the next section . The issue with such an update is the flexibility . We therefore would like to constraint the updated similarity matrix to have the form St = [ stij ] i , j∈ { 1 , ... , N } with s t ij = 〈f ti , f tj 〉 ‖f ti ‖‖f tj‖ . ( 3 ) This amounts to construct the update in features rather than in the similarity matrix directly . As such , St always remain positive semidefinite . Now , the question is how we make use of S̃t to obtain f t. Given f t−1 , in fact , we only need to move a single gradient step along the projection direction according to the Euclidean distance Lf ( f ) : = 12 ∑ i , j ( s̃ t i , j − 〈fi , fj〉 ‖fi‖‖fj‖ ) 2 . Specifically , we obtain f t by considering S̃t as an intermediate result : ∀i : f ti = f t−1i − λ ∂Lf ∂fi ( f t−1 ) with ∂Lf ∂fi ( f t−1 ) = − ∑ j ( s̃ti , j − st−1i , j ) ∂st−1i , j ∂f t−1i ( f t−1 ) ( 4 ) where λ is the step size . Indeed , this update can be seen as applying the synthetic gradient descent [ 21 , 24 ] on the features . Here , we motivate it from a different perspective . Note that the network G defines a meta-model , as the update is supposed to generalize to unseen image categories . We learn G by minimizing a task-specific loss between the target similarity matrix Ŝ and the updated similarity matrix ST . The key for this approach to work is the architecture design of the network G , which we detail in the next section . 2.2 Subgraph Similarity Refiner ( SSR ) . The adjacency matrix S is a natural way to encode a similarity graph . However , it is defined up to the order of the nodes , i.e . permutations of its column and rows . For this reason , learning directly to make a prediction from the similarity matrix is challenging . One can simply order the nodes with respect to their distance to a specific sample , such as the query in the case of image retrieval , but this gives one of the nodes a specific role and it might be hard for the network to use the similarity structure among the other nodes , especially since S is very high dimensional and overfitting might be an issue . Thus , we propose a permutation-invariant architecture illustrated in Figure 2a . Our key idea is to make N update predictions by considering only subgraphs centered on each node i and then to aggregate them to obtain an update prediction on the full graph . We can associate a similarity matrix Mi to the ith subgraph , where we order nodes according to their similarity to the node i . Note that the order will be different for each subgraph . We then predict updates g ( Mi ) for each of the subgraphs using a simple network g . Finally , we aggregate the predictions on all the subgraphs by summing the updates predicted for each subgraph at the relevant position in S : G ( S ) = GraphSum ( { g ( Mi ) } Ni=1 ) ( 5 ) where g is a multi-layer perceptron ( MLP ) and GraphSum is an aggregation operator to account for the summation with respect to graph structure since each node is contained in multiple subgraphs . We now discuss the exact structure of the subgraph and associated similarity matrix Mi that we use in the case of image retrieval or transductive few-shot classification . The choice of structures in both cases are validated by the ablation studies presented in Section 3.3 . Image Retrieval . In image retrieval , we search for images similar to a query image in a pool of test images . We assume that a first algorithm already selected the N − 1 test images most similar to the query and we focus on improving the similarities between the resulting set of N images ( the query and its N − 1 nearest neighbors ) . By increasing the similarities between the query and the positive images against those of the negatives , we aim to obtain a better retrieval result where positives should be ranked before negatives for the N − 1 test images selected for the query . During training , we optimize the InfoNCE loss [ 44 ] with a learnable temperature parameter τ : ∑ i , j : yi=yj=yqry − log ( exp ( τsTij ) exp ( τsTij ) + ∑ k : yk 6=yi exp ( τs T ik ) ) ( 6 ) where sTij signifies the similarity between the i th sample and jth sample after T similarity updates . As visualized in Figure 2b , we build the ith subgraph by considering only edges that connect nodes to either the ith sample or the query . Rather than considering Mi as the adjacency matrix of this graph , we define it as a 2 ×N matrix defined as follows . Each column corresponds to a different sample and the samples are ordered with respect to their similarity of the ith sample . The values in the first row are the similarities between the ith sample and the samples associated to each column , it is thus decreasing . The values in the second row are the similarity between the query and the samples associated with each column . Note that for the subgraph corresponding to the query , the two lines of the matrix are the same and actually correspond to the same edges in the graph . Transductive few-shot classification . In few-shot classification , the set of N images to consider can be divided in two : a support set S with known labels and an unlabelled query setQ for which we want to predict labels . Our approach assumes that Q is accessible during the inference . This setting is known as transductive few-shot classification . For each sample in the query set , we aim to increase the similarities between itself and the support images from the same class comparing to those from different classes . To this end , we minimize the Cross Entropy loss : ∑ i∈Q , j∈S : yqi =y s j − log ( exp ( τsTij ) exp ( τsTij ) + ∑ k∈S , yqi 6=ysk exp ( τsTik ) ) ( 7 ) where τ is a learnable temperature parameter , yq· and ys· are labels of query and support samples . As visualized in Figure 2c , we build the ith subgraph by considering only the edges that connect nodes either to the ith sample or to the support set . Similar to the case of retrieval , we represent this graph using a structured matrix , but we keep the nodes corresponding to the support and query set separated . We define Mi as a ( |S|+ 1 ) × ( |S|+ |Q| ) matrix with |·| the number of elements in a set . Its first |S| columns and last |Q| columns correspond to the samples in support set and query set , respectively . They are both sorted by decreasing similarity with respect to the ith sample . While the values in the first row of Mi represent the similarities between the ith sample and the samples in different columns , the last |S| rows represent similarities between support samples and different columns . Note that if the ith sample comes from the support set , one row will be repeated in Mi .

Summary: The paper is about learning how to re-rank for retrieval with global image descriptors. There is a first step of retrieval to get the top-N neighbors to the query. Then, the NxN adjacency matrix is created which is the input for the re-ranking during training and during testing. The adjacency matrix is refined in an iterative way and by processing different sub-matrices each time.  During training the NxN adjacency matrix is forced to be close to the binary matrix that is contructed according to the class labels. At the same time, with the use of synthetic gradients, the backbone that is used to generate the feature vectors is updated so that the cosine similarity of the feature vectors facilitates the update of the adjacency matrix.  - this is a novel approach to re-ranking - the whole formulation is new for this task, so does the use of synthetic gradients - the method is applied on two different tasks: image retrieval and transductive few-shot learning.