# Multiproxies Adaptive Distribution Loss with Weakly Supervised Feature Aggregation for Fine-Grained Retrieval

Authors: Hongwei Zhao; Danyang Zhang; Jiaxin Wu; Pingping Liu
Journal: Scientific Programming
 (2021)

---

## Abstract

Fine-grained retrieval is one of the complex problems in computer vision. Compared with general content-based image retrieval, fine-grained image retrieval faces more difficult challenges. In fine-grained image retrieval tasks, all classes belong to a subclass of a meta-class, so there will be small interclass variance and large intraclass variance. In order to solve this problem, in this paper, we propose a fine-grained retrieval method to improve loss and feature aggregation, which can achieve better retrieval results under a unified framework. Firstly, we propose a novel multiproxies adaptive distribution loss which can better characterize the intraclass variations and the degree of dispersion of each cluster center. Secondly, we propose a weakly supervised feature aggregation method based on channel weighting, which distinguishes the importance of different feature channels to obtain more representative image feature descriptors. We verify the performance of our proposed method on the universal benchmark datasets such as CUB200-2011 and Stanford Dog. Higher Recall@K demonstrates the advantage of our proposed method over the state of the art.


---

## Body

## 1. Introduction

Fine-grained image retrieval [1–3] is to search images in a meta-class. Each class belongs to a subclass of a metaclass, such as birds [4, 5] and dogs [6]. Since all the subclasses are very similar, different classes can only be distinguished by subtle parts, so fine-grained image retrieval faces a more difficult challenge. In recent years, fine-grained image retrieval facing more significant challenges has become a new research hotspot.The granularity in fine-grained image retrieval is used to describe the characteristics of the dataset. Coarse-grained retrieval is relatively simple and usually refers to the obvious difference between the classes of the dataset. Fine-grained means that the distinction between the classes of the dataset is small. The idea of this granularity is also widely used in various fields of computer algorithm research to speed up the processing speed. For example, with a chatbot respond, it contains two parts: a coarse-grained selection module and a fine-grained selection module [7]. Granular computing [8, 9] is to simulate the recognition method of the human brain, using simple, low-cost approximate solutions instead of precise solutions to improve efficiency and robustness. The application of granular computing theory to the knowledge discovery process of the rough set has also become a new research direction [10].The development of fine-grained image retrieval is mainly divided into two stages. Early fine-grained image retrieval was based on hand-crafted features [11–13]. Since hand-crafted features are not universal and require designers to have strong professional knowledge to get feature descriptors that have the greatest impact on retrieval performance, the subsequent fine-grained image retrieval is mainly based on deep learning methods. Similar to general content-based for image retrieval [14–17], deep convolution features are better than fully connected layer features [18, 19]. The methods of fine-grained image retrieval based on deep learning to improve retrieval accuracy mainly include (a) designing a practical loss function [20–23] to obtain a better embedding space (b) designing an effective aggregation method to obtain a more representative image descriptor.The previous fine-grained image retrieval algorithm has specific problems. (a) Pair-based losses [24–26]: all sample pairs are limited to a minibatch, and global information cannot be obtained. The final retrieval effect will be affected mainly by the sample mining strategy [27–30]. (b) The previous loss usually only considers the similarity between each proxy and each sample, or the similarity between each image sample obtained after convolution, but it does not consider the similarity between each proxy and each proxy [31–33]. And distributing a single proxy to each class cannot sufficiently characterize the intraclass changes of fine-grained images [34–36]. (c) For fine-grained retrieval problems, deep convolution features are better than those of fully connected layers [3, 25, 37–39]. A lot of information is lost when aggregating three-dimensional convolution features into one-dimensional image feature descriptors. The obtained feature descriptors are not well represented. In order to solve these existing problems and obtain better fine-grained retrieval results, in this paper, we put forward a more effective proxy-based loss called multiproxies adaptive distribution loss and a weakly supervised feature aggregation model. The main contributions of this paper can be summarized as follows:(1)
We propose a multiproxies adaptive distribution scheme, which adaptively distributes the number of proxies for each class in the training set according to the average intraclass distances and reduces the amount of calculation as much as possible while better characterizing the intraclass variations;(2)
We propose a novel multiproxies adaptive distribution loss, which simultaneously optimizes the similarity between each proxy and all the batch samples and then optimizes the similarity between each proxy and all the other proxies. As shown in Figure1, in the final embedding space, on the one hand, samples of each class are clustered together. On the other hand, the cluster centers of each class are separated from each other;(3)
We propose a weakly supervised feature aggregation model, which uses the feature response map obtained by fine-tuning CNN to extract target features, and then weighted the obtained feature descriptors according to the importance of the channel without using any object bounding box. The obtained feature descriptors have a more excellent representation.Figure 1
Comparison of our method with some previous losses based on metric learning. Small nodes represent image feature descriptors and virtual proxies in a minibatch embedding space. Let the node be a sample or proxy, and the dotted nodes are marked as virtual proxies. Conversely, the remaining nodes represent samples. (a) Triplet loss associates each anchor with a positive and a negative sample without considering sample hardness. (b) Lifted structure associates each anchor with all positive and negative samples, considering all the samples’ hardness. (c) Proxy NCA loss distributes a single proxy for each class; it associates each sample only with proxies. The final embedding space can cluster samples of the same class together, but it does not guarantee that the cluster centers of each class are separated from each other. (d) Our loss dynamically distributes multiple proxies for each class according to the multiproxies adaptive distribution scheme. The form of loss considers the relationship between not only each proxy and all the batch samples but also each proxy and the other proxies so that the final embedding space can better characterize the intraclass variations while separating the cluster centers of each class. For more details, see the subsequent sections of the paper.We performed comparative experiments on the universal benchmark fine-grained image retrieval datasets CUB200-2011 [5] and Stanford Dog [6]. And it is known from the results that our method surpasses the state-of-the-art in terms of Recall@K.


## 2. Related Work

In this section, we first briefly introduced the development of fine-grained image retrieval in recent years and then introduced the general deep metric learning loss.


### 2.1. Fine-Grained Image Retrieval (FGIR)

Fine-grained image retrieval is a research direction that has emerged in recent years. Compared with general image retrieval, it faces more complex challenges, so it has received more and more research attention. The classes of the general image retrieval are very different, such as cats, dogs, and birds, while the classes of fine-grained image retrieval belong to a subclass of a meta-class, such as Husky and Alaska. It is more challenging to complete the retrieval task from the subclasses of the same meta-class. Fine-grained image retrieval is now facing three major challenges: (1) the large intraclass variance: it is a massive challenge for fine-grained retrieval to distinguish images of a subclass from the same meta-class because of the obstruction of lighting, background, and posture; (2) the small interclass variance: since each subclass of fine-grained image retrieval belongs to a meta-class, there are only subtle differences between different subclasses, such as the shape of a dog’s ears and the colour of bird’s wings; and (3) limited datasets: the labelling of datasets usually requires professional knowledge and a lot of time.Xie et al. [11] first proposed the concept of fine-grained image “search” in 2015, which constructed a hierarchical search library. Firstly, it judges the metaclass to which it belongs and then performs fine-grained image retrieval. It uses general hand-crafted features, such as SIFT [12], to measure similarity. The selective convolutional descriptor aggregation (SCDA) [3] proposed by Wei et al. in 2017 is the first attempt to apply deep learning to fine-grained image retrieval. It innovatively proposes an unsupervised convolution feature location method to form the final descriptor for fine-grained retrieval. The centralized ranking loss [2] proposed by Zheng et al. in 2018 and the decor related centralized loss [33] proposed in 2019 combine the loss with weakly supervised feature aggregation to achieve better retrieval accuracy. A large number of subsequent fine-grained image retrieval research mainly focused on the method of deep metric learning [35, 36, 40]. This method trains deep convolutional neural networks by setting a specific loss function to make a more ideal embedding space. The general metric learning loss function will be introduced in detail in Section 2.2.


### 2.2. Deep Metric Learning

The goal of deep metric learning is to maximize the similarity of images from the same class through an optimal loss function and to minimize the similarity of images from a different class to obtain a better embedding space. In this section, we will discuss the loss based on metric learning in two parts: one part is the pair-based losses [41–44] and the other part is the proxy-based losses [2, 33, 35, 45]. Below, we review some pair-based losses and proxy-based losses.Pair-based losses include contrastive loss [21], triplet loss [1], lifted structure loss [46], N-pair loss [41], and multisimilarity loss [42]. Pair-based loss uses all the sample points in a minibatch to calculate the loss, so there are two problems. The first is that there will be a lot of calculations that greatly increase the length of training. The second is that the amount of information contained in the samples in a minibatch is limited, and the result will be largely affected by the sample mining strategy.Proxy-based losses include proxy nca loss [45], center loss [23], decorrelated centralized loss [33], and piecewise cross-entropy loss [36]. Proxy-based loss introduces the concept of the global proxy so that all classes of the training set can be considered while reducing the amount of calculation instead of being limited to samples in a minibatch. However, the previous proxy-based loss distributes a proxy to each class without considering the differences within the class. And they focus more on how to better aggregate each class and neglect to enhance the dispersion between each class.As shown in Figure1, there are some previous loss comparison charts, which are a brief introduction to how it works, and draw the effect of using proxy nca loss [45] to embed space. Based on this, we proposed a multiproxies adaptive distribution loss and compared it with the previous losses.


## 2.1. Fine-Grained Image Retrieval (FGIR)

Fine-grained image retrieval is a research direction that has emerged in recent years. Compared with general image retrieval, it faces more complex challenges, so it has received more and more research attention. The classes of the general image retrieval are very different, such as cats, dogs, and birds, while the classes of fine-grained image retrieval belong to a subclass of a meta-class, such as Husky and Alaska. It is more challenging to complete the retrieval task from the subclasses of the same meta-class. Fine-grained image retrieval is now facing three major challenges: (1) the large intraclass variance: it is a massive challenge for fine-grained retrieval to distinguish images of a subclass from the same meta-class because of the obstruction of lighting, background, and posture; (2) the small interclass variance: since each subclass of fine-grained image retrieval belongs to a meta-class, there are only subtle differences between different subclasses, such as the shape of a dog’s ears and the colour of bird’s wings; and (3) limited datasets: the labelling of datasets usually requires professional knowledge and a lot of time.Xie et al. [11] first proposed the concept of fine-grained image “search” in 2015, which constructed a hierarchical search library. Firstly, it judges the metaclass to which it belongs and then performs fine-grained image retrieval. It uses general hand-crafted features, such as SIFT [12], to measure similarity. The selective convolutional descriptor aggregation (SCDA) [3] proposed by Wei et al. in 2017 is the first attempt to apply deep learning to fine-grained image retrieval. It innovatively proposes an unsupervised convolution feature location method to form the final descriptor for fine-grained retrieval. The centralized ranking loss [2] proposed by Zheng et al. in 2018 and the decor related centralized loss [33] proposed in 2019 combine the loss with weakly supervised feature aggregation to achieve better retrieval accuracy. A large number of subsequent fine-grained image retrieval research mainly focused on the method of deep metric learning [35, 36, 40]. This method trains deep convolutional neural networks by setting a specific loss function to make a more ideal embedding space. The general metric learning loss function will be introduced in detail in Section 2.2.


## 2.2. Deep Metric Learning

The goal of deep metric learning is to maximize the similarity of images from the same class through an optimal loss function and to minimize the similarity of images from a different class to obtain a better embedding space. In this section, we will discuss the loss based on metric learning in two parts: one part is the pair-based losses [41–44] and the other part is the proxy-based losses [2, 33, 35, 45]. Below, we review some pair-based losses and proxy-based losses.Pair-based losses include contrastive loss [21], triplet loss [1], lifted structure loss [46], N-pair loss [41], and multisimilarity loss [42]. Pair-based loss uses all the sample points in a minibatch to calculate the loss, so there are two problems. The first is that there will be a lot of calculations that greatly increase the length of training. The second is that the amount of information contained in the samples in a minibatch is limited, and the result will be largely affected by the sample mining strategy.Proxy-based losses include proxy nca loss [45], center loss [23], decorrelated centralized loss [33], and piecewise cross-entropy loss [36]. Proxy-based loss introduces the concept of the global proxy so that all classes of the training set can be considered while reducing the amount of calculation instead of being limited to samples in a minibatch. However, the previous proxy-based loss distributes a proxy to each class without considering the differences within the class. And they focus more on how to better aggregate each class and neglect to enhance the dispersion between each class.As shown in Figure1, there are some previous loss comparison charts, which are a brief introduction to how it works, and draw the effect of using proxy nca loss [45] to embed space. Based on this, we proposed a multiproxies adaptive distribution loss and compared it with the previous losses.


## 3. Proposed Method

The method we put forward includes two stages of training step and testing step, as shown in Figure2. In the training step, we fine-tune a pretrained CNN by the proposed multiproxies adaptive distribution loss (Sections 3.3 and 3.4). The loss consists of two parts: (1) multiproxies adaptive distribution scheme based on the average intraclass distance and (2) a loss form in which the sample can be aggregated with the same class of proxies and the different classes of proxies are dispersed. This loss fine-tunes the network parameters through the backpropagation of the gradient and finally obtains a fine-tuned CNN suitable for fine-grained datasets. In the testing phase, we used the weakly supervised feature aggregation model to obtain more representative image feature descriptors through extracting target features and channel weighting.Figure 2
Unified search framework. In the training phase, we train the pretrained CNN by our multiproxies adaptive distribution loss. The MPAD consists of two parts: multiproxies adaptive distribution scheme and multiproxies adaptive distribution loss. In the testing phase, we extract attractive features for the test datasets by a weakly supervised feature aggregation model. The weakly supervised feature aggregation model consists of two parts: extracting target features and channel weighted descriptor. The final descriptor is used in image retrieval. We retrieve the top-K related images using L2 distance.


### 3.1. Loss Motivation

Recalling the previous pair-based loss, it depends on the sample pairs constructed by the samples in each batch. The polynomial increase in the number of sample pairs will affect the efficiency of loss, and many sample pairs have very little information, some even do not generate gradients. The effect of loss will largely depend on the sample mining strategy, but even if a very effective sample mining strategy is developed, the loss training is essentially limited to sample pairs in a batch, and the global information of the embedding space cannot be obtained.Proxy-based losses, such as proxy nca loss and center loss, solve the previous problem of pair-based losses by distributing a single center or proxy to each class. They distribute a trainable parameter to each class so that the loss training takes into account the global class information, avoiding the sampling problem in each batch.However, there are still two problems with proxy-based losses. (1) They all distribute a trainable parameter, which is a single proxy or a single center to each class. However, in fine-grained retrieval tasks, a significant challenge is the large intraclass variance. Distributing a single proxy to each class is not enough to characterize the complex intraclass variations of the fine-grained dataset, and distributing multiple proxies to each class is not necessary for classes with small intraclass variance and it just will increase the amount of calculation. (2) They only consider increasing the similarity between the same class of proxies and samples and reducing the similarity between the different classes of proxies and samples so that the final embedding space realizes the clustering of each class, but it cannot ensure that the final cluster centers are scattered. If the proxies of each class cannot achieve a great separation, in the end, the optimal embedding space will not be obtained.As shown in Figure3, the upper and lower rows belong to the same class in the fine-grained dataset, but it does not make sense to cluster them excessively so that the brown information will be mixed with the red information to represent it. But in reality, among the anonymous data, there is only a tiny probability that brown information and red information are mixed in birds of the same class. Therefore, this kind of mixed information is not universal (as opposed to only learning brown information with angle change).Figure 3
CUB200-2011 dataset-cardinal class.As shown in Figure1, we propose a new metric learning loss called multi-proxies adaptive distribution loss to improve previous losses. Our loss first distributes the best number of proxies to each class of the training set through a multiproxies adaptive distribution scheme, which reduces the amount of calculation while better characterizing the intraclass variations. Secondly, we consider the similarity between multiple proxies so that the final embedding space of different-class proxies can be far away from each other, and the same-class proxies can be gathered together to a certain extent. Our loss significantly improves the quality of the embedding space.


### 3.2. Feature Aggregation Motivation

The selective convolutional descriptor aggregation (SCDA) [3] proposed by Wei et al. locates the target area from the spatial dimension according to the response map. The implementation of this scheme will cause the channel with a lot of 0 values and many channels to have fewer activation areas.As shown in Figure4, for each image in the test set, we will get the feature map of each channel through the fine-tuned neural network. After summing all channel feature maps, the response map will be obtained. According to the characteristics of the convolutional neural network, where the large response value is the target area and the area with the small response value is the background area, we can get the mask matrix according to the response map to locate the target area. After each channel feature map is selected by mask matrix, we will get channel scda maps. Our detailed analysis of each channel feature map of the obtained channel scda maps can find that some features are fewer active areas, such as channel-27 scda map and channel-52 scda map, activating the information of the bird’s tail and head. Fine-grained retrieval tasks often use this subtle feature to distinguish different classes of birds. Therefore, it does not mean that the activation area is less and the feature channel is not essential. Often, the response value of this channel after sum pooling or average pooling is small, so we give more weight to the feature channel with less activation area.Figure 4
(a) Original image randomly selected from the CUB200-2011 dataset test set. (b) The response map obtained by the fine-tuned convolutional neural network. (c) A part of the channel feature maps obtained by the fine-tuned convolutional neural network. (d) The channel feature maps obtained by the scda algorithm. Inside the gray dashed box are some representative channel scda maps.We proposed a weakly supervised aggregation model. The channel dimensions are weighted for the obtained feature descriptors, and the number of response values of each feature channel larger than 0 is counted, for channels with fewer response values larger than 0, which have essential information channels to give more weight and get more representative feature descriptors.We have achieved a more prominent effect on fine-grained retrieval tasks through multiproxies adaptive distribution loss and weakly supervised feature aggregation methods.


### 3.3. Multiproxies Adaptive Distribution Scheme

As shown in Figure5, for some classes of fine-grained datasets, such as Class A, simply distributing one proxy is not enough, but for some classes with small intraclass variance, distributing multiple proxies is redundant; it just will increase the amount of calculation, such as Class B. In order to solve the problem of distributing the number of proxies in each class, we propose a method of distributing proxies based on the average intraclass distance.Figure 5
CUB200-2011 dataset.
(a)(b)Firstly, we obtain the feature descriptors of each class of images in the training set through the fine-tuned convolutional neural network ResNet50. Secondly, for each class in the training set, we use the following formula to calculate the average intraclass distance:(1)dc=1nc2−nc∑i∈c,j∈cfi−fj2,where nc represents the number of images in class C and fi represents the image feature descriptor obtained by fine-tuned convolutional neural network for the i-th image in class C.By obtaining the average intraclass distance, we adaptively distribute the number of proxies for each class in the training set. the larger the average intraclass distance, the larger the intraclass variance of the class. For maintaining this intraclass variance, we distribute more proxies to the class with the large average intraclass distance. On the contrary, for the class with a small intraclass distance, we distribute fewer proxies. In this manner, the use of limited proxies can be achieved while maintaining the larger intraclass variance.Thirdly, our paper adopts the method of establishing a histogram. We determine the width of each rectangle in the histogram according to the interval of the average intraclass distances of all classes in the training set, build a histogram of the average intraclass distances of all training set classes, and divide the average intraclass distance intoM blocks. Then, the calculation formula for the range V of the value contained in each block is as follows:(2)V=dmax−dminM,where M represents the number of blocks divided by the histogram, dmax represents the maximum intraclass average distance of all classes in the training set, and dmin represents the smallest average intraclass distance of all classes in the training set. The ranges of M block values areA1=dmin,dmin+V, A2=dmin+V,dmin+2V……AM=dmin+M−1V,dmin+MV.By calculating the average intra-class distance of each class in the training set, we distribute a different number of proxies to each class. The training set classes belonging to the rangeA1 are distributed k proxies, the training set classes belonging to the range A2 are distributed k+1 proxies, and so on, the training set class belonging to the range AM is distributed k+M−1 proxies. Among them, k=1.According to the above rules, before training, we first adaptively distribute the number of proxies for all training set classes for subsequent loss training.


### 3.4. Multiproxies Adaptive Distribution Loss

The main idea of our multiproxies adaptive distribution loss allocation is as follows: (1) we adaptively distribute the number of proxies for all classes according to the average intraclass distance of all classes in the training set; (2) we propose a new loss form divided into two parts: the first partLp−sp,s+,s− is used to increase the similarity between each proxy with the same class samples and reduce the similarity between each proxy with different classes samples. For each pair of proxy and sample, the size of the gradient is determined according to their self-similarity and relative similarity. The final embedding space makes the samples of the same class cluster together. The second part Lp−pp,p+,p− entropy loss is used to realize the approach and separation of multiproxies, increase the similarity between the same-class proxies, and reduce the similarity between different-class proxies. The final embedding space makes the samples of the same class cluster together and the cluster centers of each class separate from each other.The training set hasH classes, and each batch has B samples, representing the feature descriptor of the i-th sample as fiS and its corresponding label as yiS; the feature descriptors output of minibatch samples can be defined as S=s1=f1S,y1S,s2=f2S,y2S,…,sB=fBS,yBS. Besides, we use the multiproxies adaptive distribution scheme in Section 3.3 to distribute the number of proxies offline for the training set. Our proxies are trainable network layer parameters. Suppose we distribute E proxies offline adaptively. Representing the feature descriptor of the i-th proxy as fiP and its corresponding label as yiP. The proxy settings can be represented by P=p1=f1p,y1p,p2=f2p,y2p,……,pE=fEp,yEp. We create a label matrix Lab for E proxies, the size of the label matrix Lab is E×H, the number of rows represents E proxies, and the number of columns represents the one-hot label of each proxy in H classes, used to represent the proxy class. For the i,j position of the Lab label matrix, if yip=j, Labij=1; else, Labij=0.Our loss is divided into two parts, as shown in Figure 1(f);Lp−sp,s+,s− considers the relationship between samples and proxies, using adaptively distributed E proxies as anchors, samples of the same class are considered positive, and samples of different classes are considered negative.A batch of embedding samplesS is divided into two sets: Sp+ and Sp−. For a selected proxy p, samples of the same class as in a batch belong to the set Sp+. And samples of a different class belong to the set Sp−. Lposp,s+ improves the similarity between proxies and samples in the same class. Lposp,s+ is then given by(3)Lposp,s+=1P∗∑p∈P∗log1+∑s+∈Sp+e−αsimp,s+−β,where α>0 is a scaling parameter, β>0 is a margin parameter, sim.,. denotes the cosine similarity between a proxy and a sample, P∗ represents the set of proxies with positive samples in the batch. If the number of all proxies is directly used as the coefficient and the batch does not contain samples of the class proxies, the mean will become lower.Lposp,s− reduces the similarity between different-class proxies and samples. The specific calculation formula is as follows:(4)Lnegp,s−=1P∑p∈Plog1+∑s−∈Sp−eαsimp,s−+β.The number of all proxies is directly used as the coefficient because for random sampling, the probability that all samples in a batch belong to the same class is negligible, so there will be negative samples for all proxies.The lastLp−sp,s+,s− function can be achieved by equations (3) and (4). Therefore, all proxies are capable of increasing the similarity between positive samples and reducing the similarity between negative samples. To be further, it will make the same-class samples gather together and different-class samples stay away from each other:(5)Lp−sp,s+,s−=Lposp,s++Lnegp,s−.In equation (3),(6)log1+∑s+∈Sp+e−αsimp,s+−β=log1+elog∑s+∈Sp+e−αsimp,s+−β=soft pluslog∑s+∈Sp+e−αsimp,s+−β=soft plusLSE−αsimp,s+−β=maxmax−αsimp,s+−β,0.Among them, Softplus:log1+ex=maxx,0 LSE::log∑ex=maxx.Similarly, in equation (4),(7)log1+∑s−∈Sp−eαsimp,s−+β=maxmaxαsimp,s−+β,0.For equation (6), when −αsimp,s+−β<0, it is simp,s+>β.For equation (7), when αsimp,s−+β<0, it is simp,s−<−β.log1+∑s+∈Sp+e−αsimp,s+−β=0, log1+∑s−∈Sp−eαsimp,s−+β=0, and Loss is 0 without training gradient.The training of the network can make the similarity between proxies and positive samples larger thanβ, and β is the decision boundary that determines the final similarity. Through the loss of equations (3) and (4), the effect of simp,s+>β, simp,s−<−β can finally be realized.Because of the size of the derivative considering all the samples' hardness, we take the derivative ofLp−sp,s+,s−. For hard samples, we use a larger thrust, that is, a larger derivative:(8)∂Lposp,s+∂simp,s+=1P+−αe−αsimp,s+−β1+∑s+∈Sp+e−αsimp,s+−β,(9)∂Lnegp,s−∂simp,s−=1P−αeαeαsimp,s−+β1+∑sk∈Sp−eαsimp,s−+β.According to equations (8) and (9), we can know that the gradient of Lposp,s+ decreases as the self-similarity of simp,s+ increases; that is, more dissimilar positive samples are pulled closer with larger strength, as the relative similarity of ∑s+∈Sp+e−αsimp,s+−β increases. In contrast, the gradient of Lnegp,s− increases as the self-similarity of simp,s− increases; that is, more similar negative samples retreat with greater force. As the relative similarity of ∑sk∈Sp−eαsimp,s−+β increases, it decreases.ForLp−pp,p+,p− considering the relationship between each proxy with the other proxies, the adaptively distributed E proxies are regarded as samples processing, the same-class proxies are considered to be positive, and different-class proxies are considered to be negative. We use classification-based Softmax and cross-entropy loss deformation to obtain Lp−pp,p+,p−.The commonly used Softmax and cross-entropy loss [47–49] form are as follows:(10)Pryi=eyi∑jeyj,LCE=−1N∑i=1NlogPryi.InLp−pp,p+,p−, proxies P are divided into two sets: P+ and P−. For a selected proxy p, the proxies of the same class belong to the set P+. And the samples of a different class belong to the set P−. Lp−pp,p+,p− makes the same-class proxies closer and the different-class proxies pushed farther:(11)Lp−pp,p+,p−=∑p∈P−log∑p+∈P+esimp,p+∑p−∈P−esimp,p−,where sim.,. denotes the cosine similarity between a proxy and the other proxy.We take the derivative ofLp−pp,p+,p−, the size of the derivative considering all the samples’ hardness. For hard samples, we use a larger thrust, which is a larger derivative:(12)∂Lp−pp,p+,p−∂simp,p+=−esimp,p+∑p+∈P+esimp,p+,(13)∂Lp−pp,p+,p−∂simp,p−=esimp,p−∑p−∈P−esimp,p−.According to equations (13) and (14), we can know that the gradient of the positive proxy pair decreases with the increase of simp,p+; that is, the more dissimilar positive proxy pairs are pulled closer with a larger force. The gradient of the negative proxy pair increases with the increase of simp,p−; that is, the more similar negative proxy pairs retreat with a larger force.In multiproxies adaptive distribution loss, we useLp−s to optimize the similarity between proxies and samples and use Lp−p to optimize the similarity between proxies and proxies:(14)LMPADLP,S=Lp−sp,s+,s−+ρLp−pp,p+,p−,where P are the adaptive distribution proxies. S is a set of real samples in a batch, and ρ>0 is the fusion parameter. In the training phase, we obtained a better embedding space and fine-tuned network parameters by minimizing LMPADLP,S for the generation of image descriptors during testing.


### 3.5. Weakly Supervised Aggregation Deep Convolutional Features

During the test, we obtain the three-dimensional convolution feature maps of all test images through the convolutional neural network fine-tuned by multiproxies adaptive distribution loss and then aggregate the three-dimensional convolution features into one-dimensional image features descriptor through our improved weakly supervised aggregation model for subsequent similarity measurement.For a given imageI (height m width n) and a CNN model, we input the image I into the fine-tuned ResNet50 model for forwarding propagation and obtain the three-dimensional feature map of the last convolutional layer X∈Rc×h×w.We first coarsely localize the object via SCDA [3] based on selective convolutional features and then obtain one-dimensional feature descriptors through GAP and GMP. Due to the characteristics of SCDA [3], channels with fewer activation areas are also vital. We weight the channels of the obtained one-dimensional descriptors and assign large weights to the channels with fewer activation areas.Afterwards, we sum thec-dimensional feature maps in the channel direction and the resulting feature response map have the characteristics of large response values in the target area and small response values in the nontarget area. Mathematically, this function can be defined as φ:Rc×h×w⟶Rh×w, so φX=G, where G=∑k=1cXk. After this, we find the mean value g′ of all positions on a map G and use it as the threshold for coarse target positioning. In particular, if the activation response value of location i,j is higher than g′, it means that the location is the main target. Then, a mask map M of the same size G can be obtained:(15)Mi,j=1,ifGi,j>g′,0,otherwise,where i,j is a particular position in these h∗w positions.Use tensorX and mask M as a mask operation. Get the three-dimensional feature map X′. Then, the feature X′ is aggregated by a global max pooling and a global average pooling.The calculation formula of the channel weight is as follows, and we first count the number of response values larger than 0 for each channel of theX′ feature map:(16)Qk=∑ij1Xij′k>0,where Xij′k represents the value of the k-th channel at spatial location i,j and Qk represents the number of response values larger than 0 for the k-th channel.We use the log function to makeWk smaller when Qk is large, achieving the effect of giving more weight to channels with fewer activation areas:(17)Wk=log∑k=1k=cQkε+Qk,where ε is a minimal number to ensure that the denominator is not 0.We have briefly described the algorithm flow of this paper in Algorithm1.Algorithm 1: Multiproxies adaptive distribution loss with weakly supervised feature aggregation.
Input: training data:Dtrain, testing data: Dtest, Pretrained CNN model: CNNω,bOutput: testing data features：T′=t1′,t2′,……,ti′Minibatch default:The size of every minibatch isB, the embedding output of minibatch samples can be defined asS=s1=f1S,y1S,s2=f2S,y2S,……,sB=fBS,yBS; the proxies can be defined as P=p1=f1p,y1p,p2=f2p,y2p,……,pEfEp,yEp}Hyperparameters default:M is the number of blocks divided by the histogram, α>0 is a scaling parameter, β>0 is a margin parameter, ρ>0 is the fusion parameterMultiproxies adaptive distribution scheme:Calculate the proxies’ number according to equations (1) and (2)Process:For every epoch, doCalculate the loss:LMPADLP,S=Lp−sp,s+,s−+ρLp−pp,p+,p−Update CNN modelCNNω,b by every epoch:ω=ω−μ∂LMPADLP,S/∂ωb′=b−μ∂LMPADLP,S/∂ωEndFori in Dtest doX=fine−tuned CNNIG=∑k=1cXkGet mask map by equation (15).Use tensorX and mask M as a mask operationGet the three-dimensional feature mapX′ti = aggregate feature map X′ by GAP&GMPti′ = channel weighting by equation (17).EndFirstly, we use the multiproxies adaptive distribution loss in the training phase to fine-tuned the CNN model. Secondly, we use the fine-tuned CNN model in the testing phase to get the convolutional feature map. Then, we use a weakly supervised aggregation model to get a more representative feature descriptor. Finally, we use the feature descriptor to calculate similarity and the Recall@K in the fine-tuned image retrieval.


## 3.1. Loss Motivation

Recalling the previous pair-based loss, it depends on the sample pairs constructed by the samples in each batch. The polynomial increase in the number of sample pairs will affect the efficiency of loss, and many sample pairs have very little information, some even do not generate gradients. The effect of loss will largely depend on the sample mining strategy, but even if a very effective sample mining strategy is developed, the loss training is essentially limited to sample pairs in a batch, and the global information of the embedding space cannot be obtained.Proxy-based losses, such as proxy nca loss and center loss, solve the previous problem of pair-based losses by distributing a single center or proxy to each class. They distribute a trainable parameter to each class so that the loss training takes into account the global class information, avoiding the sampling problem in each batch.However, there are still two problems with proxy-based losses. (1) They all distribute a trainable parameter, which is a single proxy or a single center to each class. However, in fine-grained retrieval tasks, a significant challenge is the large intraclass variance. Distributing a single proxy to each class is not enough to characterize the complex intraclass variations of the fine-grained dataset, and distributing multiple proxies to each class is not necessary for classes with small intraclass variance and it just will increase the amount of calculation. (2) They only consider increasing the similarity between the same class of proxies and samples and reducing the similarity between the different classes of proxies and samples so that the final embedding space realizes the clustering of each class, but it cannot ensure that the final cluster centers are scattered. If the proxies of each class cannot achieve a great separation, in the end, the optimal embedding space will not be obtained.As shown in Figure3, the upper and lower rows belong to the same class in the fine-grained dataset, but it does not make sense to cluster them excessively so that the brown information will be mixed with the red information to represent it. But in reality, among the anonymous data, there is only a tiny probability that brown information and red information are mixed in birds of the same class. Therefore, this kind of mixed information is not universal (as opposed to only learning brown information with angle change).Figure 3
CUB200-2011 dataset-cardinal class.As shown in Figure1, we propose a new metric learning loss called multi-proxies adaptive distribution loss to improve previous losses. Our loss first distributes the best number of proxies to each class of the training set through a multiproxies adaptive distribution scheme, which reduces the amount of calculation while better characterizing the intraclass variations. Secondly, we consider the similarity between multiple proxies so that the final embedding space of different-class proxies can be far away from each other, and the same-class proxies can be gathered together to a certain extent. Our loss significantly improves the quality of the embedding space.


## 3.2. Feature Aggregation Motivation

The selective convolutional descriptor aggregation (SCDA) [3] proposed by Wei et al. locates the target area from the spatial dimension according to the response map. The implementation of this scheme will cause the channel with a lot of 0 values and many channels to have fewer activation areas.As shown in Figure4, for each image in the test set, we will get the feature map of each channel through the fine-tuned neural network. After summing all channel feature maps, the response map will be obtained. According to the characteristics of the convolutional neural network, where the large response value is the target area and the area with the small response value is the background area, we can get the mask matrix according to the response map to locate the target area. After each channel feature map is selected by mask matrix, we will get channel scda maps. Our detailed analysis of each channel feature map of the obtained channel scda maps can find that some features are fewer active areas, such as channel-27 scda map and channel-52 scda map, activating the information of the bird’s tail and head. Fine-grained retrieval tasks often use this subtle feature to distinguish different classes of birds. Therefore, it does not mean that the activation area is less and the feature channel is not essential. Often, the response value of this channel after sum pooling or average pooling is small, so we give more weight to the feature channel with less activation area.Figure 4
(a) Original image randomly selected from the CUB200-2011 dataset test set. (b) The response map obtained by the fine-tuned convolutional neural network. (c) A part of the channel feature maps obtained by the fine-tuned convolutional neural network. (d) The channel feature maps obtained by the scda algorithm. Inside the gray dashed box are some representative channel scda maps.We proposed a weakly supervised aggregation model. The channel dimensions are weighted for the obtained feature descriptors, and the number of response values of each feature channel larger than 0 is counted, for channels with fewer response values larger than 0, which have essential information channels to give more weight and get more representative feature descriptors.We have achieved a more prominent effect on fine-grained retrieval tasks through multiproxies adaptive distribution loss and weakly supervised feature aggregation methods.


## 3.3. Multiproxies Adaptive Distribution Scheme

As shown in Figure5, for some classes of fine-grained datasets, such as Class A, simply distributing one proxy is not enough, but for some classes with small intraclass variance, distributing multiple proxies is redundant; it just will increase the amount of calculation, such as Class B. In order to solve the problem of distributing the number of proxies in each class, we propose a method of distributing proxies based on the average intraclass distance.Figure 5
CUB200-2011 dataset.
(a)(b)Firstly, we obtain the feature descriptors of each class of images in the training set through the fine-tuned convolutional neural network ResNet50. Secondly, for each class in the training set, we use the following formula to calculate the average intraclass distance:(1)dc=1nc2−nc∑i∈c,j∈cfi−fj2,where nc represents the number of images in class C and fi represents the image feature descriptor obtained by fine-tuned convolutional neural network for the i-th image in class C.By obtaining the average intraclass distance, we adaptively distribute the number of proxies for each class in the training set. the larger the average intraclass distance, the larger the intraclass variance of the class. For maintaining this intraclass variance, we distribute more proxies to the class with the large average intraclass distance. On the contrary, for the class with a small intraclass distance, we distribute fewer proxies. In this manner, the use of limited proxies can be achieved while maintaining the larger intraclass variance.Thirdly, our paper adopts the method of establishing a histogram. We determine the width of each rectangle in the histogram according to the interval of the average intraclass distances of all classes in the training set, build a histogram of the average intraclass distances of all training set classes, and divide the average intraclass distance intoM blocks. Then, the calculation formula for the range V of the value contained in each block is as follows:(2)V=dmax−dminM,where M represents the number of blocks divided by the histogram, dmax represents the maximum intraclass average distance of all classes in the training set, and dmin represents the smallest average intraclass distance of all classes in the training set. The ranges of M block values areA1=dmin,dmin+V, A2=dmin+V,dmin+2V……AM=dmin+M−1V,dmin+MV.By calculating the average intra-class distance of each class in the training set, we distribute a different number of proxies to each class. The training set classes belonging to the rangeA1 are distributed k proxies, the training set classes belonging to the range A2 are distributed k+1 proxies, and so on, the training set class belonging to the range AM is distributed k+M−1 proxies. Among them, k=1.According to the above rules, before training, we first adaptively distribute the number of proxies for all training set classes for subsequent loss training.


## 3.4. Multiproxies Adaptive Distribution Loss

The main idea of our multiproxies adaptive distribution loss allocation is as follows: (1) we adaptively distribute the number of proxies for all classes according to the average intraclass distance of all classes in the training set; (2) we propose a new loss form divided into two parts: the first partLp−sp,s+,s− is used to increase the similarity between each proxy with the same class samples and reduce the similarity between each proxy with different classes samples. For each pair of proxy and sample, the size of the gradient is determined according to their self-similarity and relative similarity. The final embedding space makes the samples of the same class cluster together. The second part Lp−pp,p+,p− entropy loss is used to realize the approach and separation of multiproxies, increase the similarity between the same-class proxies, and reduce the similarity between different-class proxies. The final embedding space makes the samples of the same class cluster together and the cluster centers of each class separate from each other.The training set hasH classes, and each batch has B samples, representing the feature descriptor of the i-th sample as fiS and its corresponding label as yiS; the feature descriptors output of minibatch samples can be defined as S=s1=f1S,y1S,s2=f2S,y2S,…,sB=fBS,yBS. Besides, we use the multiproxies adaptive distribution scheme in Section 3.3 to distribute the number of proxies offline for the training set. Our proxies are trainable network layer parameters. Suppose we distribute E proxies offline adaptively. Representing the feature descriptor of the i-th proxy as fiP and its corresponding label as yiP. The proxy settings can be represented by P=p1=f1p,y1p,p2=f2p,y2p,……,pE=fEp,yEp. We create a label matrix Lab for E proxies, the size of the label matrix Lab is E×H, the number of rows represents E proxies, and the number of columns represents the one-hot label of each proxy in H classes, used to represent the proxy class. For the i,j position of the Lab label matrix, if yip=j, Labij=1; else, Labij=0.Our loss is divided into two parts, as shown in Figure 1(f);Lp−sp,s+,s− considers the relationship between samples and proxies, using adaptively distributed E proxies as anchors, samples of the same class are considered positive, and samples of different classes are considered negative.A batch of embedding samplesS is divided into two sets: Sp+ and Sp−. For a selected proxy p, samples of the same class as in a batch belong to the set Sp+. And samples of a different class belong to the set Sp−. Lposp,s+ improves the similarity between proxies and samples in the same class. Lposp,s+ is then given by(3)Lposp,s+=1P∗∑p∈P∗log1+∑s+∈Sp+e−αsimp,s+−β,where α>0 is a scaling parameter, β>0 is a margin parameter, sim.,. denotes the cosine similarity between a proxy and a sample, P∗ represents the set of proxies with positive samples in the batch. If the number of all proxies is directly used as the coefficient and the batch does not contain samples of the class proxies, the mean will become lower.Lposp,s− reduces the similarity between different-class proxies and samples. The specific calculation formula is as follows:(4)Lnegp,s−=1P∑p∈Plog1+∑s−∈Sp−eαsimp,s−+β.The number of all proxies is directly used as the coefficient because for random sampling, the probability that all samples in a batch belong to the same class is negligible, so there will be negative samples for all proxies.The lastLp−sp,s+,s− function can be achieved by equations (3) and (4). Therefore, all proxies are capable of increasing the similarity between positive samples and reducing the similarity between negative samples. To be further, it will make the same-class samples gather together and different-class samples stay away from each other:(5)Lp−sp,s+,s−=Lposp,s++Lnegp,s−.In equation (3),(6)log1+∑s+∈Sp+e−αsimp,s+−β=log1+elog∑s+∈Sp+e−αsimp,s+−β=soft pluslog∑s+∈Sp+e−αsimp,s+−β=soft plusLSE−αsimp,s+−β=maxmax−αsimp,s+−β,0.Among them, Softplus:log1+ex=maxx,0 LSE::log∑ex=maxx.Similarly, in equation (4),(7)log1+∑s−∈Sp−eαsimp,s−+β=maxmaxαsimp,s−+β,0.For equation (6), when −αsimp,s+−β<0, it is simp,s+>β.For equation (7), when αsimp,s−+β<0, it is simp,s−<−β.log1+∑s+∈Sp+e−αsimp,s+−β=0, log1+∑s−∈Sp−eαsimp,s−+β=0, and Loss is 0 without training gradient.The training of the network can make the similarity between proxies and positive samples larger thanβ, and β is the decision boundary that determines the final similarity. Through the loss of equations (3) and (4), the effect of simp,s+>β, simp,s−<−β can finally be realized.Because of the size of the derivative considering all the samples' hardness, we take the derivative ofLp−sp,s+,s−. For hard samples, we use a larger thrust, that is, a larger derivative:(8)∂Lposp,s+∂simp,s+=1P+−αe−αsimp,s+−β1+∑s+∈Sp+e−αsimp,s+−β,(9)∂Lnegp,s−∂simp,s−=1P−αeαeαsimp,s−+β1+∑sk∈Sp−eαsimp,s−+β.According to equations (8) and (9), we can know that the gradient of Lposp,s+ decreases as the self-similarity of simp,s+ increases; that is, more dissimilar positive samples are pulled closer with larger strength, as the relative similarity of ∑s+∈Sp+e−αsimp,s+−β increases. In contrast, the gradient of Lnegp,s− increases as the self-similarity of simp,s− increases; that is, more similar negative samples retreat with greater force. As the relative similarity of ∑sk∈Sp−eαsimp,s−+β increases, it decreases.ForLp−pp,p+,p− considering the relationship between each proxy with the other proxies, the adaptively distributed E proxies are regarded as samples processing, the same-class proxies are considered to be positive, and different-class proxies are considered to be negative. We use classification-based Softmax and cross-entropy loss deformation to obtain Lp−pp,p+,p−.The commonly used Softmax and cross-entropy loss [47–49] form are as follows:(10)Pryi=eyi∑jeyj,LCE=−1N∑i=1NlogPryi.InLp−pp,p+,p−, proxies P are divided into two sets: P+ and P−. For a selected proxy p, the proxies of the same class belong to the set P+. And the samples of a different class belong to the set P−. Lp−pp,p+,p− makes the same-class proxies closer and the different-class proxies pushed farther:(11)Lp−pp,p+,p−=∑p∈P−log∑p+∈P+esimp,p+∑p−∈P−esimp,p−,where sim.,. denotes the cosine similarity between a proxy and the other proxy.We take the derivative ofLp−pp,p+,p−, the size of the derivative considering all the samples’ hardness. For hard samples, we use a larger thrust, which is a larger derivative:(12)∂Lp−pp,p+,p−∂simp,p+=−esimp,p+∑p+∈P+esimp,p+,(13)∂Lp−pp,p+,p−∂simp,p−=esimp,p−∑p−∈P−esimp,p−.According to equations (13) and (14), we can know that the gradient of the positive proxy pair decreases with the increase of simp,p+; that is, the more dissimilar positive proxy pairs are pulled closer with a larger force. The gradient of the negative proxy pair increases with the increase of simp,p−; that is, the more similar negative proxy pairs retreat with a larger force.In multiproxies adaptive distribution loss, we useLp−s to optimize the similarity between proxies and samples and use Lp−p to optimize the similarity between proxies and proxies:(14)LMPADLP,S=Lp−sp,s+,s−+ρLp−pp,p+,p−,where P are the adaptive distribution proxies. S is a set of real samples in a batch, and ρ>0 is the fusion parameter. In the training phase, we obtained a better embedding space and fine-tuned network parameters by minimizing LMPADLP,S for the generation of image descriptors during testing.


## 3.5. Weakly Supervised Aggregation Deep Convolutional Features

During the test, we obtain the three-dimensional convolution feature maps of all test images through the convolutional neural network fine-tuned by multiproxies adaptive distribution loss and then aggregate the three-dimensional convolution features into one-dimensional image features descriptor through our improved weakly supervised aggregation model for subsequent similarity measurement.For a given imageI (height m width n) and a CNN model, we input the image I into the fine-tuned ResNet50 model for forwarding propagation and obtain the three-dimensional feature map of the last convolutional layer X∈Rc×h×w.We first coarsely localize the object via SCDA [3] based on selective convolutional features and then obtain one-dimensional feature descriptors through GAP and GMP. Due to the characteristics of SCDA [3], channels with fewer activation areas are also vital. We weight the channels of the obtained one-dimensional descriptors and assign large weights to the channels with fewer activation areas.Afterwards, we sum thec-dimensional feature maps in the channel direction and the resulting feature response map have the characteristics of large response values in the target area and small response values in the nontarget area. Mathematically, this function can be defined as φ:Rc×h×w⟶Rh×w, so φX=G, where G=∑k=1cXk. After this, we find the mean value g′ of all positions on a map G and use it as the threshold for coarse target positioning. In particular, if the activation response value of location i,j is higher than g′, it means that the location is the main target. Then, a mask map M of the same size G can be obtained:(15)Mi,j=1,ifGi,j>g′,0,otherwise,where i,j is a particular position in these h∗w positions.Use tensorX and mask M as a mask operation. Get the three-dimensional feature map X′. Then, the feature X′ is aggregated by a global max pooling and a global average pooling.The calculation formula of the channel weight is as follows, and we first count the number of response values larger than 0 for each channel of theX′ feature map:(16)Qk=∑ij1Xij′k>0,where Xij′k represents the value of the k-th channel at spatial location i,j and Qk represents the number of response values larger than 0 for the k-th channel.We use the log function to makeWk smaller when Qk is large, achieving the effect of giving more weight to channels with fewer activation areas:(17)Wk=log∑k=1k=cQkε+Qk,where ε is a minimal number to ensure that the denominator is not 0.We have briefly described the algorithm flow of this paper in Algorithm1.Algorithm 1: Multiproxies adaptive distribution loss with weakly supervised feature aggregation.
Input: training data:Dtrain, testing data: Dtest, Pretrained CNN model: CNNω,bOutput: testing data features：T′=t1′,t2′,……,ti′Minibatch default:The size of every minibatch isB, the embedding output of minibatch samples can be defined asS=s1=f1S,y1S,s2=f2S,y2S,……,sB=fBS,yBS; the proxies can be defined as P=p1=f1p,y1p,p2=f2p,y2p,……,pEfEp,yEp}Hyperparameters default:M is the number of blocks divided by the histogram, α>0 is a scaling parameter, β>0 is a margin parameter, ρ>0 is the fusion parameterMultiproxies adaptive distribution scheme:Calculate the proxies’ number according to equations (1) and (2)Process:For every epoch, doCalculate the loss:LMPADLP,S=Lp−sp,s+,s−+ρLp−pp,p+,p−Update CNN modelCNNω,b by every epoch:ω=ω−μ∂LMPADLP,S/∂ωb′=b−μ∂LMPADLP,S/∂ωEndFori in Dtest doX=fine−tuned CNNIG=∑k=1cXkGet mask map by equation (15).Use tensorX and mask M as a mask operationGet the three-dimensional feature mapX′ti = aggregate feature map X′ by GAP&GMPti′ = channel weighting by equation (17).EndFirstly, we use the multiproxies adaptive distribution loss in the training phase to fine-tuned the CNN model. Secondly, we use the fine-tuned CNN model in the testing phase to get the convolutional feature map. Then, we use a weakly supervised aggregation model to get a more representative feature descriptor. Finally, we use the feature descriptor to calculate similarity and the Recall@K in the fine-tuned image retrieval.


## 4. Experiments and Evaluation

In this section, in order to verify the rationality of the multiproxies adaptive distribution loss and the weakly supervised aggregation model we designed, we conducted comparative experiments. We validated the effectiveness of our method on CUB200-2011 [5] and Stanford Dog [6]. The experimental outcomes reveal the multiproxies adaptive distribution loss and the weakly supervised aggregation model we proposed to achieve the state of the art.


### 4.1. Dataset

In this paper, we use the universal benchmark fine-grained retrieval datasets CUB200-2011 [5] and Stanford Dog [6].The CUB200-2011 [5] contains 200 bird classes and a total of 11788 images. We use 5864 images of the first 100 classes as the training set and the remain images of the last 100 classes as the test set.The Stanford Dog [6] contains 120 dog classes and a total of 20,580 images. We use 10651 images of the first 60 classes as the training set. 9929 images of the last 60 classes are used as the test set.The FGVC aircraft [50] contains 10,000 images of aircraft, with 100 images for each of the 100 different aircraft model variants, most of which are airplanes. We use 5000 images of the first 50 classes as the training set and the remaining images of the last 50 classes as the test set.The Oxford Pets [51] is a 37 classes pet image dataset with roughly 200 images for each class. The images have large variations in scale, pose, and lighting. We use 3800 images of the first 19 classes as the training set. 3790 images of the last 19 classes are used as the test set.


### 4.2. Evaluation Protocols

The same as the previous works, to evaluate image retrieval performance, we use the standard Recall@K.Recall@K means, given a specific numberK of first returned images in image retrieval for each query image, whether there is a correct image to the number of returned images. Obviously, if there is more than one, the recall score is 1. Conversely, the recall score is 0. And then, we calculate the average recall rate of all query images.


### 4.3. Implementation Details

In our experiments, the experiments in ResNet50 are based on a RIX-2080Ti GPU, and the experiments in ResNet101 [52] are based on a Tesla T4 GPU. We used the common network structures ResNet50 and ResNet101 and initialized them with the network parameters pretrained by ImageNet ILSVRC-2012 [53]. We train our models with an SGD optimizer. We use the same parameters with a minibatch size of 60, and we use the learning rate of 0.0001; a scale parameters γ is 128.


### 4.4. Comparison with the Baseline

We compare our proposed method with several state-of-the-art fine-grained image retrieval methods including the following: (1) contrastive loss [21], (2) triplet loss [1], (3) lifted structured loss [46] (4), facility location [54] (5), N-pair loss [41] (6), proxy NCA loss [45], (7) binomial deviance [55], (8) histogram loss [29], (9) PDDM + quadruplet [56], (10) SCDA [3] proposing an unsupervised convolutional feature aggregation method, which locates objects based on the size of the response value and then aggregates to obtain a more representative image descriptor. (11) CRL-WSL [2] makes parallel efforts in a centralized ranking loss and weakly supervised localization method. Obviously, due to a center distributed a class, the centralized ranking loss is similar to the proxy-based loss. Moreover, to obtain a more representative image feature descriptor, it utilizes the weakly supervised localization method. (12) By means of adding a normalized scale layer, the DGCRL [33] unifies the inner product and Euclidean distance. Simultaneously, the DGCRL makes the centers of each class orthogonal with the Gram–Schmidt optimization so as to obtain some best space of centers. (13) PCEL [36] proposes a piecewise cross-entropy loss to solve the problem of model overfitting. The specific results on the CUB200-2011 are shown in Table 1.Table 1
Recall@K on the CUB-200-2011 dataset with baseline methods.
MethodCUB200-2011 (Recall@K)K =12481632Contrastive [21]26.437.749.862.376.485.3Triplet [1]36.148.659.370.080.288.4Lifted struct. [46]47.258.970.280.289.393.2Facility location [54]48.261.471.881.9——N-pairs [41]45.3758.4169.5179.49——Proxy NCA [45]49.2161.9067.9072.40——Binomial deviance [55]52.864.474.783.990.494.3Histogram loss [29]50.361.972.682.488.893.7PDDM + quadruplet [56]58.369.279.088.493.195.7SCDA [3]62.274.283.290.194.397.3CRL-WSL [2]65.976.585.390.394.497.0DGCRL (ResNet50) [33]67.979.186.291.894.897.1PCEL (ResNet50) [36]70.179.886.992.095.097.3PCEL (ResNet101)71.280.287.392.495.397.3Ours (ResNet50)71.8880.8788.1892.9895.9397.59Ours (ResNet101)72.9782.6689.1893.3095.7697.50The best result is highlighted in bold.As shown in Table1, our model outperforms the state of the art in terms of Recall@K on the CUB200-2011. In our method, under the ResNet50 network model, the recall 1 of the CUB200-2011 dataset reached 71.88. Under the ResNet101 network model, the Recall@1 of the CUB200-2011 dataset reached 72.97.From the results in Table1, we can find that we adaptively distribute the number of proxies for each class according to the average intraclass distance and separate different classes of proxies from each other through cross-entropy loss, and by judging the importance of each channel of the three-dimensional feature map, assigning different weights to each channel can make the three-dimensional feature map better represent the information contained in the image. Our method achieves a higher accuracy rate under the evaluation standard of Recall@K.For the richness of the experiment, we also conducted the Recall@K test on the Stanford Dog dataset. We selected the state-of-the-arts algorithms to get the value of the Stanford Dog dataset Recall@K and compared it with our method.As shown in Table2, our model outperforms the state of the art in terms of Recall@K on the Stanford Dog. In our method, under the ResNet50 network model, the Recall@1 of the Stanford Dog dataset reached 87.60. Under the ResNet101 network model, the Recall@1 of the Stanford Dog dataset reached 89.43.Table 2
Recall@K on the Stanford Dog dataset with baseline methods.
MethodStanford Dog (Recall@K)K =1248SCDA74.95———DGCRL (ResNet50)86.7692.6695.8497.79PCEL (ResNet50)87.0392.7095.9997.97PCEL (ResNet101)88.6493.4196.3797.97Ours (ResNet50)87.6092.8696.1197.92Ours (ResNet101)89.4393.8796.4698.08The best result is highlighted in bold.At the same time, under the ResNet50 network model, we also conducted a comparative experiment on the datasets FGVC aircraft [50] and Oxford Pets [51] with the previous algorithm.As shown in Table3, our model outperforms the state of the art in terms of Recall@1 on the FGVC aircraft. In our method, under the ResNet50 network model, the Recall@1 of the Stanford Dog dataset reached 70.06.Table 3
Recall@K on the FGVC aircraft dataset with baseline methods.
MethodFGVC aircraft (Recall@K)K =1248DGCRL (ResNet50)68.6478.6086.3691.88PCEL (ResNet50)68.8479.2287.0492.24Ours (ResNet50)70.0680.9088.6693.66The best result is highlighted in bold.As shown in Table4, our model outperforms the state of the art in terms of Recall@1 on the Oxford Pets. In our method, under the ResNet50 network model, the Recall@1 of the Stanford Dog dataset reached 98.31.Table 4
Recall@K on the Oxford Pets dataset with baseline methods.
MethodOxford Pets (Recall@K)K =1248PCEL (ResNet50)97.0998.7399.4799.73DGCRL (ResNet50)97.8699.1099.6399.78Ours (ResNet50)98.3199.1699.5599.79The best result is highlighted in bold.For the purpose of better exhibiting our retrieval results, we visualize the reducing method called t-SNE to visualize the embedding vectors produced by our proposed method on CUB200-2011 and Stanford Dog. According to Figures6 and 7, it is known that our proposed method works very well in grouping similar images in spite of the viewpoint and posture that have changed.Figure 6
Visualization of the embedding space obtained using our method on the CUB200-2011 dataset.Figure 7
Visualization of the embedding space obtained using our method on the Stanford Dog dataset.


### 4.5. Ablation Experiment

In this section, we test the effectiveness of the proposed models, the multiproxies adaptive distribution loss, the multiproxies adaptive distribution scheme, and the channel weighting in the weakly supervised aggregation model. The loss we chose as the baseline comparison is the DGCRL proposed by Zheng et al. because we used the same network model, batch size, optimizer, input size, and feature dimension. What is more, SCDA operations were performed in the testing step.As shown in Table5, our multiproxies adaptive distribution loss, multiproxies adaptive distribution scheme, and the channel weighting in the weakly supervised aggregation model all have positive effects on Recall@K. Compared to the DGCRL loss, our loss form has increased by 1.85% on Recall@1. The multiproxies adaptive distribution scheme we proposed can increase by 1.92% with our loss form. And the aggregation method we proposed to perform channel weighting after SCDA improves the effect in many cases. The best effect is achieved when the three are combined.Table 5
Recall@K with different models on our methods.
Method (ResNet50)CUB-200-2011 (Recall@K)K =1248DGCRL (ResNet50)67.979.186.291.8MADL (ResNet50)69.7579.9087.0792.49MADL +V (ResNet50)70.1780.1187.3492.74MADL + MADS (ResNet50)71.6780.8988.0992.99MADL + MADS +V (ResNet50)71.8880.8788.1892.98MADL + MADS (ResNet101)72.8282.3489.0493.29MADL + MADS +V (ResNet101)72.9782.6689.1893.30MADL is the multiproxies adaptive distribution loss. MADS is the multiproxies adaptive distribution scheme.V is the channel weighting in the weakly supervised aggregation model. The best result is highlighted in bold.


### 4.6. Parameter Analysis

In this section, we tested the adjustable parameters in the paper. They are the number of divisionsM of the histogram in the multiproxies adaptive distribution scheme. α, β, and ρ in the multiproxies adaptive distribution loss: α is a scaling parameter, β is a margin parameter, ρ is the fusion parameter.As shown in Table6, we can regard M=1 as a single proxy for each class. Through Table 6, we can find that when M>1, the value of Recall@K is higher than M=1. When M=2, the value of Recall@K is generally the highest. It can be explained that compared to a single proxy, the multiproxies adaptive distribution scheme we proposed is effective. But it is not that the more proxies are distributed, the better the result is. When M=2, the value of Recall@K generally achieves the best results.Table 6
Recall@K with differentM on CUB200-2011.
Method (ResNet50)CUB200-2011 (Recall@K)MADL124816M = 170.1780.1187.3492.7495.77M = 271.8880.8788.1892.9895.93M = 371.0281.1487.8392.8695.91M = 470.5180.1387.4692.8095.86TheM is the number of blocks divided by the histogram in a multiproxies adaptive distribution scheme. The best result is highlighted in bold.We also investigate the effect of the two hyperparametersα and β of our loss on the CUB200-2011 dataset.As shown in Figure8, we tested the parameters α and β in the multiproxies adaptive distribution loss. α is a scaling parameter, and β is a margin parameter. We tested when α is equal to 4, 8, 16, 32, and 64. When β is equal to 0, 0.1, 0.2, and 0.3, the value of Recall@1 on the CUB-200-2011 dataset, the experimental results show that when α=32 and β=0.1, the retrieval achieves the best results.Figure 8
Recall@1 with differentα and β on CUB-200-2011. α is a scaling parameter, and β is a margin parameter in multiproxies adaptive distribution loss.Finally, we testedρ as the fusion parameter.As shown in Table7, ρ is the fusion parameter in the multiproxies adaptive distribution loss. When ρ=0.1, the value of Recall@1 achieves the best results.Table 7
Recall@K with differentρ on CUB200-2011.
Method (ResNet50)CUB200-2011 (Recall@K)MPADL124816ρ = 0.0570.9081.0687.7692.6095.83ρ = 0.171.8880.8788.1892.9895.93ρ = 0.270.7880.3387.6892.7895.95ρ = 0.370.5380.2787.5692.8395.90ρ = 0.670.6680.3587.7692.9695.71ρ = 0.871.5681.1488.0892.9796.07ρ = 0.970.2280.2588.0892.8395.85ρ = 1.071.3081.0287.7192.5696.08ρ is the fusion parameter in multiproxies adaptive distribution loss. The best result is highlighted in bold.Through the parameter test, we have selected the parameters that are more suitable for the algorithm to achieve better results. In the multiproxies adaptive distribution scheme, the larger the number of divisionsM of the histogram is, the more proxies will be distributed to each class. When M=1, it is equivalent to the case of a single proxy for each class. The experimental results found that the idea of distributing multiple proxies to each class can achieve better retrieval results because multiple proxies can maintain the differences within the class. But M is not the bigger, the better, because a certain number of proxies can distinguish the differences within the classification. Too many divisions will only increase the amount of calculation and make the aggregated features too scattered. It is found through testing that the best effect will be achieved when M=2. α is the scaling parameter in the loss function. Experiments have found that the best effect is achieved when the scaling ratio is 32, and β is the margin parameter in the loss function, which determines how much the similarity threshold reaches and stops training. Since the range of cosine similarity is −1 to 1, the value of β is between 0 and 1. We find through testing that the best effect is achieved when β=0.1, that is, the similarity between positive samples is greater than 0.1, and the similarity between negative samples is less than −0.1, achieving the ideal clustering effect. ρ is the fusion parameter, which determines the fusion ratio of the loss between the proxy and the proxy and the loss between the proxy and the sample. Because the object is different, the value range is different, and the one-to-one fusion cannot be directly performed. The test found that ρ=0.1; the best retrieval effect can be achieved.


## 4.1. Dataset

In this paper, we use the universal benchmark fine-grained retrieval datasets CUB200-2011 [5] and Stanford Dog [6].The CUB200-2011 [5] contains 200 bird classes and a total of 11788 images. We use 5864 images of the first 100 classes as the training set and the remain images of the last 100 classes as the test set.The Stanford Dog [6] contains 120 dog classes and a total of 20,580 images. We use 10651 images of the first 60 classes as the training set. 9929 images of the last 60 classes are used as the test set.The FGVC aircraft [50] contains 10,000 images of aircraft, with 100 images for each of the 100 different aircraft model variants, most of which are airplanes. We use 5000 images of the first 50 classes as the training set and the remaining images of the last 50 classes as the test set.The Oxford Pets [51] is a 37 classes pet image dataset with roughly 200 images for each class. The images have large variations in scale, pose, and lighting. We use 3800 images of the first 19 classes as the training set. 3790 images of the last 19 classes are used as the test set.


## 4.2. Evaluation Protocols

The same as the previous works, to evaluate image retrieval performance, we use the standard Recall@K.Recall@K means, given a specific numberK of first returned images in image retrieval for each query image, whether there is a correct image to the number of returned images. Obviously, if there is more than one, the recall score is 1. Conversely, the recall score is 0. And then, we calculate the average recall rate of all query images.


## 4.3. Implementation Details

In our experiments, the experiments in ResNet50 are based on a RIX-2080Ti GPU, and the experiments in ResNet101 [52] are based on a Tesla T4 GPU. We used the common network structures ResNet50 and ResNet101 and initialized them with the network parameters pretrained by ImageNet ILSVRC-2012 [53]. We train our models with an SGD optimizer. We use the same parameters with a minibatch size of 60, and we use the learning rate of 0.0001; a scale parameters γ is 128.


## 4.4. Comparison with the Baseline

We compare our proposed method with several state-of-the-art fine-grained image retrieval methods including the following: (1) contrastive loss [21], (2) triplet loss [1], (3) lifted structured loss [46] (4), facility location [54] (5), N-pair loss [41] (6), proxy NCA loss [45], (7) binomial deviance [55], (8) histogram loss [29], (9) PDDM + quadruplet [56], (10) SCDA [3] proposing an unsupervised convolutional feature aggregation method, which locates objects based on the size of the response value and then aggregates to obtain a more representative image descriptor. (11) CRL-WSL [2] makes parallel efforts in a centralized ranking loss and weakly supervised localization method. Obviously, due to a center distributed a class, the centralized ranking loss is similar to the proxy-based loss. Moreover, to obtain a more representative image feature descriptor, it utilizes the weakly supervised localization method. (12) By means of adding a normalized scale layer, the DGCRL [33] unifies the inner product and Euclidean distance. Simultaneously, the DGCRL makes the centers of each class orthogonal with the Gram–Schmidt optimization so as to obtain some best space of centers. (13) PCEL [36] proposes a piecewise cross-entropy loss to solve the problem of model overfitting. The specific results on the CUB200-2011 are shown in Table 1.Table 1
Recall@K on the CUB-200-2011 dataset with baseline methods.
MethodCUB200-2011 (Recall@K)K =12481632Contrastive [21]26.437.749.862.376.485.3Triplet [1]36.148.659.370.080.288.4Lifted struct. [46]47.258.970.280.289.393.2Facility location [54]48.261.471.881.9——N-pairs [41]45.3758.4169.5179.49——Proxy NCA [45]49.2161.9067.9072.40——Binomial deviance [55]52.864.474.783.990.494.3Histogram loss [29]50.361.972.682.488.893.7PDDM + quadruplet [56]58.369.279.088.493.195.7SCDA [3]62.274.283.290.194.397.3CRL-WSL [2]65.976.585.390.394.497.0DGCRL (ResNet50) [33]67.979.186.291.894.897.1PCEL (ResNet50) [36]70.179.886.992.095.097.3PCEL (ResNet101)71.280.287.392.495.397.3Ours (ResNet50)71.8880.8788.1892.9895.9397.59Ours (ResNet101)72.9782.6689.1893.3095.7697.50The best result is highlighted in bold.As shown in Table1, our model outperforms the state of the art in terms of Recall@K on the CUB200-2011. In our method, under the ResNet50 network model, the recall 1 of the CUB200-2011 dataset reached 71.88. Under the ResNet101 network model, the Recall@1 of the CUB200-2011 dataset reached 72.97.From the results in Table1, we can find that we adaptively distribute the number of proxies for each class according to the average intraclass distance and separate different classes of proxies from each other through cross-entropy loss, and by judging the importance of each channel of the three-dimensional feature map, assigning different weights to each channel can make the three-dimensional feature map better represent the information contained in the image. Our method achieves a higher accuracy rate under the evaluation standard of Recall@K.For the richness of the experiment, we also conducted the Recall@K test on the Stanford Dog dataset. We selected the state-of-the-arts algorithms to get the value of the Stanford Dog dataset Recall@K and compared it with our method.As shown in Table2, our model outperforms the state of the art in terms of Recall@K on the Stanford Dog. In our method, under the ResNet50 network model, the Recall@1 of the Stanford Dog dataset reached 87.60. Under the ResNet101 network model, the Recall@1 of the Stanford Dog dataset reached 89.43.Table 2
Recall@K on the Stanford Dog dataset with baseline methods.
MethodStanford Dog (Recall@K)K =1248SCDA74.95———DGCRL (ResNet50)86.7692.6695.8497.79PCEL (ResNet50)87.0392.7095.9997.97PCEL (ResNet101)88.6493.4196.3797.97Ours (ResNet50)87.6092.8696.1197.92Ours (ResNet101)89.4393.8796.4698.08The best result is highlighted in bold.At the same time, under the ResNet50 network model, we also conducted a comparative experiment on the datasets FGVC aircraft [50] and Oxford Pets [51] with the previous algorithm.As shown in Table3, our model outperforms the state of the art in terms of Recall@1 on the FGVC aircraft. In our method, under the ResNet50 network model, the Recall@1 of the Stanford Dog dataset reached 70.06.Table 3
Recall@K on the FGVC aircraft dataset with baseline methods.
MethodFGVC aircraft (Recall@K)K =1248DGCRL (ResNet50)68.6478.6086.3691.88PCEL (ResNet50)68.8479.2287.0492.24Ours (ResNet50)70.0680.9088.6693.66The best result is highlighted in bold.As shown in Table4, our model outperforms the state of the art in terms of Recall@1 on the Oxford Pets. In our method, under the ResNet50 network model, the Recall@1 of the Stanford Dog dataset reached 98.31.Table 4
Recall@K on the Oxford Pets dataset with baseline methods.
MethodOxford Pets (Recall@K)K =1248PCEL (ResNet50)97.0998.7399.4799.73DGCRL (ResNet50)97.8699.1099.6399.78Ours (ResNet50)98.3199.1699.5599.79The best result is highlighted in bold.For the purpose of better exhibiting our retrieval results, we visualize the reducing method called t-SNE to visualize the embedding vectors produced by our proposed method on CUB200-2011 and Stanford Dog. According to Figures6 and 7, it is known that our proposed method works very well in grouping similar images in spite of the viewpoint and posture that have changed.Figure 6
Visualization of the embedding space obtained using our method on the CUB200-2011 dataset.Figure 7
Visualization of the embedding space obtained using our method on the Stanford Dog dataset.


## 4.5. Ablation Experiment

In this section, we test the effectiveness of the proposed models, the multiproxies adaptive distribution loss, the multiproxies adaptive distribution scheme, and the channel weighting in the weakly supervised aggregation model. The loss we chose as the baseline comparison is the DGCRL proposed by Zheng et al. because we used the same network model, batch size, optimizer, input size, and feature dimension. What is more, SCDA operations were performed in the testing step.As shown in Table5, our multiproxies adaptive distribution loss, multiproxies adaptive distribution scheme, and the channel weighting in the weakly supervised aggregation model all have positive effects on Recall@K. Compared to the DGCRL loss, our loss form has increased by 1.85% on Recall@1. The multiproxies adaptive distribution scheme we proposed can increase by 1.92% with our loss form. And the aggregation method we proposed to perform channel weighting after SCDA improves the effect in many cases. The best effect is achieved when the three are combined.Table 5
Recall@K with different models on our methods.
Method (ResNet50)CUB-200-2011 (Recall@K)K =1248DGCRL (ResNet50)67.979.186.291.8MADL (ResNet50)69.7579.9087.0792.49MADL +V (ResNet50)70.1780.1187.3492.74MADL + MADS (ResNet50)71.6780.8988.0992.99MADL + MADS +V (ResNet50)71.8880.8788.1892.98MADL + MADS (ResNet101)72.8282.3489.0493.29MADL + MADS +V (ResNet101)72.9782.6689.1893.30MADL is the multiproxies adaptive distribution loss. MADS is the multiproxies adaptive distribution scheme.V is the channel weighting in the weakly supervised aggregation model. The best result is highlighted in bold.


## 4.6. Parameter Analysis

In this section, we tested the adjustable parameters in the paper. They are the number of divisionsM of the histogram in the multiproxies adaptive distribution scheme. α, β, and ρ in the multiproxies adaptive distribution loss: α is a scaling parameter, β is a margin parameter, ρ is the fusion parameter.As shown in Table6, we can regard M=1 as a single proxy for each class. Through Table 6, we can find that when M>1, the value of Recall@K is higher than M=1. When M=2, the value of Recall@K is generally the highest. It can be explained that compared to a single proxy, the multiproxies adaptive distribution scheme we proposed is effective. But it is not that the more proxies are distributed, the better the result is. When M=2, the value of Recall@K generally achieves the best results.Table 6
Recall@K with differentM on CUB200-2011.
Method (ResNet50)CUB200-2011 (Recall@K)MADL124816M = 170.1780.1187.3492.7495.77M = 271.8880.8788.1892.9895.93M = 371.0281.1487.8392.8695.91M = 470.5180.1387.4692.8095.86TheM is the number of blocks divided by the histogram in a multiproxies adaptive distribution scheme. The best result is highlighted in bold.We also investigate the effect of the two hyperparametersα and β of our loss on the CUB200-2011 dataset.As shown in Figure8, we tested the parameters α and β in the multiproxies adaptive distribution loss. α is a scaling parameter, and β is a margin parameter. We tested when α is equal to 4, 8, 16, 32, and 64. When β is equal to 0, 0.1, 0.2, and 0.3, the value of Recall@1 on the CUB-200-2011 dataset, the experimental results show that when α=32 and β=0.1, the retrieval achieves the best results.Figure 8
Recall@1 with differentα and β on CUB-200-2011. α is a scaling parameter, and β is a margin parameter in multiproxies adaptive distribution loss.Finally, we testedρ as the fusion parameter.As shown in Table7, ρ is the fusion parameter in the multiproxies adaptive distribution loss. When ρ=0.1, the value of Recall@1 achieves the best results.Table 7
Recall@K with differentρ on CUB200-2011.
Method (ResNet50)CUB200-2011 (Recall@K)MPADL124816ρ = 0.0570.9081.0687.7692.6095.83ρ = 0.171.8880.8788.1892.9895.93ρ = 0.270.7880.3387.6892.7895.95ρ = 0.370.5380.2787.5692.8395.90ρ = 0.670.6680.3587.7692.9695.71ρ = 0.871.5681.1488.0892.9796.07ρ = 0.970.2280.2588.0892.8395.85ρ = 1.071.3081.0287.7192.5696.08ρ is the fusion parameter in multiproxies adaptive distribution loss. The best result is highlighted in bold.Through the parameter test, we have selected the parameters that are more suitable for the algorithm to achieve better results. In the multiproxies adaptive distribution scheme, the larger the number of divisionsM of the histogram is, the more proxies will be distributed to each class. When M=1, it is equivalent to the case of a single proxy for each class. The experimental results found that the idea of distributing multiple proxies to each class can achieve better retrieval results because multiple proxies can maintain the differences within the class. But M is not the bigger, the better, because a certain number of proxies can distinguish the differences within the classification. Too many divisions will only increase the amount of calculation and make the aggregated features too scattered. It is found through testing that the best effect will be achieved when M=2. α is the scaling parameter in the loss function. Experiments have found that the best effect is achieved when the scaling ratio is 32, and β is the margin parameter in the loss function, which determines how much the similarity threshold reaches and stops training. Since the range of cosine similarity is −1 to 1, the value of β is between 0 and 1. We find through testing that the best effect is achieved when β=0.1, that is, the similarity between positive samples is greater than 0.1, and the similarity between negative samples is less than −0.1, achieving the ideal clustering effect. ρ is the fusion parameter, which determines the fusion ratio of the loss between the proxy and the proxy and the loss between the proxy and the sample. Because the object is different, the value range is different, and the one-to-one fusion cannot be directly performed. The test found that ρ=0.1; the best retrieval effect can be achieved.


## 5. Conclusions

In this paper, we solve certain issues in fine-grained image retrieval. We proposed a multiproxies adaptive distribution loss and a weakly supervised aggregation model. The former can either maintain intraclass differences or decentralized cluster centers, and the latter can not only locate the object area but also weigh the importance of feature channels.Under a unified image retrieval framework, we have made two innovations in the loss function in the training phase and feature aggregation in the test phase. Our proposed multiproxies adaptive distribution loss fine-tuned the CNN model in the training phase to obtain a better embedding space, and the proposed weakly supervised aggregation model obtained a more representative image descriptor in the testing phase. Through the ablation experiment, we can find that the two improvements have achieved mutual promotion. First of all, the CNN model obtained by multiproxies adaptive distribution loss training can ensure that the three-dimensional features of each image are calculated. Secondly, in order to facilitate the calculation of similarity, we need to aggregate three-dimensional features into one-dimensional features. At this time, the weakly supervised aggregation model we proposed can retain the original information to the greatest extent and highlight important target regions and channels. They work together in two important steps of retrieval to improve the performance of fine-grained retrieval. We achieve the best retrieval performance on the fine-grained datasets CUB200-2011 and Stanford Dog.



---

*Source: 9945840-2021-09-21.xml*