FCNet: A feature context network based on ensemble framework for image retrieval

IET Computer VisionVolume 16, Issue 4 p. 295-306 ORIGINAL RESEARCHOpen Access FCNet: A feature context network based on ensemble framework for image retrieval Siyang Li, Siyang Li Beijing University of Technology, Beijing, China UISEE Technology (Beijing) Co. Ltd., Beijing, ChinaSearch for more papers by this authorYu Guo, Yu Guo UISEE Technology (Beijing) Co. Ltd., Beijing, ChinaSearch for more papers by this authorHao Ren, Hao Ren Fudan University, Shanghai, ChinaSearch for more papers by this authorZiyi Wang, Ziyi Wang Tsinghua University, Beijing, ChinaSearch for more papers by this authorKeyan Ren, Corresponding Author Keyan Ren keyanren@bjut.edu.cn orcid.org/0000-0002-7332-2882 Beijing University of Technology, Beijing, China Correspondence Keyan Ren, Beijing University of Technology, Beijing 100124, China. Email: keyanren@bjut.edu.cnSearch for more papers by this authorChunsheng Liu, Chunsheng Liu Shandong University, Jinan, ChinaSearch for more papers by this authorHua Lin, Hua Lin UISEE Technology (Beijing) Co. Ltd., Beijing, ChinaSearch for more papers by this authorJianbo Shi, Jianbo Shi UISEE Technology (Beijing) Co. Ltd., Beijing, ChinaSearch for more papers by this author Siyang Li, Siyang Li Beijing University of Technology, Beijing, China UISEE Technology (Beijing) Co. Ltd., Beijing, ChinaSearch for more papers by this authorYu Guo, Yu Guo UISEE Technology (Beijing) Co. Ltd., Beijing, ChinaSearch for more papers by this authorHao Ren, Hao Ren Fudan University, Shanghai, ChinaSearch for more papers by this authorZiyi Wang, Ziyi Wang Tsinghua University, Beijing, ChinaSearch for more papers by this authorKeyan Ren, Corresponding Author Keyan Ren keyanren@bjut.edu.cn orcid.org/0000-0002-7332-2882 Beijing University of Technology, Beijing, China Correspondence Keyan Ren, Beijing University of Technology, Beijing 100124, China. Email: keyanren@bjut.edu.cnSearch for more papers by this authorChunsheng Liu, Chunsheng Liu Shandong University, Jinan, ChinaSearch for more papers by this authorHua Lin, Hua Lin UISEE Technology (Beijing) Co. Ltd., Beijing, ChinaSearch for more papers by this authorJianbo Shi, Jianbo Shi UISEE Technology (Beijing) Co. Ltd., Beijing, ChinaSearch for more papers by this author First published: 24 January 2022 https://doi.org/10.1049/cvi2.12088AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In order to train an end-to-end model for better performance in distinguishing superclasses and make the model more sensitive to inter-class differences, the authors propose a Feature Context Network (FCNet), a novel ensemble framework for image retrieval based on clustering (EFC) and metric learning. In the authors' approach, the EFC framework consists of one common feature extractor and multiple learning branches. This tree-like structure not only promotes the stability of the training process but also keeps the diversity by using multiple branches. Each branch is learnt by an independent random re-labelling operation that distributes original classes into superclasses. In such a way, the framework thus can accelerate the training process. To make the EFC framework focus on the different attributes across the classes, the authors utilize k-means clustering to divide the training set into subsets and train the model sequentially with those subsets. Meanwhile, to reduce the impact of over-fitting problems, the authors design a branch weight structure during training. Further, FCNet fusions different scales of the neighbourhood information and consequently makes the feature map learn richer information with the different receptive fields. Simultaneously, it can reduce the layer of the network and the number of model parameters. Extensive experiments demonstrate that this approach outperforms the tested representative methods on CARS-196, CUB-200-2011 datasets. 1 INTRODUCTION Image retrieval has been extensively studied in the computer vision and multimedia community for decades, which aims to retrieve the most similar samples of the query image. The core task of the image retrieval model is to map the images with similar attributes to the nearby location in a common feature space. It is a kind of a metric learning task to be able to measure the distances between samples correctly. To achieve this goal, a wide range of methods has been proposed [1-9]. Deep metric learning is at the cornerstone of state-of-the-art image retrieval tasks and simultaneously leads to spectacular retrieval results. The core objective of metric learning is to learn an embedding function that maps the images to a high-dimensional feature space [8, 10-12], where the images from the same category stay close and the different categories are pulled far away. The retrieval algorithm usually measures the similarity between feature vectors by L2-normalization distance, cosine similarity, or customized methods [3]. Besides, some other works [9, 13] map the images into hash code directly for the faster retrieval speed. To enhance the representation capacity of the feature vector, Teh et al. [6] find that low-temperature scaling is important for the performance. More recent methods [7, 14] such as relabelling procedures are introduced to boost feature diversity. Xuan et al. [7] present a promising architecture called Deep Randomized Ensembles for Metric Learning (DREML). Note that this method assembles multiple independent models assigned with the different randomized relabelling operations. For one model of the ensemble, an independent relabelling operation groups the original classes into a certain amount of groups. The groups can be regarded as superclasses and the images of original classes in one superclass are relabelled with the corresponding superclass label. Thus, there are fewer classes for one model to classify and it reduces the difficulties in training models. Multiple relabelling processes can be seen as to encode one original class with the superclass code. The superclass code for one original class is much likely to be unique especially when the number of relabelling operations is enough. By training with superclass labels and stacking the outputs of multiple models, the ensemble model decodes the final outputs to get the original class predictions. Its ensemble framework with multiple independent models is more sensitive to the noises [15] and makes training inefficient. Inspired by DREML, we propose a tree-like ensemble framework with an improved relabelling procedure. Our architecture exploits a common feature extractor and tree-like multiple branches to make the training process faster and more stable. Recently, Artsiom and Vadim et al. [14] proposed DCESML, which improved Divide and Conquer by introducing a k-means clustering to get subsets that can be seen as mega-classes. Dataset and model are then divided into subsets and sub-models to match mega-classes. However, the training strategy is not efficient because only the subset and the corresponding sub-models are used at each iteration. Inspired by DCESML, we add this training concept to our ensemble framework to let branches focus more on the key features between superclasses. Different from it, for each subset, we train all the branches at each iteration and this training process is a supplement of our original training process. Additionally, we assign prior-defined weights to mega-classes to demonstrate the emphasis of the current training iteration. Moreover, mega-class weights also contribute to enhancing feature diversity and preventing the over-fitting problem, which is another method to achieve the same goal with label smoothing. Additionally, inspired by the inception network and shape context, we design a new backbone called Feature Context Image Retrieval Network (FCNet). It can explore efforts on achieving good performance in their corresponding fields by learning richer neighbourhood information. All the above details will be further explained in Section 3. The contributions of this work can be summarized as follows: We propose a novel end-to-end Ensemble Framework based on Clustering (EFC). This framework extracts common features to promote the stability of the training process. Meanwhile, the tree-like ensemble structure keeps the diversity with multiple branches To improve the efficiency of image retrieval and prevent over-fitting, we exploit k-means clustering to cluster the feature vectors and get the subsets after the ordinary training process. Then, the ensemble will be trained sequentially with those subsets. This step not only learns finer and more detailed differences via a k-means clustered subset of the dataset but also eliminates the negative effects of features from similar categories. Moreover, we design a branch weight structure in this extra training stage to let the learning branch less easy to fall into a local optimal solution and be more sensitive to an inter-class, demonstrating that the model indeed has better performance in distinguishing superclasses For expanding the width of the network and fusing different scales of the neighbourhood information, an extension to the standard ResNet backbone Feature Context Network (FCNet) is proposed. The FCNet is also made up of multiple modules. Each module contains different branches and each branch contains a different number of 3 × 3 convolutions, which can make the feature map contains richer information through multiple receptive fields. We demonstrate that the layer of the network and the number of model parameters are all reduced when using FCNet Extensive experiments demonstrate that our proposed framework outperforms state-of-the-art methods on challenging CARS-196, CUB-200-2011 datasets The rest of this paper is structured as follows. First, the related work is briefly discussed and compared to the proposed method in Section 2, highlighting the shortcoming of existing approaches and providing further motivation for this work. Then, the proposed method is derived in Section 3 while the detailed experimental evaluations and ablation studies are provided in Section 4, respectively. Finally, conclusions are drawn in Section 5. 2 RELATED WORK In recent years, deep networks and metric learning have achieved impressive results in an image retrieval task. All top-performing methods across different benchmarks apply them in their pipeline. Based on them, the method that we propose in this paper contains two main aspects: Randomized Ensemble Framework and Divide and Conquer for Metric Learning. Here, we discuss each aspect and situate our model between them. 2.1 Randomized ensemble framework The classic ensemble techniques train multiple branches individually and use a combined model to get high performance [11, 16, 17]. Hansen et al. [18] and Krogh et al. [19] utilized the diversity in error distributions across independent models to improve the performance of the ensemble. Conventional ensemble structures average the results of multiple models that are trained with the randomized initialization [20, 21] or different training subsets [22, 23]. Michael et al. [24] proposed a sequential method called BIER to train each branch by re-weighting samples from the previous branch. Following this framework, DREML [7] trained multiple independent models with randomized relabels and then concatenated the outputs of ensembles. Similar to DREML, our method forms an ensemble architecture with randomized relabels. The diversity of ensemble can be achieved with randomized initialization of training set, which is also a way to enhance the performance. The purpose of the randomized relabelling operation is to train each branch with an independent set of superclasses at the training stage and, finally, enable the branches to classify those classes. Recently, a novel method proposed by the work of DREML [7] provided a metric learning method with a randomized relabelling operation. For each branch of the ensemble architecture, the method implemented a randomized distribution that results in grouping every original class label into superclass labels randomly and evenly. Each branch had the same number of superclasses and the quantity gap of the original class between every two superclasses would be less than one. Training with those superclasses, the branches learnt to distinguish those superclass images. This training process makes the branches learn to extract the key features and map them to the feature space. Then, by learning this metric, the ensemble model was able to measure the similarity distance between images through their concatenated output vectors of multiple branches. 2.2 Divide and conquer for metric learning The core task of deep metric learning is to learn an embedding function f which maps images to high dimensional feature space. Classic methods sampled the instances into positive and negative pairs to learn the metric with the contrastive loss [25, 26]. In an ideal situation, images from similar categories are closer than those from different categories in the feature embedding space. Most of the later methods focus on the loss function, such as triplet loss [27-29], ranking loss [30], clustering loss [31, 32], N-pair loss [34], and angular loss [35]. However, most methods learn a single distance metric for all samples from the given data distribution without focussing on the unevenly distributed data. In feature space, the distribution of data in subspaces is often related to different types of characteristics, such as shape, colour, and so on. Images across classes may carry the same characteristics as that. Due to the similarity of these characteristics would stay high, it may cause more hard negative samples and have a negative impact on the learning process. In Ref. [36], the Divide-and-Conquer method combined the solutions for sub-problems into the solution for the original problem. Inspired by it, DCESML [14] divided the data into groups to reduce the complexity, solving the metric learning problem better at the same time. It clustered the data in the embedding space and then split the embedding layer of the network into several slices. Each slice represented an individual branch and could be assigned to one cluster, which was related to a certain subspace of the original embedding space. Our method utilizes the method of Divide and Conquer to divide images into several subsets. The branches with fixed weights are used to learn the features of images in corresponding subsets. 3 METHOD We propose an approach for image retrieval, which consists of an ensemble framework with a common feature extractor and multiple branches. Each branch is learnt by randomly relabelling the data class into several superclasses. Then, we divide the data into multiple mega-classes by clustering their feature vectors. Several branches are assigned to one mega-class and operated in the corresponding subspace before training, and then trained with corresponding weights. In order to get a better feature vector, we utilize a new backbone FCNet. The pipeline is described in Figure 1. It should be noted that each superclass contains several classes and images with these class labels are belong to this superclass. Mega-classes here are different from superclasses. Mega-class is the result of clustering feature vectors. Each mega-class is a subset of clustering results. Images with the same class label will have the same superclass; however, they may be divided into different mega-classes due to the difference in their feature vectors. We will present each step of our approach in detail in this section. FIGURE 1Open in figure viewerPowerPoint The schematic of the Ensemble Framework is based on Clustering. Assume that there are n classes, the operation of label distribution can be regarded as encoding the original class with superclasses, then each original class will get a unique code. For different branches, we have different superclasses. The ensemble model has one common feature extractor and several branches. Each branch is trained to identify these superclasses, which is equivalent to learning to decode the encoded samples to obtain the original classes. In this work, we use mega-class to describe the output subsets generated by clustering. After every training epoch with the original set, we send output vectors into K-means clustering to get subsets and use weight wk to train K extra epochs with those subsets 3.1 Randomized ensemble structure We implement a randomized ensemble technique that makes the model easier to learn and classify. Each original class will be assigned a superclass label in each branch and each branch will be trained independently. For the whole model, this means one original class will have one unique superclass code to indicate its superclasses in all branches. The process of training every branch is actually training the model to classify the unique code of each original class. The ensemble model is trained to classify these superclasses, which is equivalent to learning to decode the encoded samples to obtain the original classes of the samples. In the inference stage, the output of every branch will be normalized (L2). By stacking the outputs of all branches, the ensemble model would be able to predict original classes. As such, even though the testing set is some classes other than the training set, the cosine similarities between similar images' output vectors will naturally stay higher than those between dissimilar images' output vectors. To improve DREML, we replace multiple independent models with an M-heads network structure, which consists of one common feature extractor followed by multiple branches. This ensemble-like architecture maintains the diversity with multiple branches by sharing the common feature extractor. In addition, our approach with an integrated architecture enables our model to train faster and more practical. Our baseline model is based on ResNet18 [37]. Our proposed backbone network is divided into two parts, one is acting as common feature extractors and the other is branches. The common feature extractor shares basic features among all branches. Each branch is constructed with the rest of the convolution layers of the backbone and is trained to classify the same number of superclasses. The number of branches here is equivalent to the number of models in an ensemble structure. It is quite significant for the ensemble architecture to acquire diversities among models. In the randomized labels distribution step, all original classes are distributed into superclasses and all images are relabelled with superclass labels. Each branch owns a unique distribution in the training stage, which means a class along with all of its images could be relabelled with different labels in different branches. By training branches with these distributions, our model generates diversities among branches. 3.2 Train with weighted mega-class There are many similar attributes among categories, such as colour, shape, pose, angle, and so on. For example, cars of different series may also have the same colour. These attributes will make inter-class samples more similar in some dimensional spaces, leading to increase difficulties in the training process. Though our framework has reduced the number of classes through randomized distribution elaborated in Section 3.1, we can further manage branches easier to classify by helping them ignore similar features. So we design an additional mega-class relabelling method to train branches with subsets generated by the k-means clustering process. On one hand, inspired by the method DCESML [14], we explore a further training phase of cluster embedding after routine feature training. When training, we can get their feature vector {v1, v2, …, vn} for data {x1, x2, …, xn} from e branches. In the cluster training stage, we cluster images according to their feature vectors vi concatenated by the output from e branches. Correspondingly, the original image set will be clustered into several groups according to certain features. It should be mentioned that for each branch, we get the feature vector from the output of the last convolution layers other than the output of this branch. The size of this vector is 1 × 1 × C and the C is the output channel of the last convolution layer. Formally, the number of sub-problems is K, these feature vectors were clustered into a certain number of mega-classes by k-means clustering, where Euclidean distance is selected as the distance metric. As a consequence, the training set was split into K subsets {G1, …, Gk} by some specific clustering characteristics. Accordingly, the branches are divided into K groups, which correspond to K mega-classes and the branches in one group have the same weights. On the other hand, the fixed weights of ensemble structures are hard to emphasize the quality of different branches. Inspired by the Momentum gradient descent algorithm [38, 39], we design a branch weight structure to add up the loss value and increase the branch diversity. Momentum will update the gradient descent rate according to the previous gradient descent rate when performing gradient descent. When the gradient descents rapidly, the momentum of the model is very large. Therefore, it should be easier to jump out of the local optimal area. We believe that increasing the loss value during training properly can also play a role in increasing the gradient descent rate during backpropagation, especially when the model is using weight decay and the learning rate is small after many epochs. To increase the diversity between branches and let branches be sensitive to different key features, we consider a branch weight structure that we forcibly increase the loss value of certain branches when training subsets. Theoretically, a branch weight structure will lead to a larger gradient value overall and promote the model to a better performance in distinguishing superclasses. In our ensemble learning scenario, assume branch weights W are generated in advance according to the number of cluster centres K. For different mega-classes (subsets), the order of W changes to make sure {w1, w2, …, wk} for k clusters are different. W = 1 K , 2 K , … , K K (1) Such weight settings are more informative for reaching relatively good retrieval performance. Following this intuition, the output of each branch Lf will be multiplied by weight wk as follows: v k = w k × L f ( x k ) (2) Assume y as an output vector of one branch and i as the index of the class, we can implement cross entropy-loss function with log softmax as follows: C E ( y , i ) = − log exp ( y i ) ∑ j exp ( x j ) (3) And the gradient can be calculated with label t. ∂ l o s s ( y , t ) ∂ y i = y i − t i (4) To address this, we train the model with the one-hot labels that come from the original space as Equation (5). L o s s = C E ( w k × L f ( x k ) , S d e ) (5)where Sde is the ground truth. Adding weight to a specific branch can be thought of as an operation to increase the distance between the output vector and the ground truth. During training, k subspaces are generated for k subsets. Computed by the equations of 4 and 5, the gradient grows with weight w. In each subspace, Equation (5) makes wk × Lf(xk) → 1 or wk × Lf(xk) → 0. Therefore, the loss value becomes greater due to the decrease in the vector's norm, and then the gradient increases. In the case of the gradient increases, the model can avoid getting stuck in local minima. It can be viewed as Equation (6); meanwhile, the distance between different superclasses will be increased. Figure 2 is better to understand. L f ( x k ) → S d e / w k , 0 < w k < 1 . (6) FIGURE 2Open in figure viewerPowerPoint The schematic of shifting space. Assume the dimension of space is 3. Left: Without branch weight W. The vector needs to tend to the one-hot vector, which represents its ground truth location in the space. The Lf learn to map similar images to this point in the feature space. Right: With branch weight W. It is equivalent to rise the single high bit of the one-hot vector to 1/W. Then, the Lf learn to map the similar to corresponding points. It is a kind of space shifting that increases the loss value shown as a 2D coordinate graph in the picture. Blue and green represent different subspaces with different w Following this statement, the branches with the same weight are all trained with one subset and to classify the images with common features. The feature vector vk will better represent the image itself and contain more unique information about the image in the process of feature learning. Therefore, the cluster needs to be updated continually. In our algorithm, if the epoch satisfies the equation: T mod t = 0, we update the mega-class with k-means clustering. After solving sub-problems divided from the big problem, we try to solve the big problem directly. To the end, after conquering subsets, we train the model with the whole dataset and all branches without weight. More concretely, if the model is well trained, the prediction of the last fully connected layer is approximate 0 or larger than 1. After normalization, the gap between feature vectors in different categories will be widened. The whole algorithm is shown in Algorithm 1. Algorithm 1. Training and testing our approach Input: X = {x1, …, xn}: data, L = {L1, …, Le}: e branches with deleting their fully connected layer, Lf = {Lf1, …, Lfe}: e branches with fully connected layer, Sde: d superclasses for the eth branch, W = {w1, …, wk}: branch weight for k clusters. Output: {o1, o2, …, on}: the vector of data. Epoch = 0 for h iterations do divide X into Sdh end for if training then if Epoch mod t = = 0 then {v1, v2, …, vn} = Concat(L({x1, x2, …, xn})), {G1, …, Gk} = Kmeans({v1, v2, …, vn}), for k clusters do vec = wk × Lf(Gk), Loss = CE(vec, Sde), Update weights. end for else vec = Lf(X), Loss = CSE(vec, Sde), Update weights. end if else {o1, o2, …, on} = Concat(Lf({x1, x2, …, xn})). end if 3.3 New backbone The challenge in image retrieval is to recognize objects in detail with different scales. It is inherently a similar task with object detection, which is how we can distinguish both overall information and the details between images. In object detection, FPN [40] is mostly used to solve the problem by integrating information, which across the feature maps at different scales. FPN usually works well when the model does prediction on more than one feature map. However, we aim to find an architecture to deal with objects of different sizes on one feature map without multiple-scale fusion. For this purpose, we propose a new backbone that uses multiple branches with different numbers of 3 × 3 convolutions. This has the effect of making the network wider and combining neighbourhood information at different scales. The structure is shown in Figure 3. It can be divided into several parts. FIGURE 3Open in figure viewerPowerPoint The schematic of our model. Right: the structure of our backbone—FCNet. Except for the first Conv layer and Pooling layer, it still has three stages. As the architecture goes deeper, there are more branches in each stage. At the same time, more convolution layers are used in the added branch. The results of each branch are concatenated and sent into the convolution with kernel = 1 × 1. Left: the structure of our model, which uses the FCNet as the backbone. It divides the backbone into two stages. The orange layers are common feature extractors. The green or blue layers compose an individual branch that is specifically trained with each superclass re-labelling The first part consists of a 7 × 7 convolution and a max-pooling layer, which is used to down-sample and roughly extract features. The structure still contains three stages with multiple branches. Different branches with different numbers of 3 × 3 convolutions can make the feature map get more information with the different receptive fields. Then, the outputs of these branches are concatenated so that the feature map can contain rich neighbourhood information. After that, 1 × 1 convolution is used to decrease channels and improve network expression ability. As known to all, convolutions in low layers extract information like colour and shape, and convolutions in high layers extract semantic information. Therefore, just a few branches are needed when extracting shallow features, and the number of branches increases with the deepening of the network. As such, the processes enable the network to learn semantic features and fuse more semantic information of the neighbourhood. The channel number of the final feature map is 256, which is smaller than the results of most methods. Take the second stage as an example, the whole process can be represented by Equation (7). F here represents the 1 × 1 convolution, and f2, f3 represent the second and third branches. Similar to FPN and HRNet [41], our backbone can combine information from different scales. However, different from them, our features integrate neighbouring point information across the scal