Reliability verification‐based convolutional neural networks for object tracking

IET Image ProcessingVolume 13, Issue 1 p. 175-185 Research ArticleFree Access Reliability verification-based convolutional neural networks for object tracking Xiaopeng Hu, Xiaopeng Hu School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road Ganjingzi District, Dalian City, People's Republic of ChinaSearch for more papers by this authorJingting Li, Jingting Li School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road Ganjingzi District, Dalian City, People's Republic of ChinaSearch for more papers by this authorYan Yang, Yan Yang School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road Ganjingzi District, Dalian City, People's Republic of China School of Computer and Information Technology, Liaoning Normal University, No. 850 Huanghe Road Shahekou District, Dalian City, People's Republic of ChinaSearch for more papers by this authorFan Wang, Corresponding Author Fan Wang wangfan@dlut.edu.cn School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road Ganjingzi District, Dalian City, People's Republic of ChinaSearch for more papers by this author Xiaopeng Hu, Xiaopeng Hu School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road Ganjingzi District, Dalian City, People's Republic of ChinaSearch for more papers by this authorJingting Li, Jingting Li School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road Ganjingzi District, Dalian City, People's Republic of ChinaSearch for more papers by this authorYan Yang, Yan Yang School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road Ganjingzi District, Dalian City, People's Republic of China School of Computer and Information Technology, Liaoning Normal University, No. 850 Huanghe Road Shahekou District, Dalian City, People's Republic of ChinaSearch for more papers by this authorFan Wang, Corresponding Author Fan Wang wangfan@dlut.edu.cn School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road Ganjingzi District, Dalian City, People's Republic of ChinaSearch for more papers by this author First published: 01 January 2019 https://doi.org/10.1049/iet-ipr.2018.5785Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The authors propose a tracking algorithm based on the reliability analysis of the convolutional neural network to avoid drift. In general, most tracking algorithms implemented with the deep network consist of a single network; they obtain the tracking results according to the confidence and perform updates with the samples, which are collected based on the previous target state. However, this kind of algorithm relies heavily on the accuracy of tracking results, and slight deviations can lead to improperly labelled training samples and degrade the network. Therefore, they design a verification network to guarantee the reliability of the tracking network by correcting the results and it can be connected to a tracking network by sharing convolutional layers. The reliability verification network estimates the accuracy of the results of the tracking network and discards ambiguous results to avoid accumulating errors. Specifically, the verification network can distinguish the target from the confused candidates more precisely because of the optimised training data. The training samples of the verification network consist of characteristics and labels, and they are optimised by feature selection and label enhancement, respectively. The experimental results illustrate the outstanding performance compared with several state-of-the-art methods on the challenging video sequences. 1 Introduction Visual tracking is one of the fundamental problems in image processing and computer vision. With the growing demand for artificial intelligence, visual tracking has been widely used in intelligent transportation [1], pavement detection [2], video monitoring [3] and other aspects. However, visual tracking still faces the challenges in complex scenarios such as occlusion (OCC), illumination variation (IV), background clutters (BCs) and deformation (DEF). Most tracking algorithms can be divided into two categories: generative and discriminative approaches. The generative methods describe the appearance characteristics of the target and minimise the reconstructed errors by searching the candidate target. The representative algorithms include sparse coding [4, 5], density estimation [6], principal component analysis [7] and so on. The generative methods simply focus on the target and ignore the background information. It is easy to lose the tracking target if the appearance changes drastically. The discriminative methods distinguish the target from the background by training the classifiers. This kind of methods is also called track-by-detection. The representative algorithms include multiple instance learning [8], boosting [9], structured support vector machine [10] and so on. The discriminative approaches are more robust than generative approaches. Nevertheless, the discriminative capability is greatly restricted because they both depend on low-level hand-crafted features. To enhance the ability to characteristic representation, more and more researchers use the deep neural network (DNN) to achieve visual tracking such as convolutional neural network (CNN). The capability of the characteristic representation is obtained by using the image datasets to pre-train the CNN such as ImageNet. For the sake of improving the precision of tracking, researchers always focus on enhancing the characteristic representation ability of the DNNs [11-13] and optimising the update mechanism [11, 14, 15]. However, the research on the reliability of the model is not sufficient. The reliability of the model is closely related to the drift problem. The tracking algorithms based on the unreliable tracking network are prone to cause drift. It is crucial to update the tracking network with the correct samples; otherwise, the tracking network can be polluted by the ambiguous samples. Most tracking algorithms based on the neural network consist of a single network, and they lack the verification mechanism to check the validity of the tracking results. The tracking results which are generated from a single network can be mistaken. In the tracking process, the confidence of the tracking result may be continuously low. However, most existing algorithms do not have the corresponding strategies to modify the current results, and the ambiguous samples degrade the tracking network. When the confidence of the tracking result is high, the tracking network selects the candidate target with the highest confidence as the tracking result. The premise of trusting the current result is that the tracking network is reliable. It is obvious that the single tracking network lacks a verification mechanism to guarantee the accuracy of the results. In this paper, we propose a novel tracking algorithm based on the verification network. The role of the verification network is to ensure the reliability of the tracking network and avoid drift problem. The verification network can be connected to a tracking network by sharing convolutional layers. The input of the verification network is the feature maps, which are optimised by feature selection. The training labels of the verification network are optimised by comprehensively considering the content similarity and the spatial relationship to distinguish the target from the background more accurately. At the same time, the update mechanism of the verification network is divided into short term and long term for adaptiveness and robustness, respectively. The verification network generates the results when the tracking network is unreliable. The main contributions of our work are summarised below: We propose a novel reliability verification network to supervise the tracking network. The role of the verification network is to ensure the reliability of the tracking results and the network. The verification network corrects the tracking results when the reliability of the tracking network is low. The dual decision mechanism is adopted to guarantee the accuracy of the tracking results and enhance the discriminative ability of the network. The reliability verification network is trained by the optimised samples to represent the target more precisely. In terms of the characteristic representation, the feature maps are optimised by the feature selection based on the heat maps regression. In terms of the labels, the spatial relationship and the content similarity are considered comprehensively to estimate the confidence of the samples. Our experimental results demonstrate the outstanding performance of the methods, which are based on the reliability verification network. The improved methods are compared with the state-of-the-art trackers in the public benchmark: object tracking benchmark (OTB) [16]. The rest of this paper is organised as follows. We first review related work in Section 2, and then discuss the detailed proposed method in Section 3. Section 4 illustrates the experimental results in public tracking benchmark. 2 Related work According to the different modes for tracking, the algorithms can be divided into generative approaches and discriminative approaches. The generative approaches describe the appearance characteristics of the target and minimise the reconstructed errors by searching the candidate [4, 5, 17]. For example, Sevilla-Lara and Learned-Miller [17] proposed the distribution fields for tracking, which uses the L1 distance and the gradient descent to deal with the smoothed and blurred images. Meanwhile, the sparse coding is widely used to fuse the target template and reconstruct the target [4, 5]. The discriminative approaches utilise the foreground and background information to train the classifiers for visual tracking [18-20]. The representative algorithms include boosting [9], multiple instance learning [8], struck [10], tracking-learning-detection (TLD) [21] and so on. These algorithms are accurate and robust, but they all use the hand-crafted features. DNNs have widely been applied in object detection and recognition [22-27]. In recent years, there are more and more tracking algorithms based on DNNs. In the beginning, Wang and Yeung [28] apply a deep model to visual tracking and acquire the characteristic of the target from the pre-trained stacked denoising autoencoder. Afterwards, they proposed Structured Output Deep Learning Tracker (SO-DLT) algorithm to obtain the characteristic by using CNNs, and the network achieves tracking by using long-term CNN and short-term CNN. It is a successful application for CNNs in visual tracking. In addition, there are other types of networks for visual trackings such as siamese network [29], recurrent neural networks [30] and so on. To improve the accuracy of the tracking algorithms based on the CNNs, researchers generally study from the aspects of the characteristic representation, the update mechanism and the model reliability. First, in terms of representing visual data, the tracking network can obtain the capability of the characteristic representation by using plenty of training data. In [11-13], the CNN is pre-trained by using the ImageNet and other large-scale image datasets to obtain the efficient characteristic such as the famous Visual Geometry Group (VGG)-Net [31]. Wang et al. [11] obtain the feature maps of the higher layers and the lower layers by using the VGG-Net and then construct the complementary heat maps to achieve object tracking. Second, the network can predict the target accurately by designing the elaborate update mechanism. The proper update mechanism can consider integrally the initial state and the target DEF. Held et al. [32] proposed the GOTURN tracker, which performs the offline training without the online update. Its speed is fast, but it cannot adapt to the target DEF. It is easy to lose the target when the target state changes drastically. Li et al. [33] update the network in a lazy way. The network is updated when the target appearance changes a lot. Nam and Han [14] consider comprehensively the characteristic representation and the update mechanism. The tracking network is divided into the shared layers and the domain-specific layers. The tracking network is pre-trained by using the different domain-specific layers to avoid destabilising the network. Moreover, the update mechanism is also divided into the long term and short term. Moreover, in terms of the model reliability, Nam et al. [34] proposed a tree structure, which includes several CNN models to avoid the unreliable samples degrading the single network. The purpose of our method is to guarantee the reliability of the network. We design the reliability verification network to assure the accuracy of the tracking results. The verification network can supervise the tracking network to avoid drift. Plenty of tracking algorithms which are based on the deep network rely on a single network, and both the offline training and the online update are aimed at the single network. In this way, the tracking result is obtained according to the confidence of the single tracking network, and the tracking network is updated by the samples according to the previous result. In the continuous process, it is likely to degrade the network performance when the tracking results are unreliable. The verification network learns more crucial foreground and background information to distinguish the target more accurately. The current tracking result is corrected by the verification network when the tracking network is unreliable. The reliability of the entire network is assured by improving the accuracy of the tracking result in every frame. 3 Proposed method In this section, we describe the reliability verification-based CNN for visual tracking. An overview of the proposed method is visualised in Fig. 1. The whole network is divided into original tracking network and reliability verification network, and the latter makes use of feature selection module to enhance the capability of the characteristic representation. The verification network evaluates the outputs of original tracking network and performs update based on training samples, which is indicated as the blue path. In the following, we first describe the overall procedure of our method. Next, we elaborate the reliability verification network. Afterwards, we describe the detail of optimising the training data to make the verification network learn more crucial foreground and background information. Finally, we discuss the online update strategy of the reliability verification network. Fig. 1Open in figure viewerPowerPoint Pipeline of our method and the architecture of the reliability verification network. The positive and negative samples are expressed in red and white boxes, respectively 3.1 Overview of our approach Our method constructs one reliability verification network, which is essentially a CNN. The original intention of designing the verification network is to make it be connected to a tracking network by sharing the convolutional layers. In terms of the design principle, the verification network can be connected to a tracking network which has the convolutional layers, but has nothing to do with the implementation. In a way, the reliability verification network can be understood as plug and play. The verification network can judge the accuracy of the tracking results more comprehensively because of the optimised training data. As for the tracking methods based on the CNN, the quality of data directly determines the network performance. On the one hand, we use feature selection to eliminate noise and to reserve a portion of feature maps. On the other hand, most algorithms utilise the overlap ratios with the ground-truth bounding boxes as the labels to train the network, but our method optimises the labels by considering the content similarity. The content similarity is calculated by the colour histogram based on the space block. The label of the training samples is defined as the weighted sum of the overlap ratio and content similarity. The procedure of our method is shown in Algorithm 1 (see Fig. 2). The main idea is to use the verification network to estimate the accuracy of the tracking results, which are obtained from the tracking network. The current tracking result is corrected when the verification network confidence is low. To be more exact, the way to select the tracking result is determined according to the specific confidences. Fig. 2Open in figure viewerPowerPoint Algorithm 1: Procedure of our method 3.2 Reliability verification network We design the reliability verification network to estimate the accuracy of the results, which are obtained from the tracking network and correct unreliable tracking results. The verification network can learn more information than the tracking network because of the optimised training samples. The optimisation strategies are feature selection and label enhancement. Specifically, the input of the verification network is the feature maps which are obtained from the feature selection model, and the corresponding labels are the weighted sum of the overlap ratio and the content similarity between the samples and the ground-truth bounding box. The optimisation strategies above are the primary reason that the verification network can correct the ambiguous results obtained from the tracking network. The essence of the verification network is a CNN, which consists of three fully connected layers and a sigmoid layer. The dimension and number of the feature maps in the verification network change along with the tracking network because the structures of tracking networks are different. The weighted parameters of the three fully connected layers can be expressed as , and , and the bias parameters can be expressed as , and , respectively. There are rectified linear unit (ReLU) non-linear layer and dropout layer between the fully connected layers. For example, an input image patch x, assuming it obtains K feature maps by extracting the convolutional features, and the size of each feature map is . The input of the first fully connected layer can be expressed as . Assuming the input vector of the ith fully connected layer is , the feedforward process of the verification network can be represented as follows: (1) (2)where and represent the weighted parameter and bias parameter of the ith fully connected layer, represents the ReLU excitation function and represents the dropout operation. The last connected layer of the verification network needs to connect to the sigmoid layer to obtain the probability ranged 0–1. The physical significance of the probability is the similarity between the tracking target and candidate samples. The loss function of the verification network at frame t in the training stage can be explained as below: (3)where N is the number of training samples at frame t, is the integrated similarity label between the jth training sample and the tracking target at frame t which can be computed by (10); as a result, the label at frame t is the value of in (10), is the predicted output value obtained from the verification network at frame t, w is the weight coefficient of the verification network and is the accommodation coefficient . The L2-norm is added to the loss function to avoid overfitting. The verification network is designed to avoid ambiguous samples degrading the tracking network. The current tracking result is correct if the confidence of the verification network exceeds the threshold; otherwise, the tracking result is wrong. Furthermore, if the confidences of these two networks both fall below the threshold, the target state is likely to deform and the discriminant capability of tracking network is weakened. However, the tracking network may be still reliable. In this case, the verification network generates the candidate bounding boxes in the neighbour of the previous target state. The most similar K candidates are reserved, and the tracking network selects the optimal result among these candidates as the final tracking result. If the confidence of the verification network falls below the threshold while the confidence of tracking network exceeds the corresponding threshold, the tracking network is considered unreliable. Then, the predicted result of the verification network is regarded as the final tracking result. 3.3 Data optimisation To improve the characteristic representation capability of the verification network, the training data is optimised in two aspects: feature selection and label enhancement. The tracking approaches based on the deep network generally regard the feature maps as the feature vectors without any filtration. The overlap ratios between the ground-truth bounding box and the samples are regarded as the labels to perform the supervised learning. However, the characteristic representation of the general approaches is redundant and noisy. It is not enough to utilise the location information to represent the labels. The calculation of the similarity labels includes the spatial information and the content information. The detailed procedure is discussed in this section. 3.3.1 Feature selection model The feature maps of the verification network are purified by the feature selection instead of extracting from the shared convolutional layers directly. The role of the feature selection is to reserve partially the feature maps, which have significant impact on the objective function. The reserved feature maps are more relevant to the tracking target, and they are the inputs of the verification network. The advantages of the feature selection are reducing computation and enhancing the capability of the characteristic representation. Wang et al. [11] construct the fully convolutional networks for the characteristics of the different layers. The characteristic is optimised and denoised by using the feature selection. The feature selection model consists of a convolutional layer and a dropout layer. The model regards the feature maps obtained from the last shared layer as the input to acquire the target heat map. The heat map is a two-dimensional Gaussian centred at the ground-truth target location with variance proportional to the target size. The feature selection model is based on the target heat map. The loss function is to calculate the square error between the target heat map M and the predicted foreground heat map . The loss function can be explained as below: (4)The top K feature maps which have the greatest impact on the loss function are reserved by training the feature selection model. Assuming that the input feature maps are recorded as F. Denote as the ith element of F. The change of the loss function caused by the perturbation of the feature maps can be represented by a two-order Taylor expansion as follows: (5)where and are the first- and second-order derivatives of the objective function with respect to the input feature maps, respectively. However, it is too time-consuming for computing the second-order derivatives. The Hessian matrix is approximated with a diagonal matrix, and the last term of formula (5) is neglected. The first-order derivatives and the second-order derivatives can be concisely computed via back-propagation. At the same time, the significance of the element is defined as the change of the loss function after setting to 0. It can be expressed as follows: (6)Then, the impact on the loss function caused by the change of can be computed as below: (7)The significance of the kth feature map can be defined as the summation of the significance of all the elements , where is the significance of the element indexed by the location on the kth feature map. The top K feature maps which have the greatest impact on the loss function are selected according to these principles. To reduce the time consumption, the feature selection model is only conducted at the initial frame. The indices of the reserved feature maps are fixed during the execution. The feature selection model can discard the feature maps with noise to improve the accuracy of the characteristic representation. 3.3.2 Similarity label enhancement To estimate the similarity between the candidates and target more relevantly, the training labels are improved from two aspects: the spatial relationship and the content similarity. In general, the overlap ratios between the ground truth and samples are regarded as the labels to perform the supervised learning. Assuming that the ground-truth bounding box is A, a sample bounding box is B and S denotes the space area. The inersection over union (IOU) overlap ratio of the bounding boxes can be explained as follows: (8)where the value range is from 0 to 1. The IOU overlap ratio computes the similarity according to the spatial relationship between the samples and tracking results. The larger the overlap ratios, the higher the target similarity. The overlap ratio is certainly accurate at the initial frame because the tracking target is in the centre of the ground-truth bounding box and the contour occupies the most area of the bounding box. However, it is not enough to consider the spatial information. The predicted bounding box might exist in the deviation of the position in the tracking process. The contour of the predicted object deviates the centre of the bounding box and the contour is incomplete. In this case, the similarity labels based on the IOU overlap ratio are imprecise. It is likely to make the ambiguous samples degrade the tracking network. Assuming that the bounding box of the current tracking result is known, the predicted object locates on the bottom left of the bounding box and the contour is incomplete. There is a sample bounding box with a high overlap ratio, but it overlaps with the bounding box of the current tracking result on the top right direction. It is obvious that the image content of these kinds of samples is mostly the background. The content similarity is certainly less than the IOU overlap ratio. The similarity between the candidate and the tracking target can be more accurate if the spatial relationship and the content similarity are considered comprehensively. The content similarity is calculated via the distance of the colour histogram based on the regional partition. The image content in the bounding box is used to calculate the content similarity. We first obtain the colour histogram of the image content, and the red, green and blue channels are mapped to 16 partitions. The colour partition is introduced to tolerate colour variation and reduce the computational cost. These parameters can be 4, 8, 16 and 64. The larger the value is, the range of a pixel is smaller and more precise. However, it is susceptible to variation disturbances. In this paper, the 16 partitioning is determined empirically according to experimental results as indicated in Section 4. The parameter setting in the experiment is optimal. The colour histograms of the ground-truth bounding box and the candidates are regarded as the feature vectors to compute the distance, where the dimension of the feature vectors can be expressed as . The cosine distance metric between the candidate and the target is calculated to evaluate the content similarity. Assuming that the colour histogram vector of the candidate is , the colour histogram vector of the tracking target is , and then the cosine distance metric can be explained as follows: (9)where and . The more similar the two vectors are, the closer the cosine is to 1. The cosine distance metric is insensitive to the numerical value, and it focuses on the orientation of the vectors. Moreover, we adopt the proportional regional partition to make up for the lack of spatial information in histogram method. The bounding box image patches are divided into partitions in the proportion of the length and width. The number of partitions is determined empirically according to experimental results indicated in Section 4, and the slight variation of the partitions has no significant effect on the experimental results. The parameter setting in the experiment is optimal. The regional partition is beneficial to avoid the misjudgement to the unrelated images due to the high similarity of the whole colour distribution. The verification network learns the similarity between the samples and target, which is the weighted sum of the overlap ratio and content similarity. According to (8) and (9), the integrated similarity between the current samples and target can be defined as follows: (10)where represents the candidate samples at frame t, represents the tracking target at frame t, represents the tracking result at frame t − 1, is the histogram vector of , is the histogram vector of the ground-truth target in the first frame and the . The value of it makes no sense to pick minus; thus, . 3.4 Online update strategy To solve the drift problem caused by updating the tracking ne