Journal of Robotics, Networking and Artificial Life, Vol. 3, No. 3 (December 2016), 144-147

Using Multi-Target Tracking and Identification TLD Algorithm for Intelligent Mobile Robot
Jr-Hung Guo

Department of Electrical Engineering, National Chin-Yi University Of Technology, No.57, Sec. 2, Zhongshan Rd.,
Taiping Dist., Taichung 41170, Taiwan
Kuo-Hsien Hsia*

Department of Electrical Engineering, Far East University, No.49, Chung Hua Rd., Hsin-Shih. 74448 Tainan, Taiwan
Kuo-Lan Su

Department of Electrical Engineering, National Yunlin University of Science & Technology, 123 University Road,Section 3,Douliou, Yunlin 64002, TaiwanE-mail: g9710801@yuntech.edu.tw, khhsia@cc.feu.edu.tw* ,sukl@yuntech.edu.tw

Abstract
Image algorithms used herein are Tracking-Learning-Detection (TLD) and Speed UP Robust Features (SURF).
TLD is used for target tracking and SURF is used for identifying targets. We use zoning identification, with the useof statistical probability to strengthen the efficiency of TLD and SURF. With such a method, the efficiency ofimage identification and target tracking can be enhanced so that the robot can simultaneously track and identify
multiple targets.
Keywords: multi-target tracking, TLD, SURF, image identification, mobile robot.

1. Introduction
Image recognition and tracking technology areimportant issues on robotics research. Fabian et al. [1]used Microsoft Kinect and Simulink for instant object
tracking. Marković et al. [2] used omnidirectional
camera on a mobile robot to track and follow objects.
Huang et al. [3] used the SURF algorithm to make therobot have the ability to follow the object. As can beseen from previous studies, image recognition andtracking technology usually require dedicated imagingequipment or complex algorithms. But this will cause anincrease in the cost of mobile robot, or reduce theefficiency of the mobile robot. Therefore, how toenhance the efficiency of the mobile robot on image

identification and tracking is the main research purposeof this paper.
The image tracking algorithm that we use is TLD [4]which was developed by Zedenek Kalal, University ofSurrey, Czech. The TLD algorithm used for tracking
single-target for a long time. In this paper we extend theTLD algorithm to multi-target tracking based on theoriginal single-target tracking. But this algorithm cannot
identify the tracked target. The TLD framework
improves the tracking performance by combining adetector and an optical-flow tracker. Since thisalgorithm is based on optical flow, the accuracy of TLD
cannot be high. So we use Speed UP Robust Features(SURF) to assist the targets identification and as anauxiliary for TLD algorithm to track the targets.

Published by Atlantis Press
Copyright: the authors
144

Jr Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su

SURF is based on the scale-invariant feature transform
Scale-invariant feature transform (SIFT) to identify theobject. Although SURF operation is relatively light tothe SIFT, it will still affect the effectiveness of themobile robot. Therefore, in this paper, the image isdivided into some separate regions. Each region isdetected in turn. Supplemented with the statistics andprobability, it can assist TLD on tracking. The target
tracking and identification process with SURF and TLD
is shown in Fig. 1. The relevant algorithms will be
explained in the next section.
2. Algorithms
2.1. TLD
TLD technology is divided into three parts. They are thetracker, the learning process and the detector. Tracker
and detector operate in parallel in TLD. Both thetracking and detecting results will feed into the learning
process. The new model after learning is feedback to thetracker and detector resulting in a real-time updating.
The overall process ensures that even if the target
appearance changes, the target can still be tracked. The
architecture of TLD tracking is shown in Fig. 2.
TLD's tracker estimates the motion of the target using
the Lucas-Kanada [6] [7] optical flow, which is a frame
to frame tracking method. However, the precision of theoptical flow tracking is not high, thus it often results in a
tracking failure. The detector records the history oftarget locations and surface information, called learning,
and will re-detect while tracking failure.
P-N learning [9] is a very important part of tracking inTLD. Here P is Positive Constraint, also known as Pexpert or growing event, and N is Negative Constraint,
also known as N-expert or pruning event. In practice, Pexperts as well as N-experts will have a certain bias, butthe study of Zdenek Kalal found that, although there isan error, under certain conditions, the error is
acceptable, and performance testing module will
therefore be improved. The P-N learning consists offour parts:(i) A classifier for learning;(ii) A training sample set;(iii) Supervised learning;(iv) P-N experts: to generate positive and negative
samples in the learning process;Fig. 1. Target tracking and identification process with SURF
and TLD.

Fig. 2. TLD tracking architecture [4].

The relationship between these parts is shown in Fig. 3.
The “random forest” classifier, which can instantly
update and forecast, is used in the detector [4, 6]. The
information for the feature detector is generated byrandom forests 2bitBP (2 bit binary pattern). The

Published by Atlantis Press
Copyright: the authors
145

Using Multi-Target Tracking andThe sign of the determinant value can be used toclassify the point and to determine the point is anextreme point or not. In the SURF algorithm, thefunction value f(x, y) is replaced by the image pixel I(x,y), and a second-order standard Gaussian function ischosen as a filter. Then elements of H matrix can beobtained by convolution and second-order partial
derivatives as following:Fig. 3. TLD learning architecture [8].

gradient orientation information of a specific area will
be converted into encoded output. 2bitBP is similar toharr-like feature which includes the feature type and thecorresponding characteristic values. Therefore, whenusing nFeat type to express the object, each Fern is a
Quad tree because of the 2bitBP characteristics.
TLD tracking module is based on the Median Flow
tracker method. This method is performed on a frame.
Select some pixels as the feature points in the previous
frame and then find the corresponding positions in thecurrent frame by the feature values. Then sort thedisplacements of these features pixels and find the
median of these displacements. Finally let the pixels
less than 50% of the median as the next feature pixels.
Since this approach assumes that the tracked object is in
the frame, it will fail once the object is out of the frameor obscured. The SURF will be used for confirmation atthat time. Once it is confirmed, the new tracking target
will be selected by the TLD.
2.2. SURF
SURF is a technology for image recognition andtracking based on SIFT. The main difference is that
SIFT uses DoG (Difference-of-Gaussians) image, andSURF uses the determinant of a Hessian matrix toapproximate the image. The Hessian matrix of a pixel in
an image can be defined as [5]: ∂2 f
∂x 2
H ( f ( x, y )) =  2∂ f ∂x∂y∂2 f 
∂x∂y 2∂ f ∂y 2 (1)The determinant of H is:det( H ) =∂2 f ∂2 f∂x2∂y2−(∂2 f 2)∂x∂y(2)

 Lxx ( x, y, σ ) Lxy ( x, y, σ ) H ( x, y , σ ) =  Lxy ( x, y, σ ) L yy ( x, y, σ )(3)

L ( x, y , σ ) = G ( x, y , σ ) ∗ I ( x, y )(4)whereG ( x, y , σ ) =12πσ222e − ( x + y ) / 2σ 2(5)and σ is the scale. The determinant of the H matrix ofeach pixel of the image can be obtained. Hence eachpoint can be classified. For the ease of application,
Herbert Bay proposed to replace the L by anapproximation [9]. The discriminant of H matrix can beexpressed as:G (t ) =∂ 2 g (t )∂x 2(6)

Finally, what we want is a transformed image of theoriginal image, because we have to look for the featurepoints on the transformed image and then reflect it onto
the original image. And this is constituted by theapproximate determinant of the Hessian matrix of eachpixel with the approximate determinant defined by:det( H approx ) = Dxx D yy − (0.9 Dxy ) 2(7)Since SURF process the pixels at the image for Hessian
matrix processing simultaneously on both X and Y
directions, its efficiency is better than SIFT. But it stillrequires a large amount of computation. So we take thesolution as follows:(i) Identify region by region to reduce the amount ofpixel processing. Because the entire image isdivided into nine regions for TLD tracking, SURF
will therefore use the same area for identificationand tracking.
(ii) Determine the identification sequence according tothe orientation. Since the orientation of the robot

Published by Atlantis Press
Copyright: the authors
146

Jr Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su(a) turn left(b) forward(c) turn rightFig. 4. Robot orientation relationship with SURF scanningorder.

can track and identify the targets. The entire image isdivided into nine regions, in order to reduce the loadingon image processing, and to enhance the efficiency oftracking and identification. You used a mid-level
webcam in this paper resulting a lower image quality
and higher failure rate in tracking and identification.
High-level camera and CUDA (Compute Unified
Device Architecture) can increase the quality of image
and efficiency of image processing. Thus this algorithm
can be used in a more complex environment. We
believe that the algorithm can be widely used in robot
systems in the near future.
Acknowledgements
This work was partially supported by National Science
Council of Taiwan. (NSC 104-2221-E-224 -015 -).
References

Fig. 5. Tracking and identification result by TLD and SURF.

can be known, we design the scanning sequence
following the orientation of the robot, as shown in
Fig. 4. The central region is of higher priority, thenthe upper and lower regions. With this strategy, thetarget tracking for the mobile robot can be of higherefficiency.
3. Experimental Results
A Microsoft's LifeCam Cinema was used in ourexperiments. Because of its large aperture and highresolution, images with better quality can be obtained.
For the image tracking and identification system, theentire image was cut into nine regions, and TLD wasapplied to have the tracked object, and then SURF wasused for identification. The CPU of the computer for theexperiment is INTEL I5-3470 with 8G memory, and thesystem was developed by EMGUCV 2.4 and Microsoft
Visual Studio VB2010. The experimental results areshown in Fig. 5. The red box marks the tracking targetof TLD, and the blue box marks the SURF tracking andidentification results with the identification result ofSURF marked at the upper right corner.
4. Conclusion
The TLD and SURF algorithms are successfully
combined in this paper so that the robot's vision system

1. J. Fabian, T. Young, J.C. Peyton Jones, and G.M.
Clayton, Integrating the Microsoft Kinect with Simulink:
Real-time object tracking example, IEEE/ASME
Transactions on Mechatronics, 19(1) (2014) 249-257.
2. I. Marković, F. Chaumette and I. Petrovć, Moving object
detection,
trackingandfollowing
using
anomnidirectional camera on a mobile robot, 2014 IEEE
Int. Conf. Robotics and Automation (ICRA) (Hong Kong,China, 2014), pp. 5630-5635.
3. P.T. Huang, C.Y. Li, C.C. Hsu, and C.M. Hong, Object
following based on SURF for mobile robots, 2012 IEEE
1st Global Conf. Consumer Electronics (GCCE) (Tokyo,Japan, 2012), pp. 382-386.
4. Z. Kalal, K. Mikolajczyk, and J. Matas, Trackinglearning-detection, IEEE Trans. on Pattern Analysis andMachine Intelligence, 34(7) (2012) 1409-1422.
5. C. Evans, Notes on the OpenSURF library, (University ofBristol, Tech. Rep. CSTR-09-001, January, 2009).
6. B.D. Lucas, Generalized image matching by the methodof differences, (doctoral dissertation, tech. report,Robotics Institute, Carnegie Mellon University, 1984).
7. B.D. Lucas, and T. Kanade, An iterative image
registration technique with an application to stereo
vision, in Proceedings of the 7th Int. Joint Conf. Artificial
Intelligence (IJCAI) (1981) 674-679.
8. Z. Kalal, J. Matas, and K. Mikolajczyk, P-N learning:
Bootstrapping binary classifiers by structural constraints,
2010 IEEE Conf. Computer Vision and Pattern
Recognition (CVPR) (San Francisco, CA, 2010) 49-56.
9. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, SURF:
Speeded Up Robust Features, Computer Vision andImage Understanding (CVIU), 110(3) (2008) 346-359.

Published by Atlantis Press
Copyright: the authors
147