Mobil cihazlar ile derin öğrenme mimarisi kullanarak gerçek zamanlı video nesne izleme
Deep learning architectures for real-time video object tracking by mobile devices
Yazar: ULAŞ TOSUN
Danışman: PROF. DR. BİLGE GÜNSEL KALYONCU
İstanbul Teknik Üniversitesi / Fen Bilimleri Enstitüsü / Elektronik ve Haberleşme Mühendisliği Ana Bilim Dalı / Telekomünikasyon Mühendisliği Bilim Dalı
Derece: Yüksek Lisans, 2019
Konu: Bilgisayar Mühendisliği Bilimleri-Bilgisayar ve Kontrol = Computer Engineering and Computer Science and Control ; Bilim ve Teknoloji = Science and Technology ; Elektrik ve Elektronik Mühendisliği = Electrical and Electronics Engineering

Özet:
Yeni gelişen teknolojilerle ve elektronik donanımlardaki gelişmelerle birlikte başta akıllı telefonlar olmak üzere mobil cihazların performansı ve buna bağlı olarak da kullanım alanları gitgide artmaktadır. Günümüzde özellikle akıllı telefonlar en az bilgisayarlar kadar çok kullanılmaktadır ve insanoğlunun günlük yaşantısının vazgeçilmez bir unsuru haline gelmiştir.
Öte yandan literatürdeki pek çok nesne izleme ve nesne tespit yöntemi sadece bilgisayarlar üzerinde kullanıma uygun olacak şekilde tasarlanmış olup işlem gücü nispeten daha düşük cihazlar olan mobil cihazlar üzerinde kullanılmaya uygun değillerdir. Bunun bir sonucu olarak nesne tespit ve izleme alanında geliştirilen pek çok yöntem akıllı telefonlarda kullanılamamaktadır.  Bu durum nesne izleme algoritmalarının birçok günlük uygulamada kullanılmasını engellemektedir.
Bu tez çalışmasında, mobil cihazlarda çalışmaya uygun, düşük işlem gücüne ihtiyaç duyan MobileNet SSD nesne tespit algoritması kullanılarak, videolarda gerçek zamanlı nesne izleme yapabilecek bir mobil uygulama hedeflenmiştir. Mobil cihazlarda çalışmanın bir avantajı bilgisayarlarda bulunmayan çeşitli algılayıcılara sahip olmalarıdır. İvmeölçer, ışık algılayıcısı, jiroskop ve GPS gibi örneklendirilebilecek olan bu algılayıcılar aynı zamanda nesne izleme performansını arttırmak amacıyla da kullanılabilirler. Bu tez çalışmasında, nesne izleme performansının arttırılabilmesi amacıyla, ivmeölçerden alınan veri kullanılarak hedef izleme gerçeklenmiştir. Bu kapsamda öncelikle ivmeölçerden gelen veri çeşitli frekans karakteristiğine sahip gürültülerden arındırılmıştır. Ardından elde edilen anlamlı ivme verisi hız verisine dönüştürülmüştür. Kameranın üzerinde bulunduğu mobil cihaza ait olan hız verisi kullanılarak,  kamera hareketi modellenmiştir. Bu sayede hareketli kamera durumunda nesne izleme performansının arttırılması hedeflenmiştir. Çalışmada SSD MobileNet, özellikle hız yönünden benzerlerinden olumlu anlamda ayrılması ve mobil cihazlarda gerek duyulan hızlarda çalışabilmesi nedeniyle tercih edilmiştir.
Kamera hareketinin doğrulukla hesaplanabildiği bir ortamda nesne izleme performansını arttırmak amacıyla çok çeşitli yöntemlere başvurulabilir. Bu tez çalışmasında öncelikle hesaplanan kamera hareketinin nesne hareketi üzerindeki etkisi kompanze edilerek gerçek nesne hareketi modellenmiştir. Ardından elde edilen nesneye ilişkin hareket bilgisi SSD MobileNet nesne tespit algoritmasıyla birleştirilerek nesne izleme performansı arttırılmıştır.
Tez çalışmasında SSD MobileNet derin öğrenme ağı yeniden eğitilmemiş,  kullanıma açık olan ve MS COCO veri seti üzerinde eğitilmiş, 80 farklı nesneyi tespit edebilen bir MobileNet SSD eğitim modeli kullanılmıştır. Öncelikle MS COCO eğitim seti üzerinde nesne tespit performansı incelenmiş, ardından literatürde sıklıkla kullanılan VOT 2016 video veri tabanındaki başarım raporlanmıştır. Ayrıca tez çalışması için özel olarak hazırlanan ve mobil telefon ile günlük hayatın içinden çekilmiş videolar içeren özel bir veri seti üzerinde de sonuç gösterilmiştir. Bunun yanı sıra bir mobil nesne sezme ve izleme uygulaması Android ortamında gerçeklenerek başarımı raporlanmıştır.
Sonuç olarak bu tez çalışmasında öncelikle, bilgisayarlar üzerinde kullanılan nesne tespit ve izleme yöntemlerinin başta akıllı telefonlar olmak üzere mobil cihazlara da entegre edilerek günlük hayatın içerisinde daha fazla yer alabileceğinin gösterilmesi hedeflenmiştir. Ayrıca mobil cihazlar üzerinde çalışmanın en önemli kazanımlarından birisi olan algılayıcılardan, özel olarak ivmeölçerden, alınan veri kullanılarak hareketli kamera ile izlenen nesnelerin tespitinde performansın arttırılabileceği gösterilmiştir.

Abstract:
With the advances in new technologies and developments in electronic hardware, the performance of mobile devices, especially smartphones, and their usage areas are increasing. Nowadays, especially smart phones have been used as much as computers and have become an indispensable element of human life. According to market share data, people use smartphones more than computers for accessing to internet and also most of the video data have been capturing by smartphones.
In addition, manufacturers improves processing capacity of modern smartphones in every year. Smartphones, especially flagship models, have sufficient processing power to run simple neural networks that are used by mobile applications. Despite to improvements on their hardware, computational power of mobile devices is still less than computers that have a modern graphical processing unit in terms of running complex neural networks such as CNNs. On the other hand, complex neural networks such as CNNs are used by mobile applications for various tasks. For example, camera applications detect faces for face beauty perfection. They classify scenes and detects objects to optimize camera parameters and capture better images.
However, most of the existing state-of-the-art object tracking and object detection methods such as Mask R – CNN and Retina Net are designed for use only on computers and are not suitable for use on mobile devices because of their demanding processing units. Consequently, most of the research efforts on video object tracking and detection cannot be adopted to mobile devices although they are more common in everyday life than computers. This prevents development of the applications that employ object detection and tracking to find more space in daily life. In order to overcome these limitations, MobileNet SSD is introduced as a real-time object detection algorithm. MobileNet SSD architecture includes an object detection network called SSD located at the top of a backbone neural network called MobileNet. MobileNet SSD is capable of working in real-time on mobile devices, because its CNN layers perform a special type of convolution called depth wise separable convolution. This is why we prefer MobileNet SSD as the baseline algorithm within the context of this thesis.  YOLO is another deep network architecture designed to speed up the object detection. It is reported that YOLOv3 is capable of real-time object detection. Despite their speed, neither MobileNet SSD nor YOLOv3 provide the detection accuracy as high as the conventional deep object detectors. This is mainly because both of them pass over the input image once that significantly reduces the computational complexity. However, as it is expected, there is a tradeoff between the detection accuracy and the computational complexity. Recently a number of new deep architectures including lite R-CNN are introduced to improve accuracy while decreasing the computational complexity. 
In the context of this thesis, we also aim to improve the accuracy of MobileNet SSD without increasing the computational load. Unlike the existing approaches, we propose to employ a number of different measurements for object detection along with the video data extracted by the mobile camera. Because, one of the positive things about working on mobile devices is that many of the mobile devices have various sensors that are not available on computers. In particular, we propose usage of these sensors, such as accelerometers, light sensors, gyroscopes and GPS to improve object tracking performance. The combination of image data and environmental sensor data is also proposed by some other application areas such as autonomous cars, drones and other robotic applications.
To improve object tracking performance various environmental sensors could be used at the same time or standalone. However, using multiple sensor results with many different problems such as noise and synchronization. In order to simply the real-time object monitoring, in the context of the thesis, it is decided to use only the accelerometer in order to increase object tracking performance in case of moving camera. In particular acceleration measurements taken from the accelerometer can be used for the calculation of velocity. In this context, the data obtained from the accelerometer is filtered and the noise with various frequency characteristics is eliminated. It is observed that the acceleration data is distorted by two types of noise. The first one is measurement noise eliminated by low pass filtering because it has higher frequency components. The second one is gravity effects having a lower frequency characteristic thus eliminated by high pass filtering. After noise elimination, it can be reached to the velocity data by applying a discrete integral operation on the measurements. We propose to use the velocity to extract the camera motion under the assumption that the mobile device and its camera move synchronized. This is followed by the affine motion parameter estimation that provides the motion parameters of mobile camera and also allows us to classify the camera motion as pedestal, track, etc. for different semantic indexing applications. It is possible to reach actual object's motion, if it is desired, by eliminating the camera motion but conventionally the object tracking deals with the estimation of the new object location with respect to the previous frame rather than estimating actual object motion. The proposed object tracking method employs MobileNet SSD guided by the motion estimated from the accelerator measurements. It is shown that fusing the sensor measurements into the deep object detector increases the tracking accuracy especially under motion blur and severe illumination changes where the deep detector fails to track. 
In addition, since the camera motion parameters are estimated by using the velocity data, the accuracy of these estimations is tested by using a feature based motion estimation model.  The model detects the corner points on the input video frame by using Harris corner detector. Then, for each corner point, FREAK descriptor assigns a binary feature vector is. This is followed by matching the corner points of successive video frames by minimizing Hamming distance between the descriptors. Finally, under the assumption of 2D affine motion, affine motion parameters corresponding the mobile camera motion are estimated by a least squares estimator. In this work, it is shown that the camera motion parameters calculated by the feature based estimation and by using the accelerator measurements are very close, that confirms the accuracy of our measurements.
The stand-alone object detection performance of MobileNet SSD is evaluated on MS COCO 2014 dataset, since the deep detector is trained on this data set. The idea behind this is to report the highest detection accuracy achieved by the employed deep object detector. Accuracy is reported by precision, recall and F1 measure metrics, which are commonly used in object detection benchmarking challenges. In order to observe the improvement achieved by MobileNet SSD guided by the accelerometer measurements, the object tracking performance is reported on VOT 2016 dataset that is a commonly used benchmarking data set in object tracking. Accuracy and robustness metrics are reported for each video sequence as well as the mean obtained on the full data set. Finally, object tracking performance achieved on a special dataset that is prepared for this thesis study is evaluated and reported. The dataset contains daily life videos captured by a smartphone in different scenarios.
In addition, this method was investigated in terms of speed compared to similar methods. This comparison is supported with measurements on mobile devices and results have been reported.
In addition to all, 3 Android applications were written for this thesis. These applications were prepared by using Android Studio environment. OpenCV Android library was used for developing these applications and implementation details was reported. One of these applications is the main application and it runs on real time. Other two run offline and they were developed for offline tests. Matlab is the environment which is used for result analysis and plotting data.
As a result, in this thesis, it is aimed that object detection and tracking methods used on computers in general are integrated into mobile devices, especially smart phones, which are more common in daily life than computers. In addition, thanks to sensors, one of the most important gains of working on mobile devices, it is aimed to increase object tracking performance in case of moving camera. All these goals have been reached under various assumptions and details are included in the thesis study.
In this study, the other details of this method such as the combination of SSD and MobileNet and its principles were given in detail. In addition, other modern methods have been mentioned under the title of literature study. Also, the method used in the thesis study has been compared with the similar ones in literature in terms of both speed and accuracy. Organization of thesis is as follows. Firstly, an introduction and literature survey is given in Chapter 1. Then, SSD and MobileNet is discussed in detail in Chapter 2. Camera motion analysis is explained in Chapter 3 and combination of motion data and MobileNet SSD is explained in Chapter 4. After, test result for different data sets and conditions is given in Chapter 5. Finally, the conclusion is given in Chapter 6.