Makine öğrenmesi ve görüntü  işleme tekniklerini kullanarak drone ile yaprak sınıflandırma
Leaf classification with drone by using machine learning and image processing techniques
Yazar: MEHMET ÖZTÜRK
Danışman: DR. ÖĞR. ÜYESİ NAZIM KEMAL ÜRE
İstanbul Teknik Üniversitesi / Fen Bilimleri Enstitüsü / Uçak ve Uzay Mühendisliği Ana Bilim Dalı / Uçak ve Uzay Mühendisliği Bilim Dalı
Derece: Yüksek Lisans, 2018
Konu: Uçak Mühendisliği = Aircraft Engineering

Özet:
Bu tez çalışmasında makina öğrenmesi ve görüntü işleme tekniklerini kullanarak,
drone ile farklı bitki çeşitlerinden ortaya çıkan yaprakların ¸sekilsel olarak sınıflandırılması
yapılmıştır. Dünya üzerindeki insan nüfusunun artması ile beraber ortaya
çıkan "Hassas Tarım" ile birlikte teknoloji ile tarım daha içe içe olmuştur. Bu tez
çalışmasındaki temel motivasyon, insansız hava araçları vasıtasıyla otonom bir şekilde
yaprak çeşitlerini tanıyarak tarımda kimyasal ve biyolojik iyileşmenin önünü açmaktır.
İlaveten, drone görüntülemesi sayesinde bitkisel hastalıklar ve böceklenme gibi
ortaya çıkabiliecek sorunlar daha önceden fark edilerek gerekli tedbirlerin alınmasını
kolaylaştırmaktadır. Bu çalışma içerisinde yer alan metotlar kullanılarak insansız hava
araçları sayesinde taranan bölgelerin bitki veri tabanı oluşturulabilir, bu sayede daha
sürdürebilir bir bitki ekosistemi kurulabilir.
Temel olarak çalışma 4 bölüme ayrılmıştır. 1) Literatür taraması yapılarak bu
alanda yapılmış çalışmalar incelenmiştir. Böylece, bu tezde yer alması hedeflenen
çalışmalara sıfırdan başlanıp mevcut literatür ile tekrar düşmektense, var olan
çalışmaların eksik/yetersiz olduğu yönler tespit edilerek bu konuların üzerine
gidilmiştir. İlaveten, bu çalışmalar sırasında Flavia, imageclef datasetlerinin yanında,
çeşitli mesafelerden kendi drone ile çektiğimiz görüntü dataları da kullanılmıştır. 2)
Elde bulunan veritabanındaki görsellerde yaprakların resimleri arkaplanda bulunan
ambiyans ile bir bütündü çalışmanın başında. Lakin, sınıflandırmanın doğru yapılması
için, yaprak görsellerinin arkaplanda gürültü unsuru olabilecek görselliklerden
ayrılması gerekmektedir. Bu nedenle, tüm görsellerde yeşil olmayan arkaplan
çıkartılmıştır. 3) Drone ile daha uzak mesafeden görüntü alınarak uzaktan yapılan
çekimlerin sonucudaki başarı oranı sınanmıştır. Bu bölümdeki başarı oranını
arttırmak amacıyla RDP doygunluk (saturation) ve Parlaklık (value) eşitlemesi gibi
teknikler uygulanmıştır. 4) Üst üste binmiş yapraklar üzerinden elde edilen yaprak
görünütülerinin sınıflandırılması yapılmıştır. Daha önceki bölümlerde arka planda
yaprak harici nesnelerin bulunduğu durumlar incelenmiştir. Bu bölümde ise, arka
plandaki nesneler içinde yaprakların da olabileceği görseller incelenmiştir. Bu
tarz durumlardaki sınıflandırmaların başarıya ulaşması için Watershed algoritması
uygulanmıştır.
Her bölümde de, temel olarak görüntünün işlenmesi, özelliklerin cıkartılması,
özelliklerin öğretilerek SVM ile sınıflandırma yapılmıştır. Çalışmanın sonunda,
insansız hava aracı ile belirli bir bölgede görüntü işleme ve makine öğrenmesi ile
yapılacak olan yaprak sınıflandırmasının %91,3 oranı ile başarıya ulaşacağı bir sistem
tasarlanmıştır.

Abstract:
In this thesis study, classification of different tree leaves are achieved through machine
learning and image processing techniques. The study's experimental aspect is
conducted using drones to capture images from multiple aerial points of view. The
primary motivation of this thesis is the stabilization of ecological floral balance and
the progression of precision agriculture. Plants have a major role on ecology on Earth
and are the fundamental basis of the energy pyramid. Henceforth, the technological
involvement on the sustainability of plants is vital as the human population increases
exponentially. A central methodology of achieving such goal relies on autonomy,
and a widespread choice of autonomous vehicles are drones due to their cheap cost
and easy maintenance. This study, thus, focuses on the application of drones on
such appliances. This thesis targets the recognition and classification of various leaf
shapes for the purpose of paving the way for chemical and biological developments
in agriculture, as well as preventing detrimental effects on plants due to illnesses and
bug/pest contamination. In addition, with the help of drone footage captures, a floral
database of plants may be constructed, which in return, supports a progression towards
a more sustainable ecological balance.
In existing literature, there does not exist any direct work on the classification
of tree plants using machine learning and image processing. However, there do
exist studies on photographs taken from cameras. Some of these studies include:
Flavia leaf database, neural network leaf classification from leaf vessels, multi-path
curvelet transformation, et cetera. Since the application of drones is similar to static
camera approaches, previous studies can be used as to find a starting basis for the
work comprised in this thesis. However, the application of drones and autonomous
categorization is not found in existing literature. This study contributes to both the
spread of drone use and the support to agriculture and farmers. On the other hand,
there exists multiple challenges, some being: the angle and altitude of the quad-copter,
distance to leaf, vibrations inflicted on of footage capture device, luminous intensity.
The studies in this thesis can be divided into 4 parts:
1) Literature survey is conducted to gain an understanding of previous literature. By
doing so, the techniques aimed to take place in this thesis can focus on the drawbacks of
previously existing literature by benefiting from the methodologies presented in them
and consequently building the work of this thesis on them. This approach both prevents
the repetition of existing literature and enables the improvement over previous work.
In addition, this study uses drone captures along with imageclef and Flavia datasets,
which permit a much more wider range of leaf types and photo angles to study on.
Though, before the use of Flavia data, preprocessing to utilize the most important parameters of the images have been conducted. That being said, image processing has
its own difficulties, mainly due to: i) data loss from the transition of 2D from 3D, ii)
the necessity of producing a different mathematical model for each and every object,
iii) excessive amount of noise in the real world, iv) storage problems of multimedia
data, v) number of pixel and luminous effects captured. To counter such problems,
multiple properties of photographs are to be modified before classification. The RGB
is the first of these, and signifies the combination of red, green and blue pixels on the
photograph in a numerical way. The second is the intensity, which is a single matrix
signifying the closeness of a pixel to absolute white of absolute black.
The properties identified for the machine learning algorithm to use are defines as
follows: I) The diameter of the leaf, which can be measures as the distance between
the farthest two points on the leaf. This distance is found after the Laplacian and
average filtering, where the distance is in terms of Euclidean norm. A Matlab algorithm
using regionprobs is also used, which returns measurements for the set of properties
specified by properties for each 8-connected component (object) in the binary image.
II) Physiologic distance, which coincides to the distance between two points on the
leaf main vessel. This distance has to be input manually under normal circumstances.
However, the Matlab function of regionprobs was used to automatize to process. III)
Physiologic width, which are orthogonal lengths to the physiologic distance. To
automatize and simplify the interpretation, the short diameter of the leaf is taken into
account. IV) Leaf area, which is the amount of white pixels. V) Perimeter, which is
the amount of pixels covering the leaf margin.
2) The database from Flavia consists of photographs which have absolute white
backgrounds behind the leaves. However, in actual footage obtained from drones, it is
practically impossible to obtain photographs with clear backgrounds. And so, it is vital
to first clear any irrelevant objects in the background before attempting to classify the
leaves. This section covers the cases where the background only consists of non-leaf
material, where in short, the approach removes all of the non-green toned objects in the
image. By doing so, the leaf object is extracted ftom the image's irrelevant background
objects. The methodology for solving such problem was achieved via a technique
called Thresholding, which is the simplest form of segmentation of an image. If the
red and green values are increased in the RGB channel, then the extraction of the
lead can be achieved. Experimental studies conducted to verify this approach have
shown that the leaf object is easily extracted from the background when red and green
channels are intensified.
3) This part of thesis examines the success rate of long-distance drone footage captures.
Shots from 2, 4, 7 and 11 meters were taken via a drone camera. Algorithms developed
in the previous sections were put into trial. However, major reasons are due to the
distance of leaves from the camera, the leaves were mostly not recognized and an
intolerable failure surfaced. It was also seen that besides the factor of distance between
leaves and the drone, luminous intensity, color and contrast also has a major effect.
The HSV (hue-saturation-value) has been modified in the purpose to increase the
success rate of leaf recognition from photographs. When the saturation and the value
histogram equalizations are performed, the leaf was successfully extracted from the
image. However, this caused other irrelevant objects in the background with green
color to be extracted as well. The value of the HSV was transformed to histogram
equalizations for Matlab to correctly interpret, but an even better solution turned out to be the imadjust method where differecent contrast values at tried. The contrast
reduction technique is similar to histogram equalization, however, as opposition to
histogram equalization, contrast reduction equals values beneath average to minimum,
and above average to maximum. This causes a binary outcome, which is not handy in
application. This surfaces the necessity to manually remove all other irrelevant objects
in the background.
4) This section of the thesis covers the cases where different types of leaves are present
in the background of the primary leaf. For images shot from a single angle, difficulties
arise for determining the depth of the leaves. To counter this difficulty, a method called
Watershed has been implemented. The concept of Watershed relies on projecting a 3D
image on a 2D space with intensity image. This is achieved by finding three points:
regional minimum, water-drop points (where water will definitely flow), and equal
water level points (flow to multiple minimums). The second of these conditions are
catchment basins and the third of these conditions are divide lines. The main purpose
of this concept is to find the watershed lines. The intensity component becomes the
main property in distinguishing the primary leaf from other leaves.
For each of the four parts, the main application is the processing of the images,
extraction of dominant properties, learning the properties and categorization using
SVM. For the first part, the results show that the success rate is 91.3% compared to
previous studies. The necessity for manual interference has been overcome and the
process has been automatized. The the second part where the non-green background
is removed, the study does not show any significant difference with respect to distance
from leaves. For the third part where long-distant captures are classified, the error
for such cases are less than 1%. For the last part where leaves on top of each other
are examined, the segmentation is divided into three titles: good, bad, failed. A good
segmentation is where the extracted leaf is very close to the primary leaf shape in the
actual photo. A bad segmentation is where incomplete extraction is present from the
photo. A failed segmentation is is irrelevant background images are present in the
extraction and the extracted leaf image is unusable.
The main purpose of this thesis was the classification of leaf images obtained from
drones using machine learning, and the automation of this process. This, in return,
will elevate the use of real-time applications in agriculture while decreasing human
intervention to the process of plant classifications. The results of such developments
are smart environmental surveillance of flora, precision agriculture, and furthermore,
the proper determination of plant ecology on Earth. Another benefit is the use of such
system where humans are incapable of entering, such as forests and mountains where
accessibility to such area would be possible for a real-time autonomous scan of the
terrestrial texture of such inhabitable locations.
The experimental studies taken place in this thesis has been conducted in Control and
Avionics Laboratory and Aerospace Research Center in Istanbul Technical University,
where the flights of the drone, of which captured photos have been used as a dataset,
have been performed by a trained pilot.