Journal of Robotics, Networking and Artificial Life
Vol. 7(2); September (2020), pp. 86–90
DOI: https://doi.org/10.2991/jrnal.k.200528.003; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

Development of the Image-based Flight and Tree
Measurement System in a Forest using a Drone
Keiji Kamei1,*, Masahiro Kaneoka2, Ken Yanai2, Masaya Umemoto2, Hiroki Yamaguchi2, Kazuki Osawa2
Department of Production Systems, Graduate School of Engineering, Nishinippon Institute of Technology, 1-11 Aratsu, Kanda, Miyakogun, Fukuoka 800-0394, JapanDepartment of Information Systems, School of Engineering, Nishinippon Institute of Technology, 1-11 Aratsu, Kanda, Miyakogun, Fukuoka 800-0394, Japan

12ARTICLE INFOABSTRACT

Article History

Drones have been used in many purposes for a long time. Especially, development of the automatic observation systems such asmeasurement using drones for the primary sector of industry have been frequently researched in recent years. The measurementof a tree growth in a forest is also one of the aim for a drone application. In this study, our aim is to develop the automatic
measurement system for size of a tree in a forest. The difficulties are that a drone has to recognize trees, to construct a map of aforest and to measure the size of trees from a front camera. To overcome those difficulties, we propose that a drone recognizes
trees based on single shot multibox detector (SSD), constructs a map from simultaneous localization and mapping (SLAM) andmeasures a tree by structure from motion (SfM). Experimental results from the drone competition show that a drone has beenable to recognize a tree and to fly safety.

Received 02 December 2019
Accepted 08 April 2020
Keywords
Deep learning
region convolutional neural networks
structure from motion
simultaneous localization andmapping
recognition of trees
drone in a forest© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
Forests occupy about 70% of Japanese whole land area, and artificial
forests are 30% in Japanese forests. Maintenance and managementof forests is important because rundown forests cause of disasters;
however, grooming chores of forests are considered to become difficult in near future due to acceleration of demographic aging inJapan. Hence, forestry companies and its workers strongly demand
the autonomous maintenance and management system [1].
The features of Japanese forests are that trees grow on mountains, sothat there are many rocks and ground inclines in a forest. In addition,ground in a forest usually covered by underbrush. For this, a robot
with wheels or crawlers is difficult to apply for maintenance andmanagement system. To solve this difficulty, we propose to apply
a drone to developing autonomous maintenance and management
system in a forest. The abilities of a drone have been made impressive progress in recent years, so that drones are able to fly stably inany situations. On the other hand, drones have several weak points
if those fly in a forest; therefore, drones are radio-controlled bysomeone in general. Most of drones use Global Positioning System(GPS) for position estimation and measure height between a drone
and ground from altimeters or ultrasonic sensors. The signal of
GPS may be blocked by trees or be altered by reflected signal (multipath problem); therefore, receiving accurate GPS signal for construction of a map and estimating positions is difficult in a forest.
In addition to problem of flight control, a problem of development
Corresponding author. Email: kamei@nishitech.ac.jp*

for autonomous maintenance and management system is that adrone autonomously has to recognize trees, to construct map without GPS signal and to measure the size of a tree. We have proposeda method to recognize a color marker of target tree from a camera
on a drone based on particle filter algorithm [2]. Our proposal havesucceeded in recognition for the certain tree; however, our proposal
was not able to construct map and to recognize general trees without color marker.
Anzai et al. [3] have proposed to combine a Laser Range Finder(LRF) with a camera on a drone. Their proposal chooses candidatesof trees and measures of that based on the information from LRF,
then recognizes a color marker of target trees from a camera. In
addition to recognition of tree, their proposal constructed simultaneously a map of a forest. The problem of their approach is thatrecognition method needs color markers; namely, only one drone
cannot recognize category of trees. This means that their proposal
are not able to apply to the environment which has mix of several types of trees. Moreover, they need large and powerful drone
because they use several sensors and a micro-computer.
In contrast to that, our aim is that we develop a system byusing a small size and powerless drone. To realize for our aim,
we propose to apply “Object Category Recognition” based on
Convolutional Neural Network (CNN) to recognition of trees,“Structure Recognition” based on Structure from Motion (SfM) tomeasurement of size of a tree and “Map Construction” based on
Simultaneous Localization and Mapping (SLAM) to constructionof forest map. In addition, our proposal use only a camera because
our drone does not have the flight ability with other sensors.

K. Kamei et al. / Journal of Robotics, Networking and Artificial Life 7(2) 86–90

87

To the best of our knowledge, proposals which apply those algorithms to autonomous maintenance and management system forforest do not exist in the world; hence, it can be said that our proposal has academic novelty. Experimental results from the drone
competition show that a drone has been able to recognize a tree
and to fly safety.
This paper is constituted as follows. Sections 2–4 show the detailsof CNN, SfM and SLAM, respectively. Section 5 shows the experiments and its results. Finally, Section 6 concludes this paper.

Figure 1 | The structure of a typical R-CNN.

2. METHODS FOR OUR SYSTEM
In this section, we explain the summary without using the equations of algorithms for our proposal due to complexity in each algorithm and limitations of space. Please see also references for eachto know the details.

2.1. Convolutional Neural Network
Convolutional Neural Network is the one of Deep Learning
Algorithm [4]. It can be said that the success of CNN becomes aspringboard for attracting attention for multi(deep)-layered neuralnetwork. A notable successful point of CNN is that CNN is able toclassify with high rate probability for category of an object froman image.
The structure of the basic CNN is that CNN has two types of layer,i.e., convolutional and pooling layer and those layers are used inpair. Convolutional layers learns the feature of small area in images,
and pooling layers integrate outputs from convolutional layers.
From the structure, the basic CNN learns the category of object
in an image. For example, when there are dog or cat images, CNN
classify to two classes those images. To realize our aim, the classifier
has to recognize where target objects are in an image; hence, thebasic CNN is not able to be applied for our aim.
In contrast to the basic CNN, an approach for recognition of eachobject in an image based on CNN is said the Region CNN (R-CNN)[5]. Figure 1 illustrates the structure of a typical R-CNN.
The algorithm of R-CNN split an image to small region by using
sliding window as shown in Figure 1. The size and position ofsliding window is variable, so many small regions are generated.
A CNN in R-CNN learns the feature of small regions in an image
for classification of objects, and so R-CNN is able to recognize several types of objects and positions of target objects in an image.
A typical R-CNN is known to indicate the very good performance;on the other hand, it needs the very high computational capability due to huge number of input data (small regions). Several
approaches for cost down of R-CNN have been proposed, e.g., Fast
R-CNN. We apply Single Shot MultiBox Detector (SSD) [5] to recognition of trees in this research because SSD is low computationalcost. The characteristic of SSD is that an image is split into fixed
size of grids alternative to using sliding window, and SSD learns thefeatures of objects in the certain grids. For this, computational costis drastic decrease. However, recognition rate of SSD is lower thanother R-CNN methods about 10% according to paper 5, so we willvalidate whether it is capable for the SSD method to distinguish thetrees with good recognition rate in the experiments.

Figure 2 | Diagram of a structure from motion.

2.2. Structure from Motion
Measurement of size of trees is important for autonomous maintenance and management system in a forest. We propose to apply
SfM [6] to generation of 3D tree models for measurement of thesize of a tree. The measurement algorithm is that it compares acriterial tree with target tree based on generated 3D tree models.
Diagram of SfM is shown as Figure 2. The summary of SfM is generation of 3D models from integration a series of 2D images.
The details of SfM algorithm is as below. First, it discovers corresponding points from a series of images based on Optical Flowalgorithm. Secondly, sets of corresponding points are merged to aset of corresponding points, and point group data is generated fromthose points. Finally, a 3D image model is generated by pasting aseries of images to point group data. Note that point group data
correspond 3D models, and so measurement of size of trees are notnecessarily have to need 3D image models; therefore, generationand correspondence among images for point group data is the mostimportant for measurement.

2.3. Simultaneous Localization and Mapping
A drone must have a map of forest in advance to measure the sizeof trees. However, it is difficult to prepare the exact map of a forest
in advance because states of a forest vary depending on seasons.
For this, construction of exact map for a forest is also important
for autonomous maintenance and management system. We propose to apply SLAM [7] for construction of exact map. SLAM is amethod to estimate positions (localization) and to construct map(mapping) simultaneously by only using sensory information androbot control signals. Several types of SLAM have been proposed,
we apply Visual SLAM to construction of map. Visual SLAM constructs a map and estimates position based on a series of images, soit is said that Visual SLAM is as same as SfM on generating of point
group data; namely, SfM uses point group data for construction of3D image models, and Visual SLAM uses that for localization andmapping. There are several methods for Visual SLAM, we utilize

K. Kamei et al. / Journal of Robotics, Networking and Artificial Life 7(2) 86–90

88

Oriented FAST and Rotated BRIEF (ORB)-SLAM [8] because ORBSLAM shows good performance for mapping and localization.

shows the result of generation of point group data for trees in ouruniversity.
From Figure 4b, point group data are generated from only trees
instead of background objects of trees compared with Figure 4a.

3. EXPERIMENTS
We conduct two kinds of experiments, i.e., in laboratory and in “The4th Forestry Drones & Robots Competition [1].” This section shows
the details of a drone, then experimental results. From the competition, our target tree is cedar, and so our final goal is construction of
the autonomous cedar maintenance and management system.

3.1. Drone: Tello
In this research, we use “Tello [9]” drone. Tello has a front camera,
which is 5 (M) Pixel and 82.6 (°) field of view (FOV). The weight ofTello is about 80 (g), so Tello is very light drone. Our system controls Tello from a laptop PC via WiFi connection.

3.2. Experiment 1: In Laboratory
First, we show the results of R-CNN. In this study, we use Google
Object Detection API [10] for R-CNN to accelerate of development.
In R-CNN experiment, we took a video of cedars in advance in aforest where is near our university. We use that video to training ofR-CNN and its evaluation. Figure 3 shows the recognition result.
From the result, R-CNN found a cedar in region enclosed bygreen rectangle; hence, R-CNN succeeded in recognition a cedar
although a part of cedar is hidden by other trees. For this reason, it
can be said that SSD algorithm has enough ability for recognitionand finding cedars.
Second, we show the result of SfM. 3D image model of SfM wasconstructed in real time processing with flying Tello. In thisstudy, we use OpenMVG, OpenMVS and Meshlab [11]. Figure 4

Figure 5 shows the result of construction for 3D image model of apillar of our university.
From Figure 5b and 5c, we were able to generate 3D images insteadof planar image, so that we succeeded in construction for 3D image
model by using SfM.
Finally, we show the result of SLAM. We use open library of ORBSLAM [8] as same as R-CNN and SfM to construct map of the hall.
Figure 6a and 6b shows detection of corresponding points in animage and results of construction the hall map, respectively.
Results of SLAM show that corresponding points were exactly
detected in a series of images from Figure 6a, and a map surrounding a pillar in hall was correctly generated from Figure 6b; therefore, we succeeded in construction of a map based on SLAM.

3.3. Experiment 2: Competition
We integrated R-CNN, SfM and SLAM in order to develop theautonomous maintenance and management system; then, we participated in drone competition [1] to evaluate our system. Figure 7
illustrates the image of a part of session for that competition. Thereare some target trees in the competition field. One of target trees isthe criterial tree that has color markers. To get points, a drone hasto get closer to less than about 1 (m) of a target tree; then, a dronemeasures the size of that tree.
Figure 8 shows a scene of flight of a drone to measure the criterial
tree with our system.
abc

Figure 5 | Result of SfM. (a) Target pillar. (b) and (c) 3D image model.
Figure 3 | Recognition result of a cedar.

ab

Figure 4 | Generation of point group data. (a) Target trees. (b) Point
group data.

abFigure 6 | Result of SLAM. (a) Detection of corresponding points.
(b) Generated hole map.

K. Kamei et al. / Journal of Robotics, Networking and Artificial Life 7(2) 86–9089CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

REFERENCES

Figure 7 | The image of a part of session for that competition [1].

Figure 8 | The scene of flight of a drone (In competition).

From Figure 8, a drone succeeded in going to near a criterial tree
because R-CNN and SfM module correctly operated; however, adrone were not able to construct a forest map due to problem ofSLAM module. As a result of drone competition, it can be said thatwe succeeded in development a part of autonomous maintenanceand management system.

4. CONCLUSION
In this study, our aim is to develop the automatic measurement
system for size of a tree in a forest. The difficulties are that a drone
has to recognize trees, to construct a map of a forest and to measure
the size of trees from a front camera. To overcome those difficulties, we propose that a drone recognizes trees based on R-CNN,
constructs a map from SLAM and measures a tree by SfM. As aresult, it can be said that we succeeded in development a part ofautonomous maintenance and management system. Precise integration of R-CNN, SfM and SLAM is left for further study.

[1] Center for Socio-Robotic Synthesis, Forestry Drones & Robots
Competition (in Japanese), Kyushu Institute of Technology,
Wakamatsu-ku, Kitakyushu, Fukuoka, Japan, 2019, Available
from: www.lsse.kyutech.ac.jp/~sociorobo/ja/forestdrone2019.
[2] K. Kamei, K. Narumi, D. Shuto, Adaptive color tracking forparticle filter to control a drone in a forest, 2016 Joint 8thInternational Conference on Soft Computing and Intelligent
Systems (SCIS) and 17th International Symposium on
Advanced Intelligent Systems (ISIS), IEEE, Sapporo, Japan,
2016, pp. 512–517.
[3] T. Anzai, M. Zhao, F. Shi, K. Okada, M. Inaba, Sensing and navigation of aerial robot for measuring tree location and size inforest environment, 2017 International Workshop on Smart InfoMedia Systems in Asia (SISA), Kyushu Institute of Information
Sciences, Dazaifu, Fukuoka, Japan, 2017, pp. 55–60.
[4] A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classification
with deep convolutional neural networks, Adv. Neural Inform.
Process. Syst. 25 (2012), 1097–1105.
[5] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.Y. Fu, et al.
SSD: single shot multibox detector. In: B. Leibe, J. Matas, N. Sebe,
M. Welling (Eds.), Computer Vision – ECCV 2016. Lecture Notes
in Computer Science, Springer, Cham, 2016, pp. 21–37.
[6] F. Bethmann, T. Luhmann, Object-based multi-image semi-global
matching – concept and first results, Int. Arch. Photogramm.
Remote Sens. Spatial Inf. Sci. XL-5 (2014), 93–100.
[7] S. Thrun, W. Burgard, D. Fox, Probabilistic robotics, MIT Press,Cambridge, MA, USA, 2005.
[8] E. Rublee, V. Rabaud, K. Konolige, G. Bradski, ORB: an efficient
alternative to SIFT or SURF, International conference on computer vision, IEEE, Barcelona, Spain, 2011, pp. 2564–2571.
[9] Ryze Tech: Tello, Available from: www.ryzerobotics.com/jp/tello.
[10] Google Object Detection API, Available from: github.com/tensorflow/models/tree/master/research/object_detection.
[11] P. Moulon, P. Monasse, R. Perrot, R. Marlet, OpenMVG: open multiple view geometry, In: B. Kerautret, M. Colom, P. Monasse (Eds.),
Reproducible Research in Pattern Recognition, RRPR 2016, Lecture
Notes in Computer Science, Springer, Cham, 2017, pp. 60–74.

AUTHORS INTRODUCTION
Dr. Keiji Kamei
In 2007, he obtained his PhD at theDepartment of Brain Science andEngineering,Kyushu
InstituteofTechnology (Engineering). From April
2007 to March 2014, he was a lecturer.
Since April 2014, he has been an associate professor, Nishinippon Institute ofTechnology. His research interest is Artificial Neural Networks,Artificial Intelligence and Computer Science. He is a member ofJNNS, APNNS, IPSJ, IEICE, SOFT.

Mr. Masahiro Kaneoka
He is a bachelor research member inKamei laboratory at Nishinippon Instituteof Technology. His research interest isWeb Marketing and Web Design.

K. Kamei et al. / Journal of Robotics, Networking and Artificial Life 7(2) 86–90

90

Mr. Ken Yanai

Mr. Hiroki Yamaguchi
He is a bachelor research member inKamei laboratory at Nishinippon Instituteof Technology. His research interest isComputer Aided Education.

Mr. Masaya Umemoto
He is a bachelor research member inKamei laboratory at Nishinippon Instituteof Technology. His research interest is
Artificial Life System.

He is a bachelor research member inKamei laboratory at Nishinippon Instituteof Technology. His research interest is
Artificial Neural Networks.

Mr. Kazuki Osawa
He is a bachelor research member inKamei laboratory at Nishinippon Instituteof Technology. His research interest isArtificial Neural Networks.