Development of a Mobile Robot for Harvest Support
in Greenhouse Horticulture– Person Following and Mapping –Hiroaki Masuzawa, Jun Miura, and Shuji Oishi
Abstract— Agriculture has been suffering from many problems nowadays, and labor shortage is one of the most serious
ones. Automation of agricultural works is a possible solution
and many efforts have been made for developing harvest support robots. This paper discusses an application of mobile robot
technologies to harvest support in the greenhouse horticulture
domain. Although there are several harvest support robots
for fruits and vegetables, they usually require modificationof a greenhouse for autonomous navigation. We therefore
aim to develop a mobile robot that can operate and support
harvesting works in normal greenhouses, especially for flowers.
Based on an investigation of necessary technologies for harvest
support robots, we develop a prototype mobile robot withperson following and 3D mapping capabilities. We performed
preliminary experiments in a real greenhouse and evaluatedthe developed robot system.

In this paper, we first discuss necessary technologies formobile robots for harvest support. We then describe theimplementation of some of key technologies, that is, person
following and map generation in greenhouse domain. We
also describe experimental results to confirm the feasibilityof the developed system.
The rest of the paper is organized as follows. Sec. II
discusses necessary technologies for harvest support mobile
robots. Sec. III explains the design and the specifications ofa prototype robot. Sec. IV and Sec. V describe the person
following and the mapping function of the robot. Sec. VI
show experimental results in a real greenhouse. Sec. VII
concludes the paper and discusses future work.

I. I NTRODUCTION

II. N ECESSARY T ECHNOLOGIES FOR H ARVEST S UPPORTM OBILE ROBOTS

Agriculture has been suffering from many problems nowadays, and labor shortage is one of the most serious ones. The
portion of the elderly in agriculture is significantly increasingin Japan; as of 2015, the ratio of farmers over 65 years oldreached 63.5% (1.32 million) [1]. Improvement of the qualityof agricultural crops and the reduction of production cost arealso issues to be resolved for making Japanese agriculture be
more competitive in the world market. In addition, as the total
population in the world is expected to reach 9.2 billion in
2050, increasing the productivity of agriculture is a common
problem in the world.
In order to solve these problems, efforts towards theautomation of agricultural works are being carried out under
the initiative of the Government of Japan. One popular
example is the unmanned operation of agricultural machineryin outdoor such as tractors and rice planters [2]. Sucha machine can move with a positional error within 5cm
using a highly accurate GPS sensor and an inclinometer.
In the greenhouse horticulture domain, automatic harvesting
robots for, for example, strawberries [3] and tomatoes [4]are being developed; computer vision techniques are used
for recognizing fruits [5]. In many cases, however, in order
to use such automated systems, it is necessary to modify agreenhouse for robots; for example, they sometimes need to
install rails for guiding a robot. It is still difficult to use themin the current greenhouse. Therefore, in this research, we aim
to develop a mobile robot that can support harvesting workin greenhouses, especially those for flowers.
The authors are with the Department of Computer Science and Engineering, Toyohashi University of Technology, Toyohashi, Japan.

Aichi prefecture where our university exists has highest
share of many flowers including roses and chrysanthemums.
We visited several farms cultivating flowers in greenhouses
in order to investigate necessary functions that mobile robots
for flower harvest support need to possess.
One of the key issues in keeping the quality of cut
flowers is to shorten the time between reaping flowers andputting them into water. For this purpose, we are aiming at
developing a mobile robot which carries a water tank andautomatically follows a worker. The robot also moves backto a station once an enough amount of flowers are reaped.
Fig. 1 illustrates how the robot works and what its necessaryfunctions is. In this paper, we focus of two of them. One isto follow people for carrying items necessary for harvesting.
The other is to make a map and localize the robot in the map
for autonomously carrying harvest to a specific station.
Although these two functions are quite popular in themobile robotics domain, realizing them for flower harvest
support requires us to consider a special characteristic ofgreenhouse. In a greenhouse, we often see heavily-grownplants as shown in Fig. 2. Such a situation can arise as plants
grow naturally, but also is artificially made by farmers. Since
photosynthesis processes are very important for the growthof plants, branches of plants are expanded and/or bent fromcrop rows to aisles, and this expansion/bending is different
from farm to farm. As a result, plants often occlude aisles
in various ways and ordinary crop row detection approaches(e.g, [6], [7], [8]) cannot be applied. This also makes difficult
to detect people as well as perform mapping and localization.

(a) greenhouse 1.

Fig. 1.

Tasks and functions of flower harvest support robots.

Concerning people detection and tracking, we need todevelop a method which can cope with heavy occlusions
by and/or contact with plants. Leg and/or body detection
by a 2D LRF (e.g., [9]) or blob detection using 3D point
clustering (e.g., [10])do not work well.
Concerning mapping and localization, since traversable
regions (i.e., aisles) are sometimes not visible (see Fig. 2),
we need to discriminate fake obstacles, which are usuallyplants that can be pushed away, from real obstacles to avoid.
Navigation in such a scene is very challenging. We also
consider plants’ flexibility in 3D scan matching for egomotion estimation and localization. A long-term change ofplants could also be factors to be considered.

(b) greenhouse 2.

Fig. 2.

Greenhouse scene examples.

III. P ROTOTYPE ROBOTA prototype robot is shown in Fig. 3. The size of therobot is 450mm in width, 700mm in depth, and 1350mm
in height. The width and the depth were determined basedon those of a cart actually used for carrying plants ingreenhouse. The height is determined so that the view of thecamera on the top is not obstructed by plants. In addition
to a computer controlled usage, the robot has a power assist
mechanism which makes the manual control easy.
We use an RGB-D camera (Kinect v2 by Microsoft) for
measuring distance to objects. We could use a 3D LIDAR
instead but it is too costly for a usual operation. A 2D
LIDAR is also a possible option but it is not suitable forplant-covered environments. Although the robot is equippedwith a 3D LIDAR (VLP-16 by Velodyne) and a 2D LIDAR(UST-20LX by Hokuyo), they are used only for evaluation
purposes. The PC used for control and sensor data processing
is the one with Core i7-6700HQ CPU and NVIDIA GeForceGTX 960M GPU.
IV. P ERSON F OLLOWINGThe person following capability is realized by a combination of an image-based person detection and tracking and a
trajectory generation for following movements.

Fig. 3.

Prototype robot

A. Person detection and tracking
We compared three image-based person detection methods. The first one is YOLO, which is an deep neural networkbased person detection by Redmon and Farhdi [11]. The

TABLE IC OMPARISON RESULT OF PERSON DETECTION METHOD .
MethodPrecision rate
Recall rate

YOLO
0.938
0.875

Depth image
0.823
0.438

Viola & Jones
0.190
0.492

second one is a depth camera-based person detection by
Munaro and Menegatti [12]. The third one is to apply ausual Haar-like feature-based method [13] to human upper
body images. Using image sequences of person following
in a greenhouse as inputs, we compared these methods interms of recall and precision of person detection. Examples
of detection results are shown in Fig. 4 and a performancecomparison is in Table I.
Based on the comparison results, we decided to use YOLO
for a high performance; especially in recall, it exhibits about
1.7 times higher recall rate than the others. We also analyzed
false positive cases by YOLO and found what detected are
person images reflected by glass walls of a greenhouse; suchfalse positives can easily deleted using depth information.
Person tracking uses the Kalman filter to estimate theposition of a person using the system equation of constantlinear motion based on the coordinate position of the detected
person. The position and velocity of a person are represented
by a robot coordinate system, and coordinates are converted
appropriately by using the amount of movement of the robot
obtained from odometry.
For a more reliable person tracking, we use Kalman filter
for person position estimation. We use a constant velocity
model as a state transition model. The position and thevelocity of a tracked person are represented in a robot
coordinate system, and a wheel odometry is used for acoordinate transformation due to the robot movement.
B. Motion planning for person following
In a usual person tracking scenario (e.g., [14]), a robot canmake a collision-free path to the detected person position.
Detecting aisles is, however, difficult in a greenhouse, as
mentioned above, because aisles are often occluded by grownplants. We therefore use the movement of a person as a guideto a safe path planning; that is, the robot tries to trace theperson’s trajectory. As a person moves by 0.5m or morefrom the last position, the robot adds it to the sequence of
person positions, which are then used as sub-goals of therobot movement. The robot repeatedly moves to the nearest
sub-goal to approximately trace the person trajectory.
The translational and the angular velocity at time t,
denoted as vtr and ωtr respectively, are determined by:⎧ r(dpt ≥ dth1 )⎨ vmaxp
dt −dth2 r
vtr =v(dth1 > dpt > dth2 ) , (1)⎩ dth1 −dth2 max
0(dth1 ≥ dpt )ωtrf= Kθ (θtsg − θtr ) + Kd ddif
,
t

(2)

where dpt is the distance between the target person and therobot, θtsg is the direction of the line connecting the last andFig. 5.

Calculation of the translational and the angular velocity.

the current sub-goal, θtr is the robot orientation with respectfto that line, ddif
is the distance to that line (see Fig. 5).
t
Equation (1) means that the robot moves at the maximum
speed if the distance to the target person is larger thandth1 , stops if the distance is less than dth2 , and moves
at an interpolated speed if the distance is in between thethresholds. The parameters are currently set as follows:r= 0.5m/s, dth1 = 1.5m, and dth2 = 0.5m.
vmax
Equation (2) means that the robot is controlled so that the
robot is aligned to the line connecting the last and the current
sub-goal and comes closer to that line. The gain parameters
are currently set as follows: Kθ = 0.5 and Kd = 0.25.
If the distance to the target person is small enough (lessthan 1.5m), however, the robot is controlled to directly movetowards the latest target person position.
V. 3D M APPING OF I NTERIOR OF G REENHOUSE
Mapping and localization (or SLAM) [15] are necessary
for a robot to autonomously move in the environment.
The system uses RTAB-Map [16] for 3D mapping of aninterior of a greenhouse. RTAB-Map extracts feature points
in the color images, calculates their 3D position using
the corresponding depth images, and estimates the camera
motion from a set of corresponding 3D points in consecutive
frames. Since the wheel odometry is not very accurate in theenvironments like greenhouses which could cause frequentslippage, we use this method which does not rely on any
odometry information. Note that the above feature matchingis also used for loop closure detection in RTAB-Map.
In outdoor environments, the appearance of a scenechanges with those of sunlight, and this may degrade image
feature extraction and matching, and a final mapping result.
We therefore conducted a preliminary mapping experimentin a greenhouse in the TUT campus. Fig. 6(a) shows theexperimental scene. The size of the greenhouse is 7m×10m.
We manually moved Kinect v2 and collected data over sevenminutes of movement. Fig. 6(b) shows the mapping result,which is reasonably consistent over the entire region. This
result confirms the use of the combination of Kinect v2 and
RTAB-Map.

(a) YOLO [11].

(b) Depth camera-based [12].

Fig. 4.

(c) Viola& Jones [13].

Comparison of person detection methods.

(a) Experimental scene.

Fig. 7.

Experimental scene in a greenhouse.

Fig. 8.

Mapping result using RTAB-Map.

(b) Mapping result.

Fig. 6.

A preliminary mapping experiment in a TUT greenhouse.

Exploration and mapping is one of the approaches toautonomously generating a map of an unknown environment(e.g., [17]). However, this may not work well if detection of
traversable regions is difficult in a greenhouse due to heavilygrown plants. One practical approach is to guide a robot
throughout the greenhouse aisles using the person following
capability. In that case, we need to eliminate depth data of
the person so that those data are not included in the final
map.
VI. E XPERIMENT
We carried out person following and mapping experiments
at a rose cultivating site in Aichi Agricultural ResearchCenter. Fig. 7 shows a snapshot of the person following experiment in an aisle of the greenhouse. We got 1,800 framesof data in total and generated a map by using RTAB-Map, asshown in Fig. 8. We also took another approach to making a

map using 3D LIDAR data for a comparison purpose. Thismethod uses an NDT-based ego-motion estimation [18] fora 3D LIDAR-based mapping, and Fig. 9 shows the mapping
result. The colors in the map indicate the height of the 3D
points. The areas to which the maps are generated by bothmethods are rather different due to different fields of viewof the sensors. For the commonly mapped area, they are
sufficiently consistent with each other. Therefore, an RGBD camera is a feasible sensor for mapping a greenhouse with

(a) top view.

Fig. 9.

Mapping result using 3D LIDAR.

(b) side view.

Fig. 11. Comparison of the maps generated using new (in green) and old(in red) data.

Fig. 10. The person and the robot trajectory during a person following
experiment inside an aisle.

grown plants.
Fig. 10 shows the results of person following in an aisle
where the robot took a set of data used for making the
map in Figs. 8 and 9. The green and the red line indicate
the person and the robot trajectory, respectively, during theperson following (and mapping) experiment. Most of timethe distance between the robot and the person is smallenough, the robot almost always moved toward the current
person position (see Sec. IV-B). The robot successfully
tracked the person.
We then compared the map with the one which is generated from a set of data taken about two months before. One
of challenging characteristics of such a natural scene is thatplans grow and their shapes change sometimes drastically.
Fig. 11 shows the comparison results. The two independently
generated maps are aligned using NDT with a manuallygiven initial alignment. Thanks to large structural objects
such as the walls on the right, two maps are well aligned.
We can observe that the greenhouse structure is consideredsufficient to localize the robot even if the date of mapping andthat of localization are largely different. At the same time, we
can see the growth of plants clearly due to a nice alignment.
This would be beneficial to a precise plant matching overtimes for watching a growing process for timely harvesting.
VII. C ONCLUSIONS AND F UTURE W ORKThis paper investigated necessary technologies for mobile
robots that support harvesting in greenhouse horticulture.
Based on the investigation results, we developed a prototype
mobile robot with person following and mapping capabilities.
We compared three person detection methods in an actual
greenhouse scene, and selected the YOLO detector as mostsuitable one. We also developed a path planning method that
controls the robot so that it follows the trajectory of thedetected target person, and that can therefore be applied toaisles with heavily-grown plants. Concerning the mapping,
we evaluated RTAB-Map, a SLAM method which uses an
RGB-D camera and has been shown to be effective invarious environments, also in the same greenhouse scene andconfirmed that it can generate a map which is comparable
to the one generated by a more reliable 3D LIDAR. We also
compared two maps generated with a large interval to show
structural objects are effective for localization under a large

change of growing plants.
For increasing the efficiency of harvesting work, it isdesirable that a robot can autonomously move around ina greenhouse for carrying harvested plants or other items.
For this purpose, we are now developing a system which
combines the local navigation capability with a global route
planning and realize from-to operations. It is also necessary
to develop a reliable localization and traversable region
detection using RGB-D data for a robust local navigation.
ACKNOWLEDGMENT
The authors would like to thank Mr. Kiyoaki Kakihara(KER Co. Ltd.) for constructing a mobile robot base and Mr.
Mitsuo Tsume (Sinfonia Technology Co. Ltd.) for fruitful
discussions on flower harvest support. They would also liketo thank members of Aichi Agricultural Research Center forsupporting them for conducting experiments.
This work is supported by Knowledge Hub Aichi’s Priority
Research Project (2nd term) “Next Generation Robot Social
Development Technology Development Project”.
R EFERENCES[1] Yano Research Institute Ltd., “Current status and future prospects ofsmart agriculture in 2016”, 2016 (in Japanese).
[2] N. Noguchi , “Agricultural ICT and Vehicle Robot based on Spatial
Information”, J. of the Robotics Society of Japan, Vol. 35, No. 5,
pp. 372–375, 2017 (in Japanese).
[3] Y. Takahashi, Y. Kakigi, H. Kumagai, Y. Abe, H. Seki, T. Fukuda, M.
Kashiwazaki, and K. Ozaki, “Prototype of contactless quality inspection
device of strawberry using near infrared region and CIE 1931 XYZcolor space camera”, The 34th Annu. Conf. of the Robotics Society ofJapan, 2016 (in Japanese).
[4] T. Matsumoto, G. Nakamura, K. Tomoda, K. Nakao, H. Miki, H.
Hamamatsu, and T. Matsuo, “Verification of end effector and image
processing system of tomato harvesting robot”, 2017 JSME Conferenceon Robotics and Mechatronics, 2017 (in Japanese).
[5] K. Kapach, E. Barnea, R. Mairon, Y. Edan, and O. Ben-Shahar.
Computer vision for fruit harvesting robots – state of the art andchallenges ahead. Int. J. of Computational Vision and Robotics, Vol. 3,
No. 1/2, pp. 4–34, 2012.
[6] S. Jay, G. Rabatel, and N. Gorretta. In-field crop row stereoreconstruction for plant phenotyping. In Proceedings of 2nd Int. Conf.
on Robotics and Associated High-Technologies and Equipment for
Agriculture and Forestry (RHEA-2014), pp. 339–348, 2014.
[7] J. Romeo, A. Benı́tez, J.M. Guerrero, M. Montalvo, and G. Pajares.
Tractor guidance in maize fields using computer vision and geometry.
In Proceedings of 2nd Int. Conf. on Robotics and Associated HighTechnologies and Equipment for Agriculture and Forestry (RHEA2014), pp. 257–265, 2014.
[8] T.A. Troyer, S. Pilta, and E. Nutter. Inter-row robot navigation using
1d ranging sensors. IFAC-PapersOnLine, Vol. 49, No. 16, pp. 463–468,
2016.
[9] Z. Zainudin, S. Kodagoda, and G. Dissanayake. Torso Detection
and Tracking using a 2D Laser Range Finder. In Proceedings ofAustralasian Conf. on Robotics and Automation 2010, 2010.
[10] K. Kidono, T. Miyasaka, A. Watanabe, T. Naito, and J. Miura.
Pedestrian Recognition Using High-Definition LIDAR. In Proceedingsof 2011 IEEE Intelligent Vehicles Symp., pp. 405–410, 2011.
[11] J. Redmon and A. Farhadi. “YOLO9000: Better, Faster, Stronger”,
arXiv preprint arXiv:1612.08242, 2016.
[12] M. Munaro and E. Menegatti, “Fast RGB-D people tracking for service
robots”, Autonomous Robots, Vol. 37 No. 3, pp. 227–242, 2014.
[13] P. Viola and M. Jones, “Rapid object detection using a boosted cascadeof simple features”, Proceedings of 2001 IEEE Conf. on Computer
Vision and Pattern Recognition, Vol. 1, pp. 511–518, 2001.
[14] J. Miura, J. Satake, M. Chiba, Y. Ishikawa, K. Kitajima, and H. Masuzawa, “Development of a Person Following Robot and Its Experimental Evaluation”, Proceedings of the 11th Int. Conf. on Intelligent
Autonomous Systems, pp. 89–98, 2010.

[15] S. Thrun, W. Burgard, and D. Fox. Probabilistic Robotics. The MIT
Press, 2005.
[16] M. Labbe and F. Michaud, “Online Global Loop Closure Detection forLarge-Scale Multi-Session Graph-Based SLAM”, Proceedings of 2014
IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pp. 2661–2666,
2014.
[17] Y. Okada and J. Miura, “Exploration and Observation Planning for
3D Indoor Mapping”, Proceedings of 2015 IEEE/SICE Int. Symp. on
System Integration, pp. 599–604, 2015.
[18] M. Magnusson, A. Lilienthal, and T. Duckett, “Scan registration forautonomous mining vehicles using 3D-NDT”, J. of Field Robotics,
Vol. 24, No. 10, pp. 803–827, 2007.