Internationalof Computational
Intelligence
Systems
Journal ofJournal
Robotics,
Networking andArtificial Life
Vol.
XX(Z);
Month(2019),(Year), pp.
pp. 138–142
xx–yy
Vol.
6(2);
September
DOI: 10.1080/XXXXXXXXXXXXXX; ISSN
online
DOI: https://doi.org/10.2991/jrnal.k.190829.002;
ISSNXXXX–XXXX
2405-9021; eISSN
2352-6386
https://www.atlantis-press.com/journals/ijndc
https://www.atlantis-press.com/journals/jrnal

Research Article

Toward Smart Tomato Greenhouse: The Fourth Tomato
Harvesting Robot Competition
Takayuki Matsuo1,*, Takashi Sonoda2, Yasunori Takemura2, Masanori Sato3, Kazuo Ishii4
Department of Creative Engineering, National Institute of Technology, Kitakyushu College, 2-20-1 Shii, Kokuraminami, Kitakyushu, Fukuoka 802-0985, Japan
Department of Integrated System Engineering, Nishi-Nippon Institute of Technology, 1-11 Aratsu, Kanda, Miyako, Fukuoka 800-0394, Japan3
Department of Applied Information Technology, Nagasaki Institute of Applied Science, 536 Aba, Nagasaki, Nagasaki 851-0193, Japan
4Graduate School of Life Science and System Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
12ARTICLE INFO
Article History
Received 08 November 2018
Accepted 31 December 2018
Keywords
Smart greenhouse
agriculture robot
tomato harvesting robot

ABSTRACTAgriculture, one of the most important industries for human life, is faced with serious problems, the shortage of workers, thefalling birthrate and the aging population, global warming and natural disasters, etc. Tomato is one of the most important
fruit vegetables, and most tomatoes in Japan are cultivated in the greenhouses, or large scale farms as the solution for effective
production toward smart greenhouses, whereas the high temperature and high humidity, construction and management costs
are big problems and technologies such as environmental control system, factory automation technology, AI, robotics arerequired. We had held Tomato Harvesting Robot competition from 2014 toward realization of smart tomato greenhouse aiming
at promote the automated tomato harvesting to reduce the working time of harvesting. In this paper, we report on the tomato
harvesting robot competition and the results mainly from the fourth competition.
© 2019 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
According to the statistical information of the Ministry ofAgriculture, Forestry and Fisheries in Japan, Japan’s food self-­
sufficiency rate (calorie base) is about 40%, which is the lowest
level among major developed countries. In the agricultural field,
the aging and depopulation of producers is progressing, and labor
shortage due to lack of successors are serious problems. “Smartagriculture” is proposed as a future agriculture production thatsolves these problems. One of the key technologies for smart greenhouse is robotics, therefore, agricultural robots are expected to beintroduced to the agricultural field as the solution tools.
Tomato is one of the important fruit vegetables and most tomatoes are produced in the greenhouses, or large-scale farms, where
the high temperature and humidity, and long harvest age force thefarmer heavy works. To develop the tomato harvesting robot, many
research issues exist such as manipulator design, end-­effector
design, collaborative behavior, artificial intelligence, motor control,
image processing, target recognition and so on. Kawamura et al. havedeveloped a tomato harvesting robot equipped with a 5-degree-offreedom (DOF) manipulator on a mobile platform. It shows that thetarget can be grasped with sufficient accuracy for the commanded
position [1]. Tomato recognition is based on the luminance signal
and red signal from the camera, and the three-­dimensional position
estimation of the tomato fruit by stereo vision [2,3]. It points outthat the recognition rate declines due to changes in the lighting enviCorresponding author. Email: matsuo@kct.ac.jp*ronment and that it is difficult to recognize stems. As the next step,speeding up tomato harvesting is desired, and research on feed-forward control has been developing. Kondo [4] experimentally investigated the natural frequency of tomato tufts to introduce a model oftomato tufts, and introduced them into manipulator trajectory planning. The harvesting mechanism (end-effector) for harvesting tomatoes is also an important research subject [4,5], and methods suchas cutting from stems, breaking, and suction are proposed. Taking
advantage of the recent development of computers, real-time image
processing technology has also been developed. Tomato fruit gripping experiments are being carried out in an environment where thedirection of easy gripping and sunshine conditions change [6]. Xuet al. developed a methodology for automated strawberry grading
system based on image processing. Al-Hiary et al. [7] proposed andexperimentally evaluate a software solution for automatic detection
and classification of plant leaf diseases. Research is actively progressing in recent years. Wang et al. [8] developed a tomato harvesting robot used in greenhouse which consists of a four-wheelindependent steering system, a 5-DOF harvesting system, a navigation system and a binocular stereo vision system steering system.
Due to the rapid development of image processing technology inrecent years, agricultural robots using Deep Learning are attracting
attention. Wang Xinshao and Cheng [9] proposed weed seeds classification based on Principal Component Analysis (PCA) Network
Deep learning and the PCA Network variant method obtains goodclassification result and improves the recognition accuracy. And,Jeon et al. [10] developed plant lead recognition system using aConvolution Neural Network (CNN) and proposed a method toclassify leaves using the CNN model.

T. Matsuo et al. / Journal of Robotics, Networking and Artificial Life 6(2) 138–142

We have also organized Tomato Harvesting Robot competition
from 2014 aiming at smart tomato greenhouse as the tomato harvesting time engages in the whole tomato production [11]. In thispaper, we will report on the result of the fourth tomato harvesting
robot competition.

2. COMPETITION REGULATIONS
The tomato harvesting robot competition consists of two leagues
which are the Senior League and the Junior League. The target
competitors for Senior League are supposed to the team with automated and remote-controlled robots, and the Junior League are forhigh school or junior high school students who build robot using
LEGO Mindstorm. In this section, competition regulations of bothreargues will be introduced.

139(a)(b)(c)

2.1. Senior LeagueThe Senior League supposes that teams are not restricted to university nor company, just each team should have own tomato
harvesting robot(s). Two kind of competition field are prepared,
one is the rail-style area (Figure 1a) and the other is the free-style(Figure 1b). The rail-style area is designed to have the similar environment with the large-scale tomato factory with heat pipes forwarming greenhouse. Free-style area is for the robots of open-field
culture environment. Tomatoes are hanged on height from 800 to
1200 mm referring to the tomatoes arrangement of Hibikinada
Saien Co., Ltd.
The required specification of tomato harvesting robot is shown in
Table 1. Regarding the sizes of robot, the projected area of the robot
to the ground is within the 800 × 800 mm2 and no height limitation.
For safety reason, the robots should have an emergency stop switch
on the easy-to-find position. As the recommendations, the weight of
the robot is <50 kg and the electric power of motor is <70 W/each.
The first stage is intended as the inspection of basic functions needed
for tomato harvesting, so that single tomatoes are suspended asshown in Figure 2a. The team succeeded to touch the tomato, moves
to the second stage. In second stage, some sets of bunches of tomatoes are suspended as shown in Figure 2b. The five high score teams
go up to the final stage. The score is calculated based on the numberof successfully harvested tomatoes and damages to tomatoes. Also,
(a)

(b)

Figure 2 | The missions and stages of the competition. (a) First stage: pickup single tomato. (b) Second stage: pick up tomatoes from bunches.
(c) Final stage: pick up tomatoes from plants.

the unripe tomatoes are counted as the damaged tomatoes. Finalscore of second stage is the score multiplied the basic score and thecoefficient decided by a combination of choice between the control
method which are remotely and autonomous control and the areas
which are the rail area and the free style area. In final stage, the robots
harvest tomato from plant body as shown in Figure 2c.
The robots are classified mainly into two types, manual control andautonomous control, and the former robots are classified by whether
the operator observes tomato directly or indirectly using cameras
mounted on robots, and by robot locomotion whether the robot uses
rail or not. Totally, the robots are categorized into six types depending on operation and locomotion method as shown in Table 2.
The success points of one-tomato-harvesting change depending on
the robot category, e.g., the point of one-tomato-harvesting for T-1 is
2 × 1 = 2 points, and that of category T-6 is 2 × 8 = 16 points.

2.2. Junior League
In Junior League, the subject is to carry small size tomatoes toassigned positions. Students should develop the robot using LEGO
Mindstorm with the functions such as line trace, color recognition,
end-effector with mechanism design and motors with control, andtheir programming [11].

Figure 1 | Competition fields. (a) Rail-style area. (b) Free-style area.
Table 1 | Specifications of a robot
Size
Equipment
WeightMotor powerW: 800 mm, D: 800 mm (H: No limitation)
Emergency stop switch
Max. 50 kg (recommend)
Max. 70 W (recommend)

In Line Trace Challenge, robots should detect white line in the competition area and move along the line using a color sensor. On theway of the course, the tomato-box-harvesting field (harvest field)
exists, where tomato boxes are arranged. The robot must move to theharvest field in order to get the tomato boxes. In Color Identifying
Challenge, robots should explore and recognize color signs of thesame color of three boxed tomatoes (tomato box) on the course.
As guidance to harvest field, red, yellow and blue lines are drawn

T. Matsuo et al. / Journal of Robotics, Networking and Artificial Life 6(2) 138–142

140

Table 2 | Categories of tomato robots
Robot control

Manual (x1)

Autonomous (x8)Operator manipulation

Direct observation of tomato (x1)

Indirect observation of tomato (x2)Mode of locomotion
Total ratio for one tomato
Category

Rail-type (x1)x1
T-1

Rail-type (x1)x2
T-3

Free-type (x2)x2
T-2

Start and stop commands

Free-type (x2)
x4T-4Rail-type (x1)
x8T-5

Free-type (x2)x16
T-6

Table 4 | Final result of the Senior LeagueRanking
Overall winner
Overall runner-up

Team

Score

HAYASHI-LAB
Hibikino-Tom’s R

120
68

Table 5 | Result of Junior League

Figure 3 | Overview of Junior League.

RankingTeam

1st place
2nd place

Ohnishi lions (Fukuoka Technical High School)
Ooita kogyo C (Oita Technical High School)
NiAScience (High school attached Nagasaki Institute ofApplied Science)

3rd place

Table 3 | The list of finalists
TeamAffiliation

Category

HAYASHI-LAB
Kyushu Institute of Technology
Hibikino-Tom’s R
Kyushu Institute of Technology
Nagasaki GANBARANBA Nagasaki Institute of Applied Science
KPC2017A
Kyushu Polytechnic college
KPC2017B
Kyushu Polytechnic college

T-5
T-5
T-4
T-5
T-5

in the field. The robots should detect guidance line by color sensorand carry the tomato boxes to the same color positions.
In Mechanism Design and Control Challenge, robot should manipulate a tomato box or boxes using a manipulator and carry to thestorage location. To pick up the tomato box, a manipulator equipment should be made by each team. The robot is required to store,transport and relocation depending on tomato box colors. After
picking up the box, the robot should return to the course. In ObjectDetection Challenge, robots should detect a battery charging station and stop there.
The basic specification for robot is that the size of the robot is
within 300 × 300 mm2 on ground. Figure 3 shows overview ofJunior League.

3. SUMMARY OF THE FOURTH TOMATOHARVESTING ROBOT COMPETITION
Five finalists are: Hibikino Tom’s R (Category T-5, Kyushu Instituteof Technology), Nagasaki GANBARANBA (Category T-4,Nagasaki Institute of Applied Science), KPC2017A, KPC2017B(Category T-5, Kyushu Polytechnic college) and HAYASHI-LAB(Category T-5, Kyushu Institute of Technology) as shown in
Table 3. Overall winner and Overall runner-up are shown in
Table 4. In Junior League, number of the participated teams were18 teams and the result is Table 5.

Table 6 | Results of second stage (Score 1) and score after applying
Equation (2) (Score 2)
TeamA
BC
DEFG

CategoryC`afcg

Score 1

Score 2

T-5
T-5
T-4
T-5
T-5
T-3
T-1

88
4
882
1

744111
12
32
10
10

91
10
0
0
00
0000
0
0

0.50
0.39
0.57
0.50
1.00
0.50
1.00

110
86
38
24
16
6
2

31.78
42.00
20.86
12.00
16.00
3.00
2.00

4. DISCUSSION
The score evaluation of the senior league of the fourth tomato harvesting robot competition are shown in Equation (1)		 P = C(2a + b ) - 2(g + d ) - e (1)where P is the final score, C: the category coefficient in Table 2, a :the number of obtained tomatoes without damage, b : the numberof damaged tomatoes, g  : the number of dropped tomatoes, d  : thenumber of damaged tomatoes, e : deducted point when robots
damage to tomato plants (the final stage). Table 6 shows the resultof the second stage. Score 1 is final score using Equation (1). Team A
was the first place in the second stage, which could be 110 score.
However, Team A dropped nine tomatoes, the worst number in theseven teams of Table 6. In the case of using Equation (1), Total score
P is almost not influenced from g, d and e at selecting high classsuch as T-5 and T-6. This means that the positive harvesting makes
bigger score than the risk of dropping. Therefore, some teams
dropped many tomatoes intentionally for the purpose to avoid
difficult tomatoes. That is, the tomatoes are not handled carefully.
In actual tomato factory, damaged tomatoes and drop tomatoes
have no commercial value. Therefore, we considered that concept

T. Matsuo et al. / Journal of Robotics, Networking and Artificial Life 6(2) 138–142

of carefully dealing with tomatoes is employed. We proposed newequation as shown in Equation (2).		 P = gC (2` + a ) - 2(f + c ) - d (2)		 g =`(3)` + a +f +cEquation (2) used “harvest success rate η” expressed by Equation (2)and if robot damage and drop tomatoes magnification C isdecreased. Score 2 of Table 6 shows the result of changing scores
after applying Equation (2). Number of dropped tomatoes influences the score and Team A becomes the second place.

5. CONCLUSION
In this paper, tomato harvesting robot competition is introduced
toward Smart Tomato Greenhouse. The tomato harvesting robot
competition has two leagues which are Senior and Junior League.
In Junior League, number of the participated teams were 18 teams.
In Senior League, number of participated teams were nine teams. In
next competition, concept of carefully dealing with tomatoes will
be used and we proposed harvest rate η and if robot damage anddrop tomatoes, the score of team will be decreased.

ACKNOWLEDGMENTS
This works were supported by the Center for Socio-Robotics
Synthesis, Kyushu Institute of Technology, The University of
Kitakyushu, Nippon Bunri University, Nishinippon Institute ofTechnology [This support included The Ministry of Education,Culture, Sports, Science and Technology (MEXT), Center ofCommunity.], Kitakyushu College of Technology, HibikinadaSaien, City of Kitakyushu, The Kitakyushu Chamber of Commerceand Industry, Institute of Systems, Information Technologiesand Nanotechnologies (ISIT), Japan Chapter of IEEE Society onSystems, Man, and Cybernetics, The Japan Society of MechanicalEngineers (JSME), The Robotics Society of Japan (RSJ), Kitakyushu
Foundation for the Advancement of Industry Science and141

Technology (FAIS), IT and Mechatronics Chapter and Kyushu
Chapter of Japanese Society of Agricultural Machinery and FoodEngineers, The MathWorks Co. Ltd., and Ministry of Agriculture,Forestry and Fisheries (MAFF).

REFERENCES[1] Kawamura et al., Study on agricultural robot (1st report), J. JASM
46 (1984), 353–358 (in Japanese).
[2] Kawamura et al., Study on agricultural robot (2nd report), J.
JASM 47 (1985), 177–182 (in Japanese).
[3] Kawamura et al., Fruit harvesting robot, J. JASM 47 (1985), 237–241 (in Japanese).
[4] N. Kondo, K. Yamamoto, H. Shimizu, K. Yata, M. Kurita, T. Shiigi,
et al., A machine vision system for tomato cluster harvesting
robot, Eng. Agric. Environ. Food 2 (2009), 60–65.
[5] N. Kondo, K. Yata, M. Lida, T. Shiigi, M. Monta, M. Kurita, et al.,Development of an end-effector for a tomato cluster harvesting
robot, Eng. Agric. Environ. Food 3 (2010), 20–24.
[6] T. Ota, T. Yamashita, S. Hayashi, T. Komeda, Development ofa tomato harvesting robot with a vision system using specularreflection (part 1), J. JSAM 72 (2010), 587–594.
[7] H. Al-Hiary, S. Bani-Ahmad, M. Reyalat, M. Braik, Z. Alrahamneh,
Fast and accurate detection and classification of plant diseases,
Int. J. Comput. Appl. 17 (2011), 31–38.
[8] L.L. Wang, B. Zhao, J.W. Fan, X.A. Hu, S. Wei, YS. Li, et al.,Development of a tomato harvesting robot used in greenhouse,
Int. J. Agric. Biol. Eng. 10 (2017), 140–149.
[9] W. Xinshao, C. Cheng, Weed seeds classification based on PCANet
deep learning baseline, 2015 Asia-Pacific Signal and Information
Processing Association Annual Summit and Conference (APSIPA),
IEEE, Hong Kong, China, 2015, pp. 408–415.
[10] W.-S. Jeon, S.-Y. Rhee, Plant leaf recognition using a convolution neural network, Int. J. Fuzzy Log. Intell. Syst. 17 (2017),
26–34.
[11] K. Ishii, Y. Takemura, T. Matsuo, T. Sonoda, Tomato harvesting robot competition, 2016 Joint 8th International Conference
on Soft Computing and Intelligent Systems (SCIS) and 17thInternational Symposium on Advanced Intelligent Systems(ISIS), IEEE, Sapporo, Japan, 2016, pp. 537–542.

Authors Introduction
Dr. Takayuki Matsuo

Dr. Takashi Sonoda
He is an Associate Professor at
Department Creative Engineering,
National Institute of Technology(KOSEN), Kitakyushu College,
Japan. His research area is underwater robots and biomimetic robots.

He is an Associate Professor at
Department Integrated SystemEngineering, Nishinippon Instituteof Technology, Fukuoka, Japan. His
research are underwater robotics androbot manipulator systems.

142

T. Matsuo et al. / Journal of Robotics, Networking and Artificial Life 6(2) 138–142

Dr. Yasunori Takemura

Prof. Kazuo Ishii
He is an Associate Professor atDepartment Integrated System
Engineering, Nishi-Nippon Instituteof Technology, Fukuoka, Japan. His
research area is about machine learning, data mining and Robotics.

Dr. Masanori Sato
He is an Associate Professor in theFaculty of Applied Information
Technology in Nagasaki Institute ofApplied Science, Japan. His maininterests are agricultural robots,
underwater robots, and life supportrobots.

He is a Professor at GraduateSchool of Life Science and System
Engineering, Kyushu Institute ofTechnology, Japan. His research area
is about field robots and intelligent
robot systems.