https://doi.org/10.20965/jrm.2020.p1279
Tomato Growth State Map for Monitoring and Harvesting

Paper:Tomato Growth State Map for
the Automation of Monitoring and Harvesting
Takuya Fujinaga, Shinsuke Yasukawa, and Kazuo Ishii
Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
E-mail: fujinaga.takuya835@mail.kyutech.jp[Received June 14, 2019; accepted October 12, 2020]

In agriculture, a decrease in the number of farmers, anaging population, and a shortage of successors are significant problems. In response to such urgent problems, smart
agriculture is expected to be an innovative method of utilizing robot technology and information communication
technology. Some studies aiming to realize smart agriculture include the autonomous operation of agricultural
machinery, automation of harvesting, and monitoring of
the field environment and crops. For the autonomous
operation of agricultural machinery, Noguchi proposed arobot farming system that utilizes intelligent robot vehicles to automate farming activities from planting to supplying products to consumers [1]. In addition to intel-

ligent robot vehicles, they described the importance of
simultaneous operations by developing a multiple robot
system, safety when using robots, and management systems. For the automation of harvesting, Kondo et al. developed an end-effector that could harvest a tomato cluster [2]. In the results of harvesting experiments on ahigh-density plant training system, the harvesting time
per cluster was 15 s, and the success rate of harvesting
was 50% (10 tomato clusters out of 20). Yaguchi et al.
developed an end-effector that grasped a fruit using grippers and plucked it from the separation layer in the peduncle [3]. They described the results of harvesting experiments conducted on an actual farm, demonstrating thatthe harvesting time was shortened by improving the harvesting motion. Yoshida et al. developed a robot that harvested a cluster of cherry tomatoes [4]. They focusedon detecting the peduncles that the robot required whenharvesting the clusters. The success rate of harvesting inexperiments at an actual farm was 95% with seven clusters (19 tomato fruits out of 20 in seven tomato clusters).
For the monitoring of the field environment and crops,
Fukatsu et al. described a remote monitoring method that
responded flexibly and dynamically to changes in the fieldenvironment for long-term field monitoring [5]. They established field servers, which are small monitoring sensor nodes, in some countries, and their effectiveness wasreported. Fukui et al. developed a robot that estimatedthe volume of tomato fruits to create a database of fruit
growth [6]. The robot detected fruits using saliency-based
image processing and estimated the volume of the fruits.
Furthermore, they designed multiple indicators to evaluate the precision of the estimation results. If the evaluation
results are unreliable, the position for re-measurement iscalculated. Studies aimed at realizing smart agriculture
are diversifying, and various approaches using robot technology and information communication technology havebeen proposed.
Meanwhile, with the aging of agricultural workers,
quantifying the expertise of experienced farmers is asignificant concern. Choi et al. proposed a method ofquantifying the maturity stages using image analysis [7].
They followed the classification standard of the maturity stages based on the color of the tomato fruit surface as defined by the United States Department of Agriculture (USDA). Comparing the results of the proposed

Journal of Robotics and Mechatronics Vol.32 No.6, 2020

1279

To realize smart agriculture, we engaged in its systematization, from monitoring to harvesting tomato fruits
using robots. In this paper, we explain a method ofgenerating a map of the tomato growth states to monitor the various stages of tomato fruits and decide a
harvesting strategy for the robots. The tomato growthstate map visualizes the relationship between the maturity stage, harvest time, and yield. We propose ageneration method of the tomato growth state map, arecognition method of tomato fruits, and an estimation
method of the growth states (maturity stages and harvest times). For tomato fruit recognition, we demonstrate that a simple machine learning method using alimited learning dataset and the optical properties oftomato fruits on infrared images exceeds more complex convolutional neural network, although the results depend on how the training dataset is created.
For the estimation of the growth states, we conducteda survey of experienced farmers to quantify the maturity stages into six classifications and harvest times
into three terms. The growth states were estimatedbased on the survey results. To verify the tomato
growth state map, we conducted experiments in an actual tomato greenhouse and herein report the results.

Keywords: smart agriculture, agricultural robot, tomato
growth state map, recognition, estimation

1. Introduction© Fuji Technology Press Ltd. Creative Commons CC BY-ND: This is an Open Access article distributed under the terms of
the Creative Commons Attribution-NoDerivatives 4.0 International License (http://creativecommons.org/licenses/by-nd/4.0/).

Fujinaga, T., Yasukawa, S., and Ishii, K.

method and manual grading, the rate correctly classified
was 77.5% (93 tomato fruits out of 120). The USDA provides standards for both the maturity stages and size andquality tolerance as a guide to support farmers in improving the quality and marketability of their crops. Quantifying the maturity stages in these standards is synonymous with quantifying one of the farmers’ main standards. Kusui et al. proposed the Agri-Info Science platform as a concrete system to address the necessity forskill transfer in response to various food problems associated with global population growth [8]. While manyof the deep learning methods for harvest decisions focus only on images, experienced farmers have multiple
criteria such as pruning and fruit thinning. The authors
focused on assessments based on these criteria and developed an e-learning-based support system that enables
inexperienced farmers to learn such advanced craftsmanship. Its effectiveness was demonstrated in an evaluation experiment of the learning support system for kiwi
pruning operations. Wakamori et al. investigated the relationship between daily leaf wilting and stem diameter variation to estimate water stress on plants in water
stress cultivation [9]. They also indicated that deep learning estimation methods are inadequate. Instead of adopting such black-box approaches, cross-correlation analysis
was used to analyze the time lag correlation between leaf
wilt quantified by optical flow and change in stem diameter as a water stress index. They demonstrated that theanalysis results were consistent with known plant watertransport mechanisms. Thus, to quantify the expertise ofexperienced farmers, conducting surveys of their insights
and methods and experiments in an actual environment isimportant.
Our studies on the realization of smart agriculture werebased on listening to the insights of experienced farmers in cooperation with the Hibikinada Green Farm Co.,
Ltd. (hereinafter, Hibikinada Green Farm), which implemented a long-term multi-stage cultivation of tomato
plants by incorporating the cultivation technology developed in the Netherlands [10, 11]. We aimed to realize asystem that uses robots to automate operations from themonitoring of tomato plants to harvest their fruits. Fig. 1
shows an outline of this system. The robots move on a rail
installed in a tomato greenhouse. The monitoring robot
acquires the images of the cultivation area, and these data
are sent to a database in cloud computing. The states of
the tomato plants are estimated using these data. Subsequently, the harvesting robot harvests only the mature
fruits. Thus, this system enables more efficient agriculture.
In this study, we aimed to generate a map of the tomato
growth states that visualizes the relationship between thematurity stages, harvest times, and yield. Experienced
farmers would designate the maturity stage of the tomato
fruits to be harvested, and the yield would be predictedusing this map. Furthermore, the harvesting robot can determine a harvesting strategy based on this map.
In this paper, to generate the tomato growth state map,
we have described the generation method of the tomato
1280

Fig. 1. Automation of monitoring the tomato plants andharvesting the tomato fruits.

growth state map, the recognition method of the tomato
fruits, and the estimation method of the growth states(maturity stages and harvest times). The recognition rate
(Accuracy) and detection rates (Precision, Recall, and F1score) of the classifiers generated using different machinelearning methods were compared. We focused on thecharacteristics of tomato fruits using infrared imaging to
generate the classifiers. For the estimation method, we
consulted experienced farmers of the Hibikinada Green
Farm to decide the numbers of classifications for the maturity stages and terms for the harvest times. Subsequently, the growth states were estimated based on the
results of the survey conducted to quantify their expertise.
The proposed method is described in Section 2, the verification results of the tomato growth state map are detailedin Section 3. The findings are discussed in Section 4, andthe conclusions are stated in Section 5.

2. Generation of the Tomato Growth State
Map
2.1. Outline of the Generation Method
Figure 2 shows a flowchart for the generation of thetomato growth state map. The images (RGB, depth, andinfrared) acquired by the robot were used as input (process (i), Fig. 2). First, pre-processing was conducted using depth imaging (process (ii), Fig. 2). Images focusing
only on tomato plants in the target cultivation area weregenerated by removing the rear-row cultivation area andbackground on the RGB and infrared images. Next, amosaic image composed of these images was generated(process (iii), Fig. 2). Tomato fruits on the mosaic image were detected (process (iv), Fig. 2), and the growthstates (maturity stages and harvest times) of the detected
fruits were estimated (process (v), Fig. 2). In addition,
the position of the detected tomato fruits in the cultivation
area was calculated using the coordinates in the mosaic
image and depth data. Subsequently, the tomato growthstate map was generated by adding the information of thetomato fruits to the mosaic image (process (vi), Fig. 2).
In Ref. [10], we proposed a method of generating a mosaic image, which is a single image generated by overlaying images with overlapping parts based on the results
Journal of Robotics and Mechatronics Vol.32 No.6, 2020

Tomato Growth State Map for Monitoring and Harvesting

Table 1. Comparison of previous studies.

Fig. 2. Flowchart for the generation of the tomato growthstate map.

of feature point matching. As a generation condition,
the feature points extracted in the images must be distinguishable. However, objects (fruits, stems, leaves, etc.) in
the tomato greenhouse are crowded together so that thematching rate is very low owing to similarities in the features. Therefore, generating a mosaic image with a simple matching method using RGB images is difficult. The
proposed method improved the matching rate by limiting
the depth direction using infrared imaging and limiting
the search area of the feature points using the moving distance of the robot. Therefore, a mosaic image focusing on
the target cultivation area was generated. In this paper, we
present the steps to generate the tomato growth state map
from processes (iv)–(vi) (Fig. 2).
This section describes the methods of recognizing
tomato fruits and estimating their growth states. Previous studies on the recognition or detection of tomato
fruits proposed methods focusing on the color, shape, andtemperature of tomato fruits and methods using machinelearning. Teimourlou et al. proposed a method of recognizing mature fruits using three color models (RGB, HIS,
and YIQ) based on 200 RGB images acquired in a tomato
greenhouse [12]. This method consists of two steps: removing the background of the image and extracting themature fruits. The algorithm could extract 92%–96% of
mature fruits. Hatou et al. proposed a method of recognizing mature and immature fruits using thermal imaging [13]. To recognize the fruits in the cluster individually, they applied a method based on wire size reduction
and its effectiveness was demonstrated. Wang et al. proposed a method of recognizing mature fruits using a stereo
vision system that employed the Otsu algorithm and elliptic template method [14]. The success rate of recognizing
mature fruits was 99.3%, and the position of the fruits
was calculated using feature point matching. Fujiura et al.
proposed a method for recognizing mature and immature
fruits using a dichromatic 3D vision sensor [15]. Thissensor projected light of different wavelength bands andJournal of Robotics and Mechatronics Vol.32 No.6, 2020

First author

Detection(Rate [%])

Teimourlou
[12]
Hatou
[13]
Wang
[14]
Fujiura[15]
Yamamoto[16](92–96)

(–)

(99)

(–)

(88)

Maturity Harvest
stage
time

Fruit
position

1

–

–

2

–

–

1

–



2

–



2

–

–

the maturity stage was estimated using the difference inreflectance at each wavelength depending on the maturity stage. Yamamoto et al. proposed a method to detect
young fruits as well as mature and immature fruits using machine learning-based methods (decision tree andX-means) [16]. The method achieved a Recall of 0.80
and Precision of 0.88.
To generate a map of the tomato growth states, wemust detect the tomato fruits and estimate the maturity
stages, harvest times, and position of the detected fruits.
Based on these factors, Table 1 shows a comparison ofsome functions in previous studies. The functions in Table 1 are described as follows: the “Detection (Rate [%])”
is whether or not the recognition or detection of tomato
fruits are possible and their results based on the evaluation method of each study. The “Maturity stage” is how
many stages the detected fruit is classified under; when it
is one, only the mature fruits are classified and when it istwo, the mature and immature fruits are classified. The“Harvest time” indicates whether or not the harvest timeis estimated for the detected fruits. The “Fruit position”
indicates whether or not the position of the detected fruitis calculated based on the world coordinate system. In Table 1, “” means that the function was achieved, and “–”means that it was not mentioned in the study. The method
proposed in this paper achieved all these functions. Thismethod recognizes tomato fruits regardless of the maturity stages using infrared imaging and estimates the maturity stages and harvest times using probability distributions quantified by combining the color value of the detected fruits with the expertise of experienced farmers.

2.2. Tomato Fruit Recognition
2.2.1. Optical Properties of Tomato Fruits on Infrared
ImagesWe focused on the optical properties of tomato fruits
as a method of recognizing them. Some studies recognized plant parts and evaluated maturity stages using theoptical properties of plants. Kondo and Monta et al. measured the spectral reflectance properties of each part of1281

Fujinaga, T., Yasukawa, S., and Ishii, K.

the plants (fruit, leaves, stems, etc.) using a spectrophotometer capable of measuring light from the visible spectrum to a wavelength of 2500 nm [17, 18]. Experimental results demonstrated that specific wavelength bands
are effective for recognizing specific plant parts. They
also reported that, as tomato fruits mature, the reflectancefrom 500 to 600 nm decreases, and the reflectance aroundthe chlorophyll absorption band at 670 nm increases.
This indicates that the tomato fruit turns red as it matures. Li et al. classified tomato fruits into six maturity stages and reported the changes in the response ofdifferent reflection spectra to these various stages [19].
The responses change relatively significantly in the 400
to 700 nm range. Moreover, the responses in the six
maturity stages are almost the same in the near-infrared
range. Therefore, we focused on the optical propertiesof tomato fruits in the near-infrared range to recognize
tomato fruits regardless of their maturity stages. Infrared
images were acquired using a Microsoft Kinect sensor(hereinafter, Kinect), which is a time-of-flight camera thatobtains images by projecting infrared light and measuring
the time the projected light reaches an object and returns.
In addition, a known optical property of tomato fruitsis that the central part has a strong response to the projected light and the peripheral parts have a weaker response. Ota et al. proposed a method of detecting tomato
fruits by focusing on their optical properties [20]. Thishas been confirmed from experimental results of the reflection response of tomato fruits on the infrared image
acquired using Kinect (Appendix A). Furthermore, experimental results have demonstrated that the infrared wavelength band of Kinect is effective in recognizing tomato
fruits regardless of their maturity stage. In this proposed
method, classifiers are generated by utilizing the opticalproperties of tomato fruits on infrared images.
2.2.2. Classifiers Utilizing the Optical Properties ofTomato Fruits(a) Example of positive data(b) Example of negative data
Fig. 3. Example of the dataset.

Table 2. Breakdown of the dataset.
Positive data

Negative data

Training data

479

1536

Test data

205

658

Total

684

2194

on the center point of the fruits with the largest reflection
response to the infrared light. For the negative data in
Fig. 3(b), the images were extracted from parts other than
the tomato fruits on the infrared image. The resolution of
the dataset was 21 × 21 pixel. It was defined as the resolution at which the optical properties of the tomato fruitson the infrared image (512 × 424 pixel) can be sufficiently
confirmed. A breakdown of the dataset is summarized inTable 2. The numbers of positive and negative data were
684 and 2194, respectively. Among them, for the training
data, 479 were positive data and 1536 were negative data,
and for the test data, 205 were positive data and 658 were
negative data.
Classifier 1 was generated using the linear SVM with
the HOG features as the input. The output value in thelinear SVM was 1 (tomato fruits) or 0 (not a tomato fruit).
The HOG features were extracted by calculating the gradient direction and intensity of the brightness of the input
image, creating histograms of the gradient for each cell,and normalizing by block. The input image was divided
into blocks, and each block was composed of cells. The
dimension of the HOG features (DHOG ) was calculatedusing the following equation:

Three methods were verified as classifiers generated byutilizing the optical properties of tomato fruits on infrared
images. In the first (hereinafter, classifier 1), histogramsof oriented gradients (HOG) features were extracted fromthe input image and the features were classified using a
support vector machine (SVM), which is a pattern recognition model [21, 22]. The second (classifier 2) was generated by learning a five-layer convolutional neural network(CNN) constructed by referring to the network architecture of LeNet, which is the basis of the CNN [23]. The
third (classifier 3) was generated by transfer learning for
GoogLeNet, which is a 22-layer CNN that was pre-trainedusing the ImageNet dataset [24, a]. We used MATLAB to
implement these classifiers.
Here, the dataset is described. Images of tomato
fruits or other objects (stems, leaves, etc.) were extracted from 50 infrared images acquired by a robot in
the Hibikinada Green Farm. Fig. 3 shows an example
of images of positive and negative data. For the positive data in Fig. 3(a), the images were extracted based

where Nwidth is the number of pixels in the width of the input image, Ncell is the number of pixels in a cell width, and
NBO is the number of cells overlapping adjacent blocks,

1282

Journal of Robotics and Mechatronics Vol.32 No.6, 2020

DHOG = NBPI 2 Nblock 2 Nbin ,. . . . . . . . (1)where NBPI is the number of blocks per image, Nblock is
the number of cells in a block width, and Nbin is the number of gradient directions. NBPI is calculated using thefollowing equation:
Nwidth− Nblock
N
NBPI = cell
+ 1, . . . . . . . . (2)Nblock − NBO

Tomato Growth State Map for Monitoring and HarvestingFig. 4. Comparison of accuracy by varying the Ncell and Nblock .

Fig. 5. Architecture of referencing the LeNet.

which is calculated using the following equation:
Nblock. . . . . . . . . . . . . . (3)2
NBPI is rounded toward negative infinity and a valid
value only for a positive value. NBO is rounded toward
negative infinity when Nblock is one, and rounded toward
positive infinity when Nblock is one or more. In the extraction of HOG features, although the ratio of width andheight in the size of the input image, block, and cell isbasically arbitrary, we defined that the ratio is the same inthis study. We fixed Nwidth and Nbin at 21 and nine, respectively, and focused on Ncell and Nblock . To optimize theseparameters, we evaluated the recognition rate by varying
them. The results are shown in Fig. 4. The accuracy represents the correct rate of the classification results for the
categories (positive or negative) labeled on the test data. If
the HOG features cannot be extracted using Eq. (1) (e.g.,
Nblock is four and Ncell is six), no data exists. The highest accuracy is 0.965 when Nblock is three and the Ncell isfour, and Nblock is five and the Ncell is two. Here, focusing
on DHOG in these scenarios, the former is 729 and the latter is 2025 from Eq. (1). Moreover, the results are stable
when Ncell is one regardless of Nblock . Because Ncell is one
pixel, the histograms are calculated for each pixel, andthey are normalized by Nblock . Therefore, more characteristics of tomato fruits on the infrared image can be extracted, and the recognition rate result is stable. However
because the histogram is created for each pixel, DHOG becomes large. Because the processing is faster when DHOG
is lower, we extract the HOG features with a Nblock of
three and Ncell of four.
For classifier 2, Fig. 5 shows the architecture of referencing the LeNet with the 5-layer CNN. The convoNBO =Journal of Robotics and Mechatronics Vol.32 No.6, 2020

lutional layer (Conv.) and pooling layer (Pool.) are repeated twice, and finally, through the fully connected
layer (F.C.), the probability of being classified as tomato
fruits and other objects is the output. Although the sigmoid function is used as an activation function in LeNet,
the ReLU function was used in this study. In addition, although LeNet reduces the data size of each layer by subsampling, max pooling was adopted in this study. For
the resolution of the images (32 × 32, 28 × 28, 14 × 14,
10 × 10, and 5 × 5 pixel in Fig. 5), the number of featuremaps in this architecture (6 and 16 in Fig. 5), and the dimension of the fully connected layer (120 in Fig. 5), thevalues of LeNet were referenced. Although the dataset is
21 × 21 pixel in Table 2, it was resized to 32 × 32 pixel tofollow the architecture of LeNet. The output was a value
in the range of 0.0 to 1.0, using the softmax function,
which was classified as tomato fruits and other objects.
The result depended on the setting of the threshold value
for the output. In this study, the threshold value was set
at 0.99. The network was trained using stochastic gradient descent with momentum with an initial learning rateof 0.0001. The maximum number of epochs was set to 30.
For classifier 3, GoogLeNet, which is a 22-layer CNN,
was trained using ImageNet containing more than 14 million images for general object recognition and was retrained using the training dataset shown in Table 2 fortomato fruit recognition. The input image was resizedfrom 21 × 21 to 224 × 224 pixel, adjusted to be the sameas GoogLeNet. The output had a value in the range of 0.0
to 1.0 as with classifier 2. The threshold value in classifier 3 was set to 0.99. The network learning method was
the same as for classifier 2.
Here, for the threshold value, if the threshold value
was changed, the results were changed. In this study, thethreshold value was determined by prioritizing the resultof the F1-score in the detection method described in Section 2.2.3.
2.2.3. Evaluation of the Classifiers
Two approaches were used to evaluate the three classifiers: the recognition rate (focusing only on Accuracy)
when using the test data in Table 2, and the detection rates(focusing on the Precision, Recall, and F1-score) when
detecting tomato fruits using the images acquired at the
Hibikinada Green Farm. The recognition rate was thecorrect rate of the classification results for the categories(positive or negative) labeled on the test data. The detection method for the detection rates is shown in Fig. 6.
The infrared image (521 × 424 pixel) was the input (process (i), Fig. 6), and the sliding window method was
adopted to detect tomato fruits in the input image (process (ii), Fig. 6). With this method, a small region wasscanned on the input image for each fixed pixel, and aclassification result (tomato fruits or other objects) was
output for each small region. Here, the number of fixed
pixels when scanning a small region was called a step.
The detection rates depended on the number of steps. In
this study, we set the number to 1 pixel (here, called the1283

Fujinaga, T., Yasukawa, S., and Ishii, K.

Table 3. Comparison of the results in each classifier.
Classifier(Dim.)

Recognition
rate
Accuracy

Precision

Recall

F1-score

1 (769)

0.965

0.934

0.775

0.847

2 (1,024)

0.931

0.711

0.769

0.739

3 (50,176)

0.957

0.651

0.610

0.630

Detection rate

Fig. 6. Flowchart of tomato fruit detection.

entire search) to evaluate the classifiers, although the computation times were large. The resolution of the small region was 21 × 21 pixel, which was the same as the dataset.
As shown in process (ii) of Fig. 6, one fruit may have two
or more recognition points. Therefore, we adopted themean shift method, which is an efficient and robust solution to the mode search problem, to converge two or
more recognition points into one recognition point (process (iii), Fig. 6) [25].
Table 3 shows the input dimensions of each classifier
and the results of the recognition and detection rates. Accuracy was the result when the test data in Table 2 wasused, and Precision, Recall, and F1-score were the results when 30 infrared images were used (test data andthe 30 infrared images were different). The most accurate as well as the highest F1-score, which had a tradeoff relationship with Precision and Recall, was classifier 1. In this study, 50 infrared images were used to create
the dataset, although neural network with deep layers require more training data [a]. Therefore, when the entire
search was performed, the unlearned data was the input,
and the detection rates became lower than the recognition
rate. Moreover, these results indicated that high recognition and detection rates in classifier 1 could be obtainedeven with a limited dataset and simple pattern recognition
model by focusing on the optical properties of the tomato
fruits and extracting the characteristics in advance. In thisstudy, classifier 1 was used as a method for recognizing
the fruits by considering the recognition rate, detection
rates, and input dimension.

∗1 Maturity stage: Green, Breakers, Turning, Pink, Light Red, Red∗2 Harvest time: 1st week, 2nd week, 3rd week, 4th week

Fig. 7. Examples of images used in the survey.

2.3. Growth State Estimation
2.3.1. Survey of Experienced Farmers
The growth states (maturity stages and harvest times)
were added to the mosaic image based on the resultsof the surveys of experienced farmers at the Hibikinada
Green Farm. These farmers can predict the yield in thetomato greenhouses based on the tomato growth state map
by designating the maturity stages of the tomato fruits tobe harvested. Moreover, they can create a shipping plan
based on the demand forecasts from their clients and adjust production to reduce food loss by obtaining the harvest times for the fruits from this map.

The numbers of classifications for the maturity stages
and terms for the harvest times were determined by discussions with experienced farmers. The USDA has categorized the maturity stages into six classifications basedon the color of the fruit surface: “Green,” “Breakers,”“Turning,” “Pink,” “Light Red,” and “Red.” Choi et al.
used these as an index to assess the maturity stages, andLi et al. used them to investigate the response of the reflectance spectra [7, 19]. We also focused on these six maturity stages. Next, the harvest times are described. The
tomato growth state map in this study focused on the area
(80–120 cm from the ground in height) where the workers can easily harvest the fruits. The harvest times within
this area are known to the farmers from the 1st to the 2nd
weeks. However, fruits are still harvested even after these
two terms; therefore, the harvest times are divided into
four terms: the 1st, 2nd, 3rd, and 4th weeks.
We conducted the survey with the cooperation of nine
experienced farmers at the Hibikinada Green Farm toquantify the growth states based on their expertise.
In this survey method, the experienced farmers wereasked to select the maturity stages and harvest times
that they considered suitable for the fruits in the images(Fig. 7). For example, for fruit number one in Fig. 7, theexperienced farmer selected Breakers for maturity stage
and 1st week as harvest time. In this survey, we asked nine
experienced farmers for their insights and used 50 images
existing 684 fruits in total (each image was structured asshown in Fig. 7).

1284

Journal of Robotics and Mechatronics Vol.32 No.6, 2020

Tomato Growth State Map for Monitoring and Harvesting(a) Results of maturity stages in this survey

Fig. 8. Example of survey results.

2.3.2. Survey Results and Their Quantification
The color value of the tomato fruits on the images usedin the survey was linked to the survey results of experienced farmers. Fig. 8 is an example showing the survey results for the maturity stages of respondents A, B,
and C for each tomato fruit. The results are summarized
by defining the color value on the horizontal axis and thefrequency of the number of respondents on the verticalaxis. Here, the hue on the horizontal axis is normalized
from 0.0 to 1.0. For example, for fruit number one in
Fig. 8, all respondents selected Red, so the frequency of
Red was three for hue 0.00.
The survey results for the maturity stages and harvest
times are shown in Figs. 9(a) and 10(a), respectively.
There were apparent false answers in this survey, e.g.,
Green as selected for mature fruits. Therefore, to quantify
the expertise of experienced farmers using these results,
we had to extract only the valid data. In this study, thefrequency of the number of respondents as valid data wasdefined as one-third or more of the total number of respondents, i.e., three or more. In this definition, for the harvest
times, since the frequency of selecting the 4th week was
low, it was redivided into three terms: 1st, 2nd, and 3rd
weeks.
The probability distribution of each valid data at eachmaturity stage and harvest time was calculated based on
the probability density function f (x), as follows:
1(x − μ )2
f (x) = √exp −, . . . . (4)
2σ 2
2πσ 2
where x is the color value (hue), μ is the average ofvalid data at each maturity stage or harvest time, and σ
is their standard deviation. The analysis results of the maturity stages and harvest times are shown in Figs. 9(b)and 10(b), respectively. The maturity stages and harvest
times for the detected fruits were estimated by the ranges
using points intersections of the adjacent probability density functions for each maturity stage and harvest time.
Journal of Robotics and Mechatronics Vol.32 No.6, 2020(b) Probability distribution at each maturity stage
Fig. 9. Results of the maturity stages.

(a) Results of harvest times in this survey(b) Probability distribution at each harvest time
Fig. 10. Results of the harvest times.

1285

Fujinaga, T., Yasukawa, S., and Ishii, K.

3. Evaluation of the Tomato Growth State Map

Fig. 11. Estimation method of maturity stages and harvest times.

That is, the hue of each pixel in the detected fruit image was inputted into the probability density function ofeach maturity stage and each harvest time, and those withthe highest values are determined to be the maturity stage
and harvest time for the pixel. However, for Red andGreen in the maturity stages and the 1st and 3rd weeks
in the harvest times, to decide the thresholds in both ends,
we used 10σ (the quantification limit) of each probability density function. Here, since the hue was normalized
from 0.0 to 1.0, the left end of Red and the 1st week were
negative, and one was added to these values.
2.3.3. Estimation Method
Figure 11 shows a method to estimate the maturity
stages and harvest times: “Term 1,” “Term 2,” and“Term 3” represent the 1st, 2nd, and 3rd weeks, respectively. The region in which the fruits were detected wasused as an input image (process (i), Fig. 11), and the image was analyzed for each pixel based on the probability distribution described in Section 2.3.2 (process (ii),Fig. 11). The resolution of the input image was 10 ×
10 pixel based on the center coordinates of the detected
fruits. This was because 10 × 10 pixel was half the resolution of the fruit area (21 × 21 pixel) on the image, considering that the target fruits overlapped with other fruitsin the clusters. The hue of each pixel was inputted to theprobability density function of each maturity stage andeach harvest time, and the maturity stage and harvest timeat which the value was the highest were the classification
results of the pixel. When the hue as an input fell outside
the threshold of the maturity stage and harvest time, i.e., if
it exceeded the quantification limit, that pixel was considered to be noise. The results analyzed for each pixel wereconstructed into a histogram, and the one with the highest frequency was used as the maturity stage and harvest
time of the fruit (process (iii), Fig. 11). In the histogramof the fruits in Ex. 1 of Fig. 11, the results indicated thatthe distribution was high for Red as the maturity stage andTerm 1 as the harvest time.
1286

To evaluate the effectiveness of the tomato growth state
map, we conducted experiments at the Hibikinada Green
Farm from July 6 to August 4, 2020, targeting 30 tomato
plants. The map was generated before harvesting work on
July 6, 2020, and the results are reported in this paper forthat date. A part of the tomato growth state map generated
on July 6, 2020 is shown in Fig. 12. This map was the result of tomato fruit detection and growth state estimation
performed on a mosaic image composed of 50 images.
The illuminance was measured every time the image was
acquired, and the average illuminance in the experiments
was 0.9k lx and the standard deviation was 0.4k lx. In
Fig. 12, the growth states (“M.S.” and “H.T.” mean thematurity stage and harvest time, respectively) and the position (“Pos.” indicates the x, y, and z coordinates in theworld coordinate system; the unit is meter) are clearlyshown for three representative fruits among the detected
fruits, but all the detected fruits have the same information.
Table 4 shows the detection rates (Precision, Recall,F1-score) and the estimation results for the maturity
stages and harvest times. In this map, 136 tomato fruits
were confirmed and 104 could be detected. The numberof false recognition points was 11. For the detection rate
results, the F1-score was 0.829. The maturity stages andharvest times of the detected fruits (104 fruits) were estimated. Table 5 shows a comparison of the estimation results and actual number of harvested fruits. “Estimation”
is the estimation results for the proposed method and “Actual” is the actual results. Additionally, the term 4th week
is deemed invalid, as described in Section 2.3.2. However,since there was one scenario in the actual results, this term
was added to Table 5. For the “Yield,” we compared thenumber of actual harvested fruits and the estimation results based on the maturity stages Pink as designated byexperienced farmers on July 6, 2020. The Yield of the Estimation (53, Table 5) was the total number of Pink (14,Table 4), Light Red (24, Table 4), and Red (15, Table 4)
tomato fruits in the tomato growth state map. For the harvest times, we compared the estimation results and actual
harvest times based on the date when the tomato growthstate map was generated (July 6, 2020) by recording the
growth states of the detected fruits after this date.

4. Discussion
4.1. Comparison with Previous Studies
Table 6 shows the results of this method for the functions mentioned in Section 2.1. “Detection (Rate [%])”
refers to the F1-score in Table 4. For the six classifications in the “Maturity stage,” they represent Green, Breakers, Turning, Pink, Light Red, and Red as defined by the
USDA. To generate the tomato growth state map, this proposed method achieved all the functions. In particular,
this study contributed to achieving the estimation of harJournal of Robotics and Mechatronics Vol.32 No.6, 2020

Tomato Growth State Map for Monitoring and HarvestingFig. 12. Tomato growth state map.

Table 4. Results in the tomato growth state map.
Detection rates
Precision

0.904

Recall0.765

F1-score

Table 6. Evaluation of the proposed method.

 104 


 115
104

Proposedmethod136

0.829

Detection(Rate [%])

Maturity
stagesHarvest
times

Fruit
position


(83)

6





Maturity stages
Green

32

Breakers

5

Turning

14

Pink

14

Light Red

24

Red

15
Harvest times

1st week

66

2nd week

20

3rd week

18

Table 5. Comparison of estimation results and the actual
number of harvested fruits.

Yield

Estimation

Actual53

48

1st week

66

65

2nd week

20

38

3rd week

18

0

4th week–

1Fig. 13. Existence of the occlusion of tomato fruits depending on the image acquisition position.

4.2. Comparison of Detection Results and Actual
HarvestIn the experiments, 30 tomato plants were targeted,and 205 tomato fruits were confirmed among the tomato
plants. However, 136 tomato fruits were confirmed in
the tomato growth state map. This was because of themosaic image generation method. Frequently, a mosaic
image was generated using images captured at a location at which the distance from the camera to the ob-ject is sufficiently large. In contrast, in the tomato greenhouse, the distance from the camera to the tomato plants(tomato fruits, leaves, stems, etc.) was relatively close,
and this distance was not constant. Therefore, when theimages were overlaid, they had discontinuities between
them [10]. In addition, since the mosaic image used asingle-viewpoint image, some fruits could not be confirmed in the image owing to occlusion between fruits inthe tomato clusters or owing to leaves or stems. However, the mosaic image was generated using images captured at adjacent points and with overlapping regions, i.e.,
multi-viewpoint images were acquired for certain tomato
fruits or clusters. Fig. 13 shows the scenario in which
images were acquired at positions P1 and P2 . Figs. 13(i)and (ii) indicate the viewpoints of the cultivation area inthe tomato greenhouse from the front and top views, respectively. For position P1 , fruits number one to three ofthe center tomato cluster in Fig. 13(iii) were confirmed.

Journal of Robotics and Mechatronics Vol.32 No.6, 2020

1287

vest time, which could not be achieved by previous studies(Table 1).

Fujinaga, T., Yasukawa, S., and Ishii, K.

In contrast, for position P2 , all the tomato fruits, i.e.,fruits numbers one to five of the center tomato cluster in
Fig. 13(iv), were confirmed.
Therefore, we examined the actual tomato fruits anddetected tomato fruits in each of the 50 images used togenerate the mosaic image for this study. Each fruit inthese images was identified and counted: 198 fruits wereidentified out of an actual count of 205, while 165 fruits
out of the 198 were detected with the proposed method.
Seven fruits could not be completely confirmed owing to
occlusion. We can conclude that when generating the mosaic image, each fruit can be identified based on the position information of the fruits, and information on thehidden fruits is then added to the map.

4.3. Estimation Results
The total number of fruits at the Pink maturity stage
or higher designated by the experienced farmers was 53,
whereas the actual fruits harvested were 48. The workers harvest each tomato fruit in approximately 4 s, but thetime to judge its maturity stage is only a moment. Therefore, in some scenarios, the tomato fruits to be harvested
are missed, or a decision to not harvest it is made. The
actual number of harvests without variation can be estimated using this estimation method based on the expertiseof experienced farmers.
For harvest time, 66 fruits were estimated to be the 1st
week whereas 65 fruits were actually harvested so that theestimated results were equivalent to the actual results. In
contrast, for the 2nd and 3rd weeks in Table 5, the estimated result was 20 fruits for the 2nd week and 18 fruits inthe 3rd week, but in the actual harvest, there were 38 fruits
for the 2nd week. In the survey to quantify the expertiseof experienced farmers, we asked them to make decisions
based on the images; however, they also cultivated tomato
plants in consideration of daily changes in temperature,humidity, and other factors. Therefore, we observed thatthey had difficulty making an estimation only by judging
the images.
In addition, one fruit was harvested in the 4th week
and it required four weeks to attain maturity. Thus sometomato fruits grow irregularly. On the other hand, almost
all of the tomato fruits from 80 to 120 cm from the groundwere harvested within two weeks based on the date when
the tomato growth map was generated. The position of
the tomato fruits on the map and harvest times may beclosely related. In the future, we will propose an estimation method that considers images and environmental
factors such as temperature and humidity, as well as focus
on the position of tomato fruits, to clarify the relationshipbetween fruit position and harvest time.

5. Conclusion
In this study, we aimed to realize a system that uses
robots to automate farming processes by monitoring the
growth states to harvest tomato fruits. The generation
1288

method for a map of the tomato growth states, used forthe growth management of tomato plants and decision of
harvesting strategies for robots, is described in this paper.
The tomato growth state map was evaluated through experiments in an actual tomato greenhouse, and we demonstrated its effectiveness.
For tomato fruit recognition, we demonstrated that asimple machine learning method with a limited learning
dataset exceeded the recognition results of the more complicated CNN using the optical properties of tomato fruits
on infrared images. Although the sliding window method
was applied to evaluate these classifiers, the method is inferior to other deep learning detection methods in terms ofcomputation times [26, 27]. To implement the proposed
method in the system, we must consider the computation
times.
For growth state estimation, the maturity stages andharvest times were quantified based on a survey of experienced farmers. The proposed method proved to be effective for short-term estimations (the number of harvests
on a certain day and the harvest time of the 1st week). In
addition, we observed that even experienced farmers havedifficulties estimating the long-term state of the fruits andharvest times for the 2nd week or 3rd week when using
only image information. Further studies are underway toanalyze both the images and environmental factors (temperature, humidity, etc.) and position of the tomato fruits
to evaluate fruit growth states for both short- and longterm estimations. In addition, in the estimation method,
the highest frequency of growth state was used as the output result from the histogram of each pixel in the input
image, but this method could not utilize the distributionof the histogram. A future task we must consider is an
estimation method using the distribution of each pixel on
the input image.
Acknowledgements
In this survey and the experiments, we received cooperation fromthe Hibikinada Green Farm Co., Ltd., Hideyuki Ikari, Kazushi
Kawajiri, and Keisuke Kimoto. We would like to express our sincere gratitude for their kind assistance. This work was supported
by a Grant-in-Aid for JSPS Fellows 19J10900.

References:[1] N. Noguchi, “Agricultural Vehicle Robot,” J. Robot. Mechatron.,
Vol.30, No.2, pp. 165-172, 2018.
[2] N. Kondo, K. Yata, M. Iida, T. Shiigi, M. Monta, M. Kurita, andH. Omori, “Development of an End-effector for a Tomato Cluster
Harvesting Robot,” Engineering in Agriculture, Environment andFood, Vol.3, No.1, pp. 20-24, 2010.
[3] H. Yaguchi, K. Nagahama, T. Hasegawa, and M. Inaba, “Development of An Autonomous Tomato Harvesting Robot with Rotational
Plucking Gripper,” 2016 IEEE/RSJ Int. Conf. on Intelligent Robots
and Systems, pp. 652-657, 2016.
[4] T. Yoshida, T. Fukao, and T. Hasegawa, “Fast Detection of Tomato
Peduncle Using Point Cloud with a Harvesting Robot,” J. Robot.
Mechatron., Vol.30, No.2, pp. 180-186, 2018.
[5] T. Fukatsu and M. Hirafuji, “Field Monitoring Using Sensor-Nodeswith a Web Server,” J. Robot. Mechatron., Vol.17, No.2, pp. 164172, 2005.
[6] R. Fukui, K. Kawae, and S. Warisawa, “Development of a Tomato
Volume Estimating Robot that Autonomously Searches an Appropriate Measurement Position – Basic Feasibility Study Using a

Journal of Robotics and Mechatronics Vol.32 No.6, 2020

Tomato Growth State Map for Monitoring and Harvesting

Tomato Bed Mock-Up –,” J. Robot. Mechatron., Vol.30, No.2,
pp. 173-179, 2018.
[7] K. Choi, G. Lee, Y. J. Han, and J. M. Bunn, “Tomato Maturity Evaluation Using Color Image Analysis,” American Society of Agricultural Engineers, Vol.38, No.1, pp. 171-176, 1995.
[8] D. Kusui, H. Shimazu, K. Suezawa, and A. Shinjo, “A Novel Agricultural Skill Learning Support System and its Development and
Evaluation,” 2018 Joint 10th Int. Conf. on Soft Computing and Intelligent Systems and 19th Int. Symp. on Advanced Intelligent Systems, pp. 1020-1025, 2018.
[9] K. Wakamori and H. Mineno, “Optical Flow-Based Analysis of theRelationships between Leaf Wilting and Stem Diameter Variations
in Tomato Plants,” Plant Phenomics, Vol.2019, Artile ID 9136298,
2019.
[10] T. Fujinaga, S. Yasukawa, B. Li, and K. Ishii, “Image Mosaicing
Using Multi-Modal Images for Generation of Tomato Growth State
Map,” J. Robot. Mechatron., Vol.30, No.2, pp. 187-197, 2018.
[11] T. Fujinaga, S. Yasukawa, and K. Ishii, “System Development ofTomato Harvesting Robot Based on Modular Design,” 2019 Int.
Workshop on Smart Info-Media Systems in Asia, SS1-1, pp. 1-6,
2019.
[12] R. F. Teimourlou, A. Arefi, and A. M. Motlagh, “A Machine Vision System for the Real-Time Harvesting of Ripe Tomato,” J. of
Agricultural Machinery Science, Vol.7, No.2, pp. 159-164, 2011.
[13] K. Hatou and Y. Hashimoto, “Recognition System for Tomato Fruits
Based on the Wire Size Reduction Using Thermal Image,” Environmental Control in Biology, Vol.40, No.1, pp. 75-80, 2002 (inJapanese).
[14] L. Wang, B. Zhao, J. Fan, X. Hu, S. Wei, Y. Li, Q. Zhou, and C.
Wei, “Development of a tomato harvesting robot used in greenhouse,” Int. J. of Agricultural and Biological Engineering, Vol.10,
No.4, pp. 140-149, 2017.
[15] T. Fujiura, S. Nakao, N. Kondo, M. Dohi, and J. Yamashita,“Dichromatic 3-D Vision Sensor for the Agricultural Robot,” J. of
Japanese Society of Agricultural Technology Management, Vol.2,
No.1, pp. 59-64, 1995 (in Japanese).
[16] K. Yamamoto, W. Guo, Y. Yoshioka, and S. Ninomiya, “On Plant
Detection of Intact Tomato Fruits Using Image Analysis and Machine Learning Methods,” Sensors, Vol.14, No.7, pp. 12191-12206,
2014.
[17] N. Kondo, “Selection of Suitable Wavelength Bands for Discrimination between Parts of Plant Body Using Their Spectral
Reflectances,” Environmental Control in Biology, Vol.26, No.4,
pp. 175-183, 1988 (in Japanese).
[18] M. Monta, N. Kondo, S. Arima, and K. Namba, “Robotic Vision
for Bioproduction Systems,” J. Robot. Mechatron., Vol.15, No.3,
pp. 341-348, 2003.
[19] B. Li, J. Lecourt, and G. Bishop, “Advances in Non-Destructive
Early Assessment of Fruit Ripeness towards Defining Optimal Timeof Harvest and Yield Prediction – A Review,” Plants, Vol.7, No.3,
2018.
[20] T. Ota, S. Hayashi, K. Kubota, K. Ajiki, T. Komeda, and S. Otsuka, “Tomato Fruit Detection System Using Specular Reflection,”J. of the Japanese Society of Agricultural Machinery, Vol.67, No.6,
pp. 95-104, 2005 (in Japanese).
[21] N. Dalal and B. Triggs, “Histograms of Oriented Gradients for Human Detection,” 2005 IEEE Computer Society Conf. on Computer
Vision and Pattern Recognition, pp. 886-893, 2005.
[22] C. Cortes and V. Vapnik, “Support-Vector Networks,” Machine
Learning, Vol.20, pp. 273-297, 1995.
[23] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based
Learning Applied to Document Recognition,” Proc. of the IEEE,
Vol.86, No.11, pp. 2278-2324, 1998.
[24] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov,
D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper withconvolutions,” 2015 IEEE Conf. on Computer Vision and Pattern
Recognition, 2015.
[25] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach Toward Feature Space Analysis,” IEEE Trans. on Pattern Analysis andMachine Intelligence, Vol.24, No.5, pp. 603-619, 2002.
[26] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards
Real-Time Object Detection with Region Proposal Networks,” Advances in Neural Information Processing Systems, pp. 91-99, 2015.
[27] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and
A. C. Berg, “SSD: Single Shot MultiBox Detector,” European Conf.
on Computer Vision, pp. 21-37, 2016.

Supporting Online Materials:[a] ImageNet. http://www.image-net.org [Accessed July 29, 2020]

Journal of Robotics and Mechatronics Vol.32 No.6, 2020

Fig. 14. Outline and evaluation method of the experiments
to investigate the reflection response of tomato fruits.

Appendix A. Reflection Response of Tomato
Fruits on Infrared Images
This appendix describes the experiments conducted toinvestigate the reflection response of tomato fruits to theinfrared light projected by Kinect. In these experiments,
we investigated the reflection response when the maturity
stages were changed and the distance between the Kinect
and tomato fruits was changed.
The outline and evaluation method for the experiments
are shown in Fig. 14. The experiments were conducted
one at a time with tomato fruits placed on a desk. The
distance between the tomato fruits in the target cultivation
area and the Kinect mounted on the robot was between 60
to 80 cm when the robot moved on rails in the tomato
greenhouse. Therefore, for the distance d between thetomato fruits and the Kinect in Fig. 14(b), three scenarios
were considered: 60, 70, and 80 cm. For each distance,
30 tomato fruits were used: five tomato fruits each for six
different maturity stages (Green, Breakers, Turning, Pink,
Light Red, and Red).
To evaluate the reflection response, the 21×21 pixel region centered on the middle of the tomato fruit in the image, which was the position that responded most strongly
to infrared light, was extracted from the infrared image(processes (i) and (ii), Fig. 14). Here, 21 × 21 pixel was
the resolution of tomato fruits when d was 60–80 cm.
With a horizontal line (0◦ ) that passed through the center point of the extracted image and lines that were tilted
by 45◦ , 90◦ , and 135◦ from the horizontal line, the pixel
rows corresponding to the four lines were extracted (process (iii), Fig. 14). The extracted pixel rows were used
to evaluate the response of tomato fruits to the infrared
image (process (iv), Fig. 14). Here, the data type of theinfrared image acquired by Kinect was uint (unsigned integer) 16, but for convenience, by normalizing from 0.0
to 1.0, this value was used as the response for each position of the pixel row.
1289

Fujinaga, T., Yasukawa, S., and Ishii, K.

(a) d = 60 cm(a) Darkroom(b) d = 70 cm(c) d = 80 cm
Fig. 16. Response results for each maturity stage when thed changed in the darkroom.

(b) Tomato greenhouse
Fig. 15. Response results of each pixel row for six different
maturity stages at a d of 60 cm.

Experiments were conducted in two places, i.e., in a
darkroom and in a tomato greenhouse, to confirm thechanges in the response owing to the difference in the effect of ambient light. The illuminance in the darkroom
was 0.0 lx, and the illuminance in the tomato greenhouse
was 8.2k lx. The illuminance in the tomato greenhouse
was the average of 90 image acquisitions (six (maturity
stages) × five (number of tomato fruits) × three (d)), andthe standard deviation of 0.7k lx.
Figure 15 shows the response results of each pixel1290

row for six different maturity stages at the d of 60 cm.
The response results for each pixel row in Fig. 15 were
the average value for each position of five tomato fruits.
Figs. 15(a) and (b) show the results for the darkroom andgreenhouse, respectively. Fig. 16 shows the response results of each maturity stage of the tomato fruits in thedarkroom, and Fig. 17 shows the results in the tomato
greenhouse for a changing d. The response results at eachmaturity stage in Figs. 16 and 17 are the average values at
each position of all pixel rows for five tomato fruits.
The figures indicate that the response to infrared light
exhibited the same tendencies regardless of the maturity
stages of the tomato fruits in Fig. 15, i.e., the response at
the center of the fruits was high and the surrounding area
was weaker than the center of the fruits. In addition, under
the conditions of these experiments, we confirmed that thesame tendency in reflection response in both the darkroom
and tomato greenhouse. Figs. 16 and 17 indicate that theintensity of reflection response changed, but the transitionof reflection response did not change even if the distance
between tomato fruits and the Kinect changed.

Journal of Robotics and Mechatronics Vol.32 No.6, 2020

Tomato Growth State Map for Monitoring and Harvesting

Name:
Shinsuke Yasukawa

Affiliation:
Associate Professor, Department of Human Intelligence Systems, Kyushu Institute of Technology(a) d = 60 cm

Address:
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan

Brief Biographical History:
2014-2017 Research Associate, Kyushu Institute of Technology
2017 Received Ph.D. from Division of Electrical, Electronic andInformation Engineering, Osaka University
2017-2018 Research Associate, The University of Tokyo
2018- Associate Professor, Kyushu Institute of Technology

Main Works:(b) d = 70 cm• “Real-Time Object Tracking Based on Scale-Invariant Features
Employing Bio-Inspired Hardware,” Neural Networks, Vol.81, pp. 29-38,
2016.
• “Vision system for an autonomous underwater vehicle with a benthos
sampling function,” J. Robot. Mechatron., Vol.30, No.2, pp. 248-256,
2018.

Membership in Academic Societies:• The Institute of Electrical and Electronics Engineers (IEEE)• The Robotics Society of Japan (RSJ)

Name:
Kazuo Ishii

Affiliation:
Professor, Department of Human Intelligence
Systems, Kyushu Institute of Technology(c) d = 80 cm
Fig. 17. Response results for each maturity stage when thed changed in the tomato greenhouse.
Address:Name:
Takuya Fujinaga

Affiliation:
Department of Life Science and Systems Engineering, Kyushu Institute of Technology

2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan

Brief Biographical History:
1996- Researcher, Institute of Industrial Science, The University of Tokyo
1996-1998 Assistant Professor, Kyushu Institute of Technology
1998-2011 Associate Professor, Kyushu Institute of Technology
2011- Professor, Kyushu Institute of Technology

Main Works:

2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan• “An Optical Image Transmission System for Deep Sea Creature
Sampling Missions Using Autonomous Underwater Vehicle,” IEEE J. of
Oceanic Engineering, Vol.45, Issue 2, pp. 350-361, 2020.
• “Image Mosaicing Using Multi-Modal Images for Generation of Tomato
Growth State Map,” J. Robot. Mechatron., Vol.30, No.2, pp. 187-197,
2018.

Brief Biographical History:

Membership in Academic Societies:

Address:
2016 Received Bachelor’s degree from Department of Mechanical
Information Science and Technology, Computer Science and Systems
Engineering, Kyushu Institute of Technology
2018 Received Master’s degree from Department of Human Intelligence
Systems, Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology
2018- Doctoral Candidate in Department of Life Science and Systems
Engineering, Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology• The Institute of Electrical and Electronics Engineers (IEEE)• The Japan Society of Mechanical Engineers (JSME)• The Robotics Society of Japan (RSJ)• The Japan Society of Naval Architects and Ocean Engineers (JSNAOE)

Main Works:• “Image Mosaicing Using Multi-Modal Images for Generation of Tomato
Growth State Map,” J. Robot. Mechatron., Vol.30, No.2, pp. 187-197,
2018.

Membership in Academic Societies:• The Japan Society of Mechanical Engineers (JSME)

Journal of Robotics and Mechatronics Vol.32 No.6, 2020

Powered by TCPDF (www.tcpdf.org)

1291