# Tomato Maturity Estimation Using Deep Neural Network
## 3 Materials and Methods  
### Data Collection  
The tomato (_Solanum lycopeneum_ L.) \"Dafnis\" variety, was cultivated in a general hydroponic greenhouse in South Korea for this study; Figure 1 shows an interior view of the greenhouse, as well as representative sample images. Tomato fruits had various sizes, with a weight range of 150 g to 250 g, and an almost round shape. Images of tomatoes were captured automatically using a developed monitoring system [20] that can travel remotely along the hot water pipes installed in the greenhouse. The monitoring system can track a straight path following a magnetic line installed between both sides of the pipes and can detect the start and end points of crop beds using proximity sensors. A camera (RealSense D435; Intel, Santa Clara, CA, USA) was installed on the side of the monitoring robot to capture the images of the tomatoes, and the images were saved with 800 \(\times\) 600 pixel resolution at 30 fps. The environmental conditions were 20.6 \({}^{\circ}\)C, 67.7%, and 52.9 w/m\({}^{2}\) for humidity, temperature, and light intensity, respectively.  
Each collected image contains multiple tomatoes at various growth stages, and the tomatoes were annotated for use as samples in training the deep learning model. An annotation tool was developed using Python 3.8 and OpenCV 4.1, which can support the drawing of a polygon shape around each object (tomato). The tomato objects in each image  
Figure 1: The greenhouse structure (**left**) and a sample captured image (**right**). The greenhouse is of a general hydroponic type, and hot water pipes and a magnetic line were installed between the two crop beds.  
were extracted as rectangles, that is, the smallest bounding box including the polygon of the tomato object. The extracted samples were used with the background removed to represent the maturity features by deep learning. The overall process is depicted in Figure 2.  
The background removed samples had different sizes; thus, all of the samples were resized to 128 \(\times\) 128 pixels (which is the input size of the model) for use as training data, after making each sample square with padding in order to maintain the aspect ratio. In addition, each sample was classified into four maturity stages (green, turning, pink, and red), in order to provide the labels for supervised learning. In general, the maturity stages of tomatoes can be divided into six stages (green, breaker, turning, pink, light red, and red), depending on the ratio of the red region to the entire region [21]; however, it is hard to determine the maturity stage using only the visual information in RGB images examined by humans, as the boundary of the red region is ambiguous due to the continuous color change in the tomato skin. Therefore, the intermediate stages are more difficult to assess. We aimed to evaluate the successive values between maturity stages through classification learning between specific classes; thus, the use of four stages was considered appropriate in this study, as these stages can be easily distinguished by humans. The samples labeled by maturity stage were randomly divided into training and test sets in a 1:1 ratio, and half of the training set was used as a validation set. The numbers of samples were 472, 472, and 944 in the training, validation, and test sets, respectively.