# Green Sweet Pepper Fruit and Peduncle Detection Using Mask R-CNN in Greenhouses
###### Abstract
Ref. [28], aiming to achieve real-time detection of apple fruit, improves the YOLO-V3 model, and uses the DenseNet method to process the initial layers of low-resolution features. The experimental results showed that the improved model performed better than the Fast R-CNN model and the original YOLO-V3 model. However, the deep learning models mentioned above (R-CNN, Fast R-CNN, Faster R-CNN, and YOLO) can only approximately calculate the position of the object using bounding boxes; ergo, such models are not capable of accurately extracting shape and contour features and information. It is well known that hard nuts such as pears, apples, and citrus, among others, can be harvested by pulling, that is, identifying the crop's fruit is sufficient. In comparison, the harvesting of sweet peppers is achieved by cutting the stem at the point of collection, in this case, the peduncle, this is to avoid damaging the integrity and value of the product. In this way, a high precision and real-time recognition of the shape and contour of the crop is necessary, with special attention to its peduncle, which indicates that neither the machine vision methods individually nor the previously mentioned machine learning methods can meet the detection and processing time requirements of the fruit and peduncle in the autonomous harvesting of sweet peppers. However, ref. [29] first proposed the Mask Region Convolutional Neural Network (Mask R-CNN) in 2017, which manages to integrate target detection (bounding boxes), and instance segmentation (binary masks, which pixels belong to the object detected within the bounding boxes), demonstrating its ease of generalization to other tasks and potential by outperforming all existing (at its release date), single-model entries on every task (all three tracks of the COCO Common Objects in Context suite of challenges: person keypoint detection, bounding box object detection, and instance segmentation), including the COCO 2016 challenge winners [29]. In this paper, a method for the simultaneous detection of the fruit and peduncle of sweet pepper in its most common color varieties (green, yellow, orange, and red), focusing on green sweet pepper, the most complicated color due to the environment and usingaround 14,600 objects (85% of the dataset), besides considering examples in all its phases of ripeness based on Mask R-CNN, capable of working in real-time in a real production environment (in this case, the sweet pepper greenhouse at CAETEC Experimental Agricultural Field of the Tec de Monterrey campus Queretaro) was proposed. In an occluded environment, where the targets are so similar to their background and obstacles such as leaves and branches cannot be removed during the growth process, Mask R-CNN not only manages to recognize the categories (fruit and peduncle) in about 1 s with a medium-high precision and mark the regions of the objects with bounding boxes but also extracts the regions of the objects at the pixel level, similar to a binary mask. We propose a real-time simultaneous 2D detection method for green sweet pepper peduncles and fruits based on a Mask R-CNN implementation (the most complex color variety of sweet pepper), thus aiming to strengthen the state-of-the-art of simultaneous sweet pepper peduncle and fruit detection methods based on the use of a state-of-the-art model in instance segmentation not previously reported for this application (Mask R-CNN), in addition to generating an implementation capable of processing high-resolution two-dimensional images in real-time, and finally achieving instance segmentation results not previously reported in this field of research. The structure of the remainder of this paper is as follows: Section 2 introduces the data acquisition process, the image annotation method, and dataset construction, the Mask R-CNN instance segmentation algorithm, the evaluation metrics, and the implementation of the Mask R-CNN model developed. Section 3 presents the experimental results, analysis, and corresponding discussion.