# Green Sweet Pepper Fruit and Peduncle Detection Using Mask R-CNN in Greenhouses
###### Abstract  
In this paper, a mask region-based convolutional neural network (Mask R-CNN) is used to improve the performance of machine vision in the challenging task of detecting peduncles and fruits of green sweet peppers (_Capsicum annuum_ L.) in greenhouses. One of the most complicated stages of the sweet pepper harvesting process is to achieve a precise cut of the peduncle or stem because this type of specialty crop cannot be grabbed and pulled by the fruit since the integrity and value of the product are compromised. Therefore, accurate peduncle detection becomes vital for the autonomous harvesting of sweet peppers. ResNet-101 combined with the feature pyramid network (FPN) architecture (ResNet-101 + FPN) is adopted as the backbone network for feature extraction and object representation enhancement at multiple scales. Mask images of fruits and peduncles are generated, focused on green sweet pepper, which is the most complex color variety due to its resemblance to the background. In addition to bounding boxes, Mask R-CNN provides binary masks as a result of instance segmentation, which would help improve the localization process in 3D space, the next phase of the autonomous harvesting process of sweet peppers, since it isolates the pixels belonging to the object and demarcates its boundaries. The prediction results of 1148 fruits on 100 test images showed a precision rate of 84.53%. The prediction results of 265 peduncles showed a precision rate of 71.78%. The mean average precision rate with an intersection over union at 50 percent (mAP@IoU=50) for model-wide instance segmentation was 72.64%. The average detection time for sweet pepper fruit and peduncle using high-resolution images was 1.18 s. The experimental results show that the proposed implementation manages to segment the peduncle and fruit of the green sweet pepper in real-time in an unmodified production environment under occlusion, overlap, and light variation conditions with effectiveness not previously reported for simultaneous 2D detection models of peduncles and fruits of green sweet pepper. green sweet pepper; peduncle; fruit detection; Mask R-CNN; instance segmentation +
Footnote †: journal:  
crops, along with the background of the same color as the fruits and stems). In general, research works such as [3] allow us to observe clear examples of this phenomenon. During the last decades, researchers have dedicated their efforts to improving the metrics, time cycles, and detection of the technological frameworks for harvesting various relevant crops. The Agrobot strawberry harvesting robot is fully automated [4], with a harvesting efficiency of 3-5 s per fruit. The cucumber harvesting robot [5] in occluded environments has a harvest success rate of 85% and an efficiency of 8.6 s per fruit. The apple harvesting robot from Abundant Robotics [6] has an average efficiency rate of 1 s per fruit on trees with a standardized shape. Robotics Plus [7] kiwi harvesting robot has a 51 percent success rate and a harvesting efficiency of 5.5 s per fruit. In the same way, in the field of sweet peppers, there is no exception: the SWEEPER sweet pepper harvesting robot [8] is capable of working day and night in greenhouses with an average harvest success rate of 61% and a harvest efficiency of 24 s per fruit. Ref. [9] proposes a method for segmenting green sweet peppers on remarkably similar colored backgrounds, obtaining a mean average precision of 0.55 or 55%. Ref. [10] proposes an adaptive segmentation algorithm under various illuminations for red sweet pepper, reaching a recognition rate of 87%. Ref. [11] establishes a support vector machine (SVM) for the detection of pepper fruit, achieving a recognition rate of 74.2%. Ref. [12] takes color space parameters such as brightness Y, Cb, Cr, hue, and saturation as inputs to the neural network that he uses for the recognition of red sweet peppers, reaching 82.16% accuracy. Recent works have started using machine learning models combined with classical machine vision algorithms [11; 12].