# Object-Environment Fusion of Visual System for Automatic Pear Picking
## 1 Introduction  
Picking is the most time-consuming and labor-intensive part of fruit harvesting, accounting over 50% of the productive process [1]. Therefore, it is crucial to develop an automatic fruit harvesting robot that works in the field. Automatic fruit picking is generally composed of several key parts [2], including a visual system, hand-eye calibration, robotic arm path planning, and platform navigation. The visual system is responsible for tasks such as fruit recognition, localization, and picking pose estimation. The function of hand-eye calibration is to transform visual information into the coordinate system of the robot arm. Path planning helps the robot arm to reach the object via a specific trajectory. Platform navigation moves the picking system to different positions for autonomous operation. The visual system provides inputs and constraints for hand-eye calibration and path planning, and this plays a significant role in automatic fruit harvesting [3]. The visual system usually consists of one or more tasks of object perception, environmental perception, and picking pose estimation [4; 5; 6; 7]. However, most visual systems only focus on a single task. For example, they only recognize fruits [8; 9] in the field. Moreover, it faces complex environmental conditions, such as the occlusion of branches and leaves and uneven illumination [10; 11]. These problems can cause picking failure and affect its practicability. Therefore, this paper comprehensively analyzes a visual system for automatic fruit picking. The primary task of the visual system is object perception, which recognizes the fruit from a red, green, blue (RGB) image. In recent years, deep learning has become an advanced method, including the you only look once (YOLO) series [12; 13], mask-region-based convolutional neural network (Mask R-CNN) [14], and faster region-based convolutional neural network (Faster R-CNN) [15]. [PERSON] et al. [16] used YOLOv4 for real-time pear counting. They found that YOLOv4 had a very low false negative rate in detection. [PERSON] et al. [17] aimed to improve the YOLOv5s model for grape and stem identification. They introduced a Ghost module and replaced traditional convolutions with deep convolutions to realize a low computational load. In the study by [PERSON] et al. [18], in order to detect and classify ripe strawberries, an ECA attention mechanism and the Focal-EIOU loss were used to obtain a balance between easy- and difficult-to-classify samples. [PERSON] et al. [19] utilized bilinear interpolation and edge loss to improve Mask R-CNN, which solved the problem of blurred edge features caused by the overlap between retained and pruned branches. The results demonstrated that the improved Mask R-CNN outperformed Mask R-CNN by 6.7% in terms of the recognition rate of retained branches. [PERSON] et al. [20] proposed adding attention mechanism modules to improve Mask R-CNN; they introduced squeeze-and-excitation attention, convolutional block attention, and coordinate attention to the backbone network, respectively. To reduce the impact of irrelevant features in identifying bruises on apples, [PERSON] et al. [21] improved Faster R-CNN by incorporating a feature pyramid network (FPN) into ResNet50 [22] and integrating a normalization-based attention module into the residual network. In the study by [PERSON] et al. [23], in order to improve Faster R-CNN, Res2 Net and a feature pyramid network were used as the backbone network, and RoIAlign was replaced with RoIPool. The results demonstrated that the improved Faster R-CNN achieved average precision of 63.1%. [PERSON] et al. [9] evaluated the performance of Mask R-CNN and Faster R-CNN in detecting pears. They found that Mask R-CNN was better than Faster R-CNN. Their mAPs were 0.9945 and 0.8752, respectively. Most of the object perception models above use a convolutional neural network (CNN) as the backbone network. However, with the emergence of Transformer, its use as the backbone to extract image features has become a trend.