# Object-Environment Fusion of Visual System for Automatic Pear Picking
## 1 Introduction
Some researchers [24] have demonstrated that it performs better than CNNs in image classification, detection, and segmentation. It is necessary to implement environmental perception after object perception [25; 26]. It can obtain 3D information about the object and the surrounding obstacles. Environmental perception needs depth information collected by a red, green, blue depth (RGB-D) camera to obtain point clouds [25]. The three mainstream types of RGB-D cameras [27] are stereo, structured light, and time-of-flight cameras. [PERSON] et al. [4] used the Kinect v2 camera to obtain the the point clouds of tea shoots. Environmental perception mainly focuses on improving the accuracy of object 3D localization and normalizing the point clouds. [PERSON] et al. [5] applied many voxel boxes to fit the environment, which was convenient for the planning of paths without collision. [PERSON] et al. [25] used an octree to represent the occupied space and simplify the complex environment. [PERSON] et al. [27] employed an ellipse to fit the mango shape to estimate its size, with a root mean square error of 4.9 mm. The final task of visual perception is to estimate the picking pose. In the early stage, many researchers [10; 28] applied direct apple picking without considering the picking pose. As a result of ignoring the integrity judgment of the object, the partially occluded apples would likely collide during picking. Therefore, an appropriate picking pose can enable more efficient picking suggestions. [PERSON] et al. [29] used the direction from the center of the fitting sphere to the center of the point cloud as the picking orientation. [PERSON] et al. [25] estimated the picking pose based on the 3D spherical Hough transform of the fruit. The above studies do not consider the influence of obstacles on the picking pose, easily leading to collisions during picking. As a result, some researchers have begun to study the prediction of the picking pose from object and environment point clouds by neural networks. [PERSON] et al. [30] proposed a collision detection network based on deep learning to avoid collisions during grasping. Then, [PERSON] et al. [31] provided a heuristic-based sampler to give several six-degrees-of-freedom (6 Dof) picking candidates. Then, a deep network classified each candidate into graspable and non-graspable categories. Moreover, some studies [32; 33; 34; 35] have directly computed the 6 DoF picking pose with an end-to-end regression network. However, these methods are more suitable for structured environments and rely on the complex and time-consuming process of collecting and labeling datasets. Therefore, it is necessary to combine object and environment point clouds to explore more efficient methods of picking pose estimation in the field. [PERSON] et al. [7] proposed a global-to-local picking pose estimation method using RGB-D camera color and depth data. From the experimental results, the picking success rate reached 80%. To prevent the gripper from colliding with the branches, the picking pose was defined as the vector direction passing through the center of the fruit and perpendicular to the mother branch of the fruit [36], and its 3D pose error was \(23.43^{\circ}\pm 14.18^{\circ}\). The visual system for automatic fruit picking faces complex environmental conditions and involves a series of tasks. It is necessary to pay attention to the specific implementation of each task and comprehensively fuse the information from different tasks to enhance the adaptability of the visual system in the field. Therefore, we propose an object-environment fusion visual system for automated pear picking. Our contributions are as follows. * An object-environment fusion visual system is proposed for pear picking. It includes three modules: object perception, environmental perception, and picking pose estimation. In particular, picking pose estimation depends on fusing the information of the objects and environment. * We provide specific implementations for the system.