# A Vision Servo System for Automated Harvest of Sweet Pepper in Korean Greenhouse Environment
## 2 Material and Methods
### Vison Servo Control  
Since an image processing-based harvest uses a non-contact measurement method, it is appropriate for robot harvest works. The visual detection and control can be seen through the open-loop mode which \"sees\" and \"moves.\" In addition, the accuracy of works depends on the accuracy of the image sensor and the accuracy of the end-effector control [20]. As an alternative to enhance the accuracy of the servo system, this study used a visual-feedback control loop, and increased the accuracy of the system. The mechanical vision can provide a closed-loop position for the robot's control of the end-effector and this mechanism is referred as a vison servo system. As shown in Figure 9, an eye-in-hand vision servo system was used in this study. The eye-in-hand vision servo system is a type of camera that is fixed at a robot's end-effector. In this method, there are already known and fixed relations between the poses of the camera and end-effector. In this case, the pose relation between the end-effector and a camera is referred to as \(X_{c}^{e}\), and the pose relation between an object and a camera is referred to as \(X_{t}^{c}\). The image-based control method was used to control the pose of the end-effector approaching the object. According to [PERSON] [31], in the image-based vision servo, control is carried out by the position and pose of an object that exists in the image. In order to realize a vision servo-type sweet pepper harvest algorithm, this study acquired the center of a sweet pepper through the camera installed at end-effector, and then controlled the end-effector to be matched with the said sweet pepper, thus extracting feature points and adjusting stereo matching, and then measuring distances. After acquiring a three-dimensional coordinate, the end-effector approaches a sweet pepper based on the information acquired and then detects its position through an image, thus manipulating the end-effector. Vision servo system's working processes for the automated harvest of sweet pepper are shown in Figure 10. The center of the sweet pepper was first extracted through Cam3 installed in the end-effector, and the end-effector was controlled so that the direction of the end-effector coincided with the direction of the sweet pepper. Then, the distance was measured through stereo matching. Based on the 3D coordinate information measured by Cam1 and Cam2, after the end-effector approaches the sweet pepper, the posture of the end-effector is controlled while confirming the position of the fruit by the image of Cam3. After that, the vision servo system checks the image of Cam3 and repeats the process so that the peduncle part of the sweet pepper approaches the cut part of the end-effector. When the ROI (region of interest) of the sweet pepper is extracted, the manipulator rotates to place the center of the ROI in the center of Cam3, thus controlling the Y-axis manipulator and then making the center of Cam3 matched with the bottom of the ROI. The reason for this process is to allow Cam1 and Cam2 to acquire images of sweet pepper fruit during stereo matching. Experiments were conducted to confirm that the center of the camera and the center of the paprika were controlled to coincide with each other and confirmed the results. The results were analyzed by measuring the unit of pulse for controlling the manipulator, the rotation angle of the rotation axis manipulator, and the movement distance of the vertical axis manipulator. Figure 9: Eye-in-hand vison servo system. When the center of the ROI and the center of Cam3 are matched, the system extracts the source of laser light irradiated and then carries out the adjustment of irradiated points, thus measuring distances. It extracts green laser that exists in the sweet pepper and then examines the original form of the laser. In case the form is circle, it extracts the center of the circle and carries out adjustments on the basis of the center of the circle. When it carries out the adjustment, it uses epipolar geometry [32].