## 2 Materials and Methods  
### Hand Gesture Recognition  
#### 2.1.1 Data Acquisition  
The data acquisition process is an essential element in order to properly train an ML algorithm. In the present study, the data acquisition was performed using a ZED 2 depth camera (Stereolabs Inc., San Francisco, CA, USA) placed at a fixed height of 88 cm on a UGV ([PERSON], SAGA Robotics SA, Oslo, Norway). Five participants were involved in the present acquisition of data on both sunny and cloudy days at different sites of an orchard with _Pistacia vera_ trees in the region of Thessaly, in central Greece. Additionally, the distance between the participant and the camera was changed in favor of increasing variability. Concerning the speed of the tracking, it was set equal to 60 fps, while the focus was on the left hand. The predefined hand gestures, which are depicted in Figure 1a-e, are commonly known as \"fist\", \"okay\", \"victory\", \"flat\", and \"rock\".  
For the purpose of identifying the position of hands, the MediaPipe framework was implemented [54], similarly to recent studies such as [55; 56; 57]. MediaPipe is an open-source framework created by Google that offers various ML solutions for object detection and tracking. An algorithm was developed using Python that implements this framework for gesture recognition, by using the holistic solution including the hand detection (for the left hand) and excluding the face detection. As described in [58], a model regarding the palm detection works on the entire image and returns an oriented bounding frame of the hand. A model of hand landmarks then works on the cropped picture and returns 3D keypoints via a regression process. Each of the identified landmarks has distinct relative \(x,y\) coordinates to the picture frame. Consequently, the requirement for data augmentation, including translation and rotation, is remarkably reduced, while the previous frame can be used in case the model cannot identify the hand so as to re-localize it. Moreover, the MediaPipe model \"learns\" a consistent hand pose representation and turns out to be robust even to self-occlusions and partially visible hands. Lastly, the developed algorithm computes the Euclidian distance between all the identified landmarks along with shoulder and elbow angles. The aforementioned angles refer to the body side that corresponds to the tracked hand. In Figure 2, the aforementioned landmarks are depicted pertaining to the examined hand gestures, with the images taken in the laboratory for the sake of better distinguishing the landmarks.  
The dataset, which was used for training, consisted of 4819 rows created from images taken from the RGB-D camera in the orchard. Each row of the dataset represents the generated landmarks from the MediaPipe hand algorithm for each video frame. On the other hand, each column (except the last three) indicates the Euclidian distance between two landmarks to the given frame. The last three columns represent the shoulder angle, the elbow angle, and the class, respectively. As mentioned above, the calculation of the Euclidian distance and angles was carried out by the developed algorithm, while the classification of the hand gesture was performed manually for each case.  
Figure 1: Images of the investigated hand gestures: (**a**) “fist”, (**b**) “okay”, (**c**) “victory”, (**d**) “flat”, and (**e**) “rock”.  
In general, when using an ML algorithm, it is of central importance to train it by utilizing a dataset with about the same number of samples for each class. In this analysis, since the participants carried out the five preset hand gestures in the same time period, the classes resulted in approximately the same number of instances and, consequently, in balanced classes, as depicted in Figure 3.