## 4 Experimental Evaluation
### Effectiveness of Bee Identification  
The bee identification process tested within these experiments was performed with the AutoML Vision Edge service in the Google Cloud Platform. Due to its compatibility with Edge TPU, AutoML is recommended for preparing detection and classification models. The AutoML Vision Edge service builds models through transfer learning and searching neural architecture to find the best network architecture. Moreover, it optimizes hyperparameters of that architecture to minimize the loss function of the trained models.  
As a training set for the bee segmentation model, we used 300 images (video frames) extracted from the video data stream, where bees were marked with rectangles (frames). The images were taken by the camera mounted on the monitoring IoT device described in Section 3.1. Since images were acquired during the whole day, the training set included pictures taken in various light conditions, with different numbers and positions of bees. The example of an image included in the training set is shown in Figure 5.  
While evaluating the effectiveness of the detection processes, we used the following rates and effectiveness indicators:  
* A confusion (error) matrix--presents the performance of the classification algorithm in a visual manner, describes the number of actual and predicted cases classified properly as true positives (TP) and true negatives (TN) or in a wrong manner as false positives (FP) and false negatives (FN).
* Precision (positive predicted value--PPV)--the number of properly classified positive results (e.g., correctly identified bees) to a sum of all positives, including those incorrectly classified as positives, which is defined as: \[PPV=\frac{TP}{TP+FP},\] (1)  
Figure 4: General image analysis employing CNNs with reinforced learning for bee identification and varroosis detection.  
Sensitivity (recall, true positive rate--TPR)--the number of properly classified positives (e.g., correctly identified bees) divided by all samples that should have been classified as positives, defined as: \[TPR=\frac{TP}{P}=\frac{TP}{TP+FN},\] (2)
* F1-score, which stands for the harmonic mean of precision and sensitivity: \[F_{1}=2*\frac{PPV*TPR}{PPV+TPR}=\frac{2 TP}{2 TP+FP+FN}.\] (3)  
The effectiveness of bee identification was evaluated with a test set containing 20 images with various numbers of bees, their positions, and in different light conditions. These images were processed by the bee identification model. After this step, we obtained a copy of the image where detected bees were marked with ROIs (bee segmentation). To ensure the readability of presented data, the images were named with numbers, beginning from 1 up to 20. Then, the results were analyzed manually to determine whereas all the bees were detected properly. For every single image, we prepared the confusion matrix that allowed us to calculate the values of precision (1), sensitivity (2), and F1 score (3) rates, which are presented in Table 2.  
Analyzing the obtained results, we can observe that only two bees were identified for one image (no. 19), even though they were not present in the image (False Positive rate). The bee identification model performed a bit worse while finding bees assembled in groups, staying close to each other. The sensitivity rate (TPR) and F1 score reflect this situation--both of them have a wide range of values for different images. Figure 6 presents images no. 7 and no. 10, for which we obtained the worst detection results. However, considering that bees are leaving and entering a beehive a couple of times a day, and the IoT device will take their pictures several times, the outcome of this experiment is auspicious.  
Figure 5: A sample picture extracted from the video data stream with bees marked with rectangles (frames) used for training the model of bee detection.