# A Voice-Enabled ROS2 Framework for Human-Robot Collaborative Inspection
## 4 Case Study
### Evaluation Criteria
#### 4.3.2 Quality Inspection Efficiency  
To assess the accuracy of the quality inspection module, a distinct validation dataset was employed. The mean Average Precision (mAP) served as the evaluation metric for the model, a widely used metric for object detection tasks. To compute mAP, it is essential to initially form the precision-recall curve by calculating the corresponding sub-metrics for a range of different confidence thresholds. Usually, object detection models output a confidence score which indicates the confidence of the model that a predicted bounding box contains an object of a specific class. Therefore, confidence and IoU thresholds differ, since the first refers to the class prediction while the latter to the bounding's box coordinates. The area under the precision-recall curve (AUC) is a single value that represents the model's performance for a specific class, and it is named Average Precision (AP). In order to find a metric that contains the desired information for all the classes, the mean Average Precision (mAP) is used, which is basically the mean of all the Average Precisions that have been calculated for each class separately. In the presented implementation, the calculation of the mAP on the evaluation dataset is occurring simultaneously with the training of the model in order to provide an overview of the \"learning\" across different epochs. More precisely, the training procedure is interrupted after every three epochs to conduct an evaluation process. During that process, two variations of mAP are calculated. The first is the mAP with a constant IoU threshold equal to 0.5, while the second is referred to as mAP:0.5:0.95 and is basically the mean Average Precision calculated for a range of IoU thresholds from 0.5 to 0.95, in increments of 0.05. This metric provides a more complete picture of the model's performance in detecting objects at different levels of overlap with the ground truth boxes. The difference between the two metrics also appears in the following figures displaying the mAP values during all training steps. Also, for this use case scenario, a total of 18 tests were conducted in order to evaluate the model's real-time operation. During these tests, the operator and the robot jointly examined the motor engine to identify any missing or incorrectly positioned components. The tests were conducted under different lighting conditions in order to examine the robustness of the proposed inspection module. Since the main purpose of the proposed system is to determine the quality of a product, the model's output object categories that were considered for the conducted tests revolved around identifying defective components such as screws, connectors and valves. Nevertheless, as already mentioned, the model also underwent training to accurately identify properly assembled parts. This additional training ensures that the network can effectively understand the different states of the components, thus minimizing any potential confusion, revolving around labeling a correctly functioning component as a faulty one. The confidence threshold of the model was set to 0.92, which is the value in which the system operates in real time. To measure the illumination variance between the different tests, a modified version of the structural similarity metric (SSIM) was implemented. In general, SSIM is used as a comparison metric between two images and its value ranges from \(-1\) to 1, with 1 indicating identical images and \(-1\) a complete lack of structural similarity. The SSIM is based on three main components: luminance, contrast and structural similarity. In these experiments, since the robot positions, when inspecting the motor, are more or less the same for the three views, the structural similarity is cast out from the calculations. Therefore, the proposed metric is used only to calculate the difference between the luminance and the contrast of the different images returning a value between 0 and 1.