## 4 Study and Evaluation
### Evaluation Metrics  
The analysis of on-device models goes beyond the mere evaluation of detector performance (typically modeled in terms of accuracy and speed), additionally covering cost-specific aspects (i.e., model size and computational complexity), which are fundamental to assessing the feasibility of such solutions in resource-limited devices. Although cost is a concern that can be considered inherent to the on-device paradigm, that premise is not reflected in the data presented in Table 4. Instead, they largely omit information regarding the size [49; 50; 52; 57; 58; 59; 63; 65; 66; 68; 69; 72; 73; 75; 76; 77; 78; 84; 88; 89; 91; 92; 93; 94; 96; 97; 103; 104; 106; 108; 110]and complexity of the models produced [52; 54; 56; 57; 58; 59; 62; 63; 65; 66; 67; 68; 69; 70; 72; 73; 75; 76; 77; 78; 80; 81; 82; 83; 84; 87; 89; 91; 93; 104; 106; 107; 108; 109; 110], with no values in those respect reported in more than half of the papers. On the contrary, speed and accuracy are addressed in virtually all the AmI studies analyzed [48; 52; 53; 54; 55; 56; 57; 59; 62; 64; 65; 66; 67; 69; 70; 71; 74; 75; 76; 77; 78; 82; 85; 87; 90; 93; 93; 95; 96; 98; 100; 103; 105; 107; 109], revealing a trend consistent with the experimentation traditionally conducted for evaluating conventional ML and DL techniques. As far as accuracy is concerned, the related metrics that emerge in Table 4 as the most popular options are average precision (AP) and its variant, the mean average precision (mAP) [50; 52; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 67; 74; 75; 78; 80; 82; 83; 84; 85; 89; 93; 96; 98; 102; 105; 107; 109; 110], distantly followed by precision (P) [55; 65; 68; 69; 89; 96; 97; 99; 102; 103; 106; 109], recall (R) [55; 65; 68; 89; 96; 97; 99; 102; 103; 106; 109], and F1 score [55; 66; 76; 80; 91; 96; 97; 107; 109]. AP is computed by selecting the 11 best predictions produced by a detection framework for a single object. Specifically, given the number of true positives (TP), for each of the 11 selected predictions, both precision (P) and recall (R) values are calculated, being TP every correct detection present in the ground truth, represented as a bounding box. For its part, precision is the ratio between the number of TP and the total number of detections, representing the ability of a model to identify only the relevant objects, while recall is the percentage of TPs over the total number of detections present in the ground truth, being the result indicative of the model's ability to find all relevant cases (all ground-truth bounding boxes). Lastly, the F1 score is computed as the weighted average of precision and recall, resulting in a value between 0 and 1, where 1 represents the highest precision. Furthermore, the object detection framework's speed is the time it requires to process a given image or video frame to output the various bounding boxes representing the location of the objects of interest together with the labels identifying the class that objects are associated with. Such inference speed is usually denoted as the number of frames processed per second [48; 49; 52; 53; 54; 55; 57; 60; 61; 62; 67; 69; 71; 75; 76; 78; 79; 82; 83; 84; 85; 92; 93; 100; 103; 105; 107; 109], although, as shown in the \"Speed\" column of Table 4, there are a good number of authors who choose to directly report the average time spent in the inference process [55; 56; 59; 64; 65; 66; 67; 68; 69; 74; 77; 87; 90; 91; 95; 96; 98; 100; 101] (typically measured in milliseconds). Accuracy and speed are closely related to the complexity of the trained model: a superior model representation capacity makes it possible to produce more accurate predictions, but, on the flip side, it generally results in a longer inference time.