# A State Validation System for Augmented Reality Based Maintenance Procedures
## 5 Framework Evaluation
### Results Analysis  
The tests performed with the proposed system allowed to evaluate both its precision and its limits. First of all, due to the number of features needed to correctly evaluate the status of the object, it may be difficult to correctly identify changes when working with small components. The smallest component involved in the test and correctly recognized by the system measured 8 \(\times\) 16 \(\times\) 11.2 mm, corresponding to a bounding box area of 85 \(\times\) 90 pixels (2,49%) for a 640 \(\times\) 480 pixels frame of the scene. Thus, enhancing the proposed system with a High Definition or 4K camera will further improves the recognition precision of small objects in the scene. Figure 16: Starting position (red bounding box) and final position (black bounding box) for the given step, on the real object (**left**) and the virtual model (**right**). Figure 17: Industrial laser calibrator’s shutters displacement. The object status is calculated (**A**), the instruction provided to the user (**B**), the user remove the external cover (**C**) and then the system computes the object status again (**C**). Moreover, depending on the environment and the camera perspective, the correct computation of the features may be critical when the object removal operation is performed, because the algorithm might identify it as still present, thus missing a status change identification. The system may detect key features from the environment in the same area where the removed component was and perceive the scene as unaltered, thus leading to a false positive detection during the status analysis phase. The ViSP's tracking algorithm provides additional insights for feature identification, thus, it is possible to identify uncertain features and to classify them into two categories: Features with threshold problems and features with tracking problems. Comparing the features from these two groups it is possible to reduce the occurrences of false recognitions. Whenever the number of features that have problems is higher than the number of correct features, the component is recognized as missing, thus leading to a status change. However, in order to increase the system precision and to address the false recognition problem, it could be useful to use a depth camera to further improve the analysis of the scene's features. Another relevant aspect is that the validation step is not real-time in terms of responsiveness. The time complexity is related to two operations, the status comparison and the rendering of the virtual scene. The first one is affected by the number of features to be compared for a given element, the second one is affected by the model complexity (e.g., number of polygons and textures) and the scene complexity (e.g., environmental lighting). Thus, the time complexity strictly depends on the specific use case due to the several variables involved. In the Lego use case, the 3D model has 182 polygons and a laptop computer with an AMD A6-3400M CPU and 6 GB of RAM has been used. In these conditions, the features computation requires 4-5 s whereas the comparison takes 10-11 s and the rendering 3-4 s. It was possible to make this phase optional, thus allowing the technician to perform the validation or to avoid it and proceed to the next step. However, a through improvement would be to provide real-time validation in order to always perform it without impacting on the performances and timing of the procedure. Finally, in order to minimize the time needed by the different modules to perform the visual analysis of the scene, in each phase only the component involved in the current step is taken into account for the status analysis and validation steps. If the user accidentally performs an elementary operation on a different component than the one expected for the given step, depending on the area interested by the operation, the system may not be able to identify the status change, thus it would not be possible to provide the user any hint on what he/she is doing wrongly.