# A State Validation System for Augmented Reality Based Maintenance Procedures
## 4 Workflow
### Validation  
The validation phase has the goal to verify if the status change identified by the system corresponds or not to a correct operation performed by the user. In order to verify if the operation was performed correctly, it is necessary to compare the real object after the status change with the expected final status for the current step of the procedure. In order to properly compare the real scene with the 3D representation of the final status, it is necessary to define some guidelines. If the environment is known in advance, it would be possible to create a close representation of the real scene in Blender. If this is not the case, it is necessary to assume that the real environment is properly lit, with a stable and uniform light which minimizes shadows, glints, flares or spotlights on the target object. This is necessary to perform a fair comparison between real scene and 3D scene. Assuming that it is possible \begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**BOUNDING BOX**} \\ \hline  & X\_max & 7 \\ \hline  & X\_min & 1 \\ \hline  & Y\_max & 10 \\ \hline  & Y\_min & 1 \\ \hline \hline \multicolumn{4}{c}{**FEATURE KEY POINTS**} \\ \hline
**X** & **Y** & **LAST FRAME** & **FIRST FRAME** & **COUNT** \\ \hline
2 & 2 & 45 & 1 & 44 \\ \hline
6 & 9 & 45 & 7 & 38 \\ \hline
6 & 3 & 45 & 7 & 38 \\ \hline
4 & 8 & 45 & 12 & 32 \\ \hline
3 & 5 & 45 & 22 & 23 \\ \hline
2 & 7 & 45 & 25 & 20 \\ \hline
1 & 9 & 45 & 45 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The bounding box’s coordinates (left table) and the key points detected from the first frame to the 45 th frame (right table). For each key point which is inside the bounding box’s coordinates, the count value (which is the difference between the first frame and the last frame it was detected in) determine the steady features. to interact with the illumination features of the working environment, it is necessary to avoid all the setups that would introduce noise in the scene. Figure 6 shows the workflow of the validation module. First of all, the validation algorithm takes a snapshot of the scene, right after a status change is detected. This snapshot will be used as reference for the real object final status. Then, a virtual representation of the expected final status is created taking into account the following variables:  
* The camera's specifications;
* The camera's intrinsic and extrinsic parameters;
* The coordinate system provided by the tracking system;
* The object coordinates;
* A 3D model representing the final status of the object. Since the proposed solution relies on a tracking system independent of the environment, it is not possible to have a prior formal description of the environment and of the lights surrounding the object. Lights have been added in Blender to the virtual scene in order to increase the scene's realism, providing shadows and environment illumination. Finally, the scene is rendered to obtain an image that could be compared with the snapshot of the real object; the rendering is performed in Blender through a Python script. The snapshot and the render are then processed by the tracking algorithm in order to extract the two sets of feature key points, \(S_{kp}\) and \(R_{kp}\). Then, the number of key points that are equals between the two sets is computed (\(M_{kp}\)). Two key points are equals if they occupy the same 2D coordinates relative to the object's bounding box. Since the rendered image provides a lower level of detail compared to the real image, it is reasonable to assume that the tracking algorithm will find a lower number of key points for the rendered image. Moreover, even if a lot of parameters have been taken into account to render the scene as similar as possible to the real one, some differences from the two images should be expected. Since such differences may preclude a meaningful comparison, experiments have been carried out to identify a reasonable tolerance value that would minimize both false negative and false positive comparisons.